CN107392213A - Human face portrait synthetic method based on the study of the depth map aspect of model - Google Patents

Abstract

A face portrait synthesis method based on deep graph model feature learning. The steps are: (1) generate a sample set; (2) generate an image block set; (3) extract depth features; (4) solve face portrait reconstruction block coefficients; (5) reconstruct face portrait blocks; (6) ) to synthesize a face portrait. The present invention uses a deep convolutional network to extract the depth features of face photo blocks, uses the Markov graph model to solve the depth feature map coefficients and face portrait block reconstruction coefficients, and uses the face portrait block reconstruction coefficients to weight the face portrait blocks The summation is performed to obtain reconstructed face portrait blocks, and the reconstructed face portrait blocks are spliced to obtain a composite face portrait. The present invention uses depth features extracted from deep convolutional networks to replace original pixel value information of image blocks, has better robustness to environmental noise such as illumination, and can synthesize extremely high-quality face portraits.

Description

Face portrait synthesis method based on deep graph model feature learning

technical field

The invention belongs to the technical field of image processing, and further relates to a face image synthesis method based on feature learning of a depth map model in the technical field of pattern recognition and computer vision. The invention can be used for face retrieval and recognition in the field of public security.

Background technique

In the criminal investigation and pursuit, the public security department has a database of citizens’ photos, combined with face recognition technology to determine the identity of criminal suspects. However, it is generally difficult to obtain photos of criminal suspects in practice, but they can be obtained with the cooperation of painters and witnesses. Sketch portraits of criminal suspects for subsequent face retrieval and recognition. Due to the great difference between portraits and ordinary face photos, it is difficult to obtain satisfactory recognition results by directly using traditional face recognition methods. Combining the photos in the citizen photo database into a portrait can effectively reduce the gap in their textures, thereby improving the recognition rate.

In the paper "X.Gao, J.Zhou, D.Tao, and X.Li, Local face sketchsynthesis learning" (Neurocomputing, vol.71, no.10-12, pp.1921- 1930, Jun.2008) proposed the use of embedded hidden Markov models to generate pseudo-portraits. This method first divides the photos and portraits in the training library into blocks, and then uses the embedded hidden Markov model to model the corresponding photo blocks and portrait blocks. For any photo, it is also divided into blocks. For any A block, using the idea of selective integration, selects the model generated by some blocks to generate a pseudo-portrait and fuses it to obtain the final pseudo-portrait. The disadvantage of this method is that because the method uses selective integration technology, the generated pseudo-portraits need to be weighted and averaged, resulting in unclean backgrounds and unclear details, thereby reducing the quality of generated images.

H.Zhou et al published the paper "H.Zhou, Z.Kuang, and K.Wong.Markov WeightFields for Face Sketch Synthesis" (In Proc.IEEE Int.Conference on ComputerVision,pp.1091-1097,2012) A face portrait synthesis method based on Markov weight field is proposed in . In this method, the training image and the input test image are evenly divided into blocks, and for any test image block, several neighbors are searched to obtain the candidate block of the image form to be synthesized. Then use the Markov graph model to model the test image block, neighbor blocks and candidate image blocks, and calculate the reconstruction weight. Finally, the composite portrait block is reconstructed by using the reconstruction weight and the candidate portrait block, and the composite portrait is obtained by splicing. The disadvantage of this method is that the image block features use the original pixel information, which has insufficient representation ability and is greatly affected by environmental noise such as illumination.

The patent "Face Image Synthesis Method Based on Orientation Graph Model" (Application No.: CN201610171867.2 Application Date: 2016.03.24 Application Publication No.: CN105869134A) filed by Xidian University discloses a face recognition method based on a directional graph model. image synthesis method. In this method, the training image and the input test image are evenly divided into blocks, and for any test photo block, several adjacent photo blocks and corresponding adjacent image blocks are searched. Then extract the direction feature for the test photo block and the neighbor photo block. Then use the Markov graph model to model the test photo block, the directional features of the neighbor photo blocks and the neighbor portrait blocks, and calculate the reconstruction weights of the synthetic portrait blocks reconstructed from the neighbor portrait blocks. Finally, the composite portrait blocks are reconstructed by reconstructing the weights and the adjacent image blocks, and spliced to obtain a composite portrait. The disadvantage of this method is that the image block features use artificially designed high-frequency features, the adaptive ability is insufficient, and the features are not fully learned.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art, and propose a face image synthesis method based on depth map model feature learning, which can synthesize high-quality images that are not affected by environmental noise such as illumination.

The concrete steps that realize the object of the present invention are as follows:

(1) Generate a sample set:

(1a) Take M face photos from the face photo sample set to form a training face photo sample set, 2≤M≤U-1, U represents the total number of face photos in the sample set;

(1b) form the test face photo set with the remaining face photos in the face photo sample set;

(1c) from the face portrait sample set, take out the face portrait corresponding to the face photos of the training face photo sample set one by one, form the training face portrait sample set;

(2) Generate a collection of image blocks:

(2a) randomly select a test face photo from the test face photo set, divide the test face photo into photo blocks with the same size and the same degree of overlap to form a test photo block set;

(2b) Divide each photo in the training face photo sample set into photo blocks with the same size and the same degree of overlap to form a training photo sample block set;

(2c) Divide each portrait in the training face portrait sample set into portrait blocks with the same size and the same degree of overlap to form a training portrait sample block set;

(3) Extract depth features:

(3a) Input all the photo blocks in the training photo block set and the test photo block set into the deep convolutional network VGG trained for object recognition on the object recognition database ImageNet, and carry out forward propagation;

(3b) The 128-layer feature map output by the middle layer of the deep convolutional network VGG is used as the depth feature of the photo block, and the coefficient of each layer of the feature map is u _i,l , and Among them, ∑ represents the summation operation, i represents the sequence number of the test photo block, i=1, 2,..., N, N represents the total number of test photo blocks, l represents the sequence number of the feature map, l=1,... ,128;

(4) Solve the face portrait block reconstruction coefficient:

(4a) Use the K nearest neighbor search algorithm to find out the 10 nearest neighbor training photo blocks that are most similar to each test photo block in the training photo sample block set, and at the same time select one of the nearest neighbor training photo blocks from the training image sample block set. There are 10 adjacent training image blocks corresponding to each other, and the coefficient of each adjacent training image block is w _i,k , where, k represents the sequence number of the training image block, k=1,...,10;

(4b) Using the Markov graph model formula, the depth features of all test photo blocks, the depth features of all neighboring training photo blocks, all the neighboring training image blocks, the coefficients u _i,l of the depth feature map, and the coefficients of the neighboring training image blocks w _i,k modeling;

(4c) solving the Markov graph model formula to obtain the face portrait block reconstruction coefficient w _i,k ;

(6) Reconstruct the face portrait block:

Multiply the 10 nearest neighbor training portrait blocks corresponding to each test photo block with their respective coefficients w _{i, k} , and sum the results after multiplication, as the reconstructed face portrait block corresponding to each test photo block;

(7) Synthetic face portrait:

The reconstructed face portrait blocks corresponding to all test photo blocks are stitched together to obtain a synthetic face portrait.

Compared with the prior art, the present invention has the following advantages:

First, because the present invention uses the depth features extracted from the deep convolutional network to replace the original pixel value information of the image block, it overcomes the problems of insufficient feature representation capabilities used in the prior art and is greatly affected by environmental noise such as lighting, making this The invention has the advantage of being robust to environmental noise such as illumination.

The 2nd, since the present invention uses the Markov graph model to jointly model the depth feature map coefficients and face portrait block reconstruction coefficients, it overcomes the problems that the background of the face portrait synthesized in the prior art is not clean and the details are not clear, so that The invention has the advantages of clean background and clear details of the synthesized human face portrait.

Description of drawings

Fig. 1 is a flowchart of the present invention;

Fig. 2 is a simulation effect diagram of the present invention.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings.

With reference to Fig. 1, concrete steps of the present invention are as follows.

Step 1, generate a sample set.

Take M face photos from the face photo sample set to form a training face photo sample set, 2≤M≤U-1, U represents the total number of face photos in the sample set.

Use the remaining face photos in the face photo sample set to form a test face photo set.

From the face portrait sample set, the face portraits corresponding to the face photos in the training face photo sample set are taken out to form the training face portrait sample set.

Step 2, generate a set of image blocks.

Randomly select a test face photo from the test face photo set, and divide the test face photo into photo blocks with the same size and the same overlapping degree to form a test photo block set.

Each photo in the training face photo sample set is divided into photo blocks with the same size and the same overlapping degree to form a training photo sample block set.

Divide each portrait in the training face portrait sample set into portrait blocks with the same size and the same overlapping degree to form a training portrait sample block set.

The degree of overlap means that the overlapping area between two adjacent image blocks accounts for 1/2 of each other.

Step 3, extracting deep features.

All the photo blocks in the training photo block set and the test photo block set are input into the deep convolution network VGG for object recognition that has been trained on the object recognition database ImageNet for forward propagation.

The 128-layer feature map output by the middle layer of the deep convolutional network VGG is used as the depth feature of the photo block, and the coefficient of each layer of the feature map is u _i,l , and Among them, ∑ represents the summation operation, i represents the sequence number of the test photo block, i=1, 2,..., N, N represents the total number of test photo blocks, l represents the sequence number of the feature map, l=1,... ,128.

The middle layer refers to the activation function Relu2_2 layer of the deep convolutional network VGG

Step 4, solving the face portrait block reconstruction coefficient.

Use the K nearest neighbor search algorithm to find out the 10 nearest neighbor training photo blocks that are most similar to each test photo block in the training photo sample block set, and select the one-to-one corresponding neighbor training photo block from the training image sample block set. 10 adjacent training image blocks, the coefficient of each adjacent training image block is w _i,k , where, k represents the serial number of the training image block, k=1,...,10.

The specific steps of the K nearest neighbor search algorithm are as follows:

The first step is to calculate the Euclidean distance between the depth feature vectors of each test photo block and the depth feature vectors of all training photo blocks;

The second step is to sort all training photo blocks according to the order of Euclidean distance value from small to large;

The third step is to select the first 10 training photo blocks as neighbor training photo blocks.

Using the Markov graph model formula, the depth features of all test photo blocks, the depth features of all neighboring training photo blocks, all neighboring training image blocks, the coefficient u _i,l of the depth feature map, and the coefficient w _i, of the neighboring training image blocks _kModeling .

The Markov graph model formula is as follows:

Among them, min represents the minimum value operation, ∑ represents the sum operation, |||| ² represents the modulus square operation, w _i,k represents the coefficient of the kth neighbor training image block of the i-th test photo block, o _{i , k} represents the pixel value vector of the overlapping part of the kth neighbor training image block of the i-th test photo block, w _{j, k} represents the coefficient of the j-th neighbor training image block of the j-th test photo block, o _{j, k} represents the pixel value vector of the overlapping part of the kth neighbor training image block of the jth test photo block, u _i,l represents the coefficient of the depth feature map of the lth layer of the depth feature of the ith test photo block, d _l (x _i ) represents the h-th layer feature map of the depth feature of the i-th test photo block, d _l ( _xi,k ) represents the u-th depth feature map of the k-th neighbor training photo block of the i-th test photo block The layer feature map, the values of l, h, and u are correspondingly equal. .

The Markov graph model formula is solved to obtain the face portrait block reconstruction coefficient w _i,k .

Step 5, reconstruct the face portrait block.

Multiply the 10 adjacent training portrait blocks corresponding to each test photo block with their respective coefficients w _i,k , and sum the results after multiplication, and use it as the reconstructed face portrait block corresponding to each test photo block.

Step 6, synthetic face portrait.

The reconstructed face portrait blocks corresponding to all test photo blocks are stitched together to obtain a synthetic face portrait.

The method of splicing the reconstructed portrait blocks corresponding to all test photo blocks is as follows:

The first step is to place the reconstructed portrait blocks corresponding to all test photo blocks located in different positions of the portrait according to their positions;

The second step is to get the average value of the pixel values of the overlapping parts between adjacent two reconstructed face portrait blocks;

The third step is to replace the pixel values of the overlapping parts between two adjacent reconstructed face portrait blocks with the average value of the pixel values of the overlapping parts between two adjacent reconstructed face portrait blocks to obtain a synthetic face portrait.

The effects of the present invention are further illustrated by the following simulation experiments.

1. Simulation experiment conditions:

The computer configuration environment of simulation experiment of the present invention is Intel (R) Core i7-4790 3.6GHZ, memory 16G, Linux operating system, programming language uses Python, database adopts CUHK student database of Chinese University of Hong Kong.

The comparison method of the prior art used in the emulation experiment of the present invention comprises following two kinds:

One is a method based on local linear embedding, which is recorded as LLE in the experiment; the reference is "Q.Liu, X.Tang, H.Jin, H.Lu, and S.Ma" (A Nonlinear Approach for Face Sketch Synthesis and Recognition .In Proc.IEEE Int.Conference on Computer Vision,pp.1005-1010,2005);

The other is a method based on the Markov weight field model, which is recorded as MWF in the experiment; the reference is "H. Zhou, Z. Kuang, and K. Wong. Markov Weight Fields for Face Sketch Synthesis" (InProc.IEEE Int . Conference on Computer Vision, pp.1091-1097, 2012).

2. Simulation experiment content:

The present invention has a group of simulation experiments.

The portraits were synthesized on the CUHK student database, and compared with those synthesized by local linear embedding LLE and Markov weight field model MWF methods.

3. Simulation experiment results and analysis:

The simulation experiment result of the present invention is as shown in accompanying drawing 2, and wherein Fig. 2 (a) is a test photo that takes out arbitrarily from test photo sample collection, and Fig. 2 (b) is to use prior art local linear embedding LLE method to synthesize Portrait, Fig. 2 (c) is a portrait synthesized using the Markov weight field model MWF method of the prior art, and Fig. 2 (d) is a portrait synthesized using the method of the present invention.

It can be seen from Figure 2 that since the present invention uses depth features to replace the original pixel value information of image blocks, it has better robustness to environmental noise such as illumination, so for photos that are greatly affected by illumination, compared with local linear embedding LLE, Markov weight field model MWF method, the synthetic image quality is higher and the noise is smaller.

Claims (6)

1. A method for synthesizing face portraits based on depth map model feature learning, comprising the steps of: (1) Generate a sample set: (1a) Take M face photos from the face photo sample set to form a training face photo sample set, 2≤M≤U-1, U represents the total number of face photos in the sample set; (1b) form the test face photo set with the remaining face photos in the face photo sample set; (1c) from the face portrait sample set, take out the face portrait corresponding to the face photos of the training face photo sample set one by one, form the training face portrait sample set; (2) Generate a collection of image blocks: (2a) randomly select a test face photo from the test face photo set, divide the test face photo into photo blocks with the same size and the same degree of overlap to form a test photo block set; (2b) Divide each photo in the training face photo sample set into photo blocks with the same size and the same degree of overlap to form a training photo sample block set; (2c) Divide each portrait in the training face portrait sample set into portrait blocks with the same size and the same degree of overlap to form a training portrait sample block set; (3) Extract depth features: (3a) Input all the photo blocks in the training photo block set and the test photo block set into the deep convolutional network VGG trained for object recognition on the object recognition database ImageNet, and carry out forward propagation; (3b) The 128-layer feature map output by the middle layer of the deep convolutional network VGG is used as the depth feature of the photo block, and the coefficient of each layer of the feature map is u _i,l , and Among them, ∑ represents the summation operation, i represents the sequence number of the test photo block, i=1, 2,..., N, N represents the total number of test photo blocks, l represents the sequence number of the feature map, l=1,... ,128; (4) Solve the face portrait block reconstruction coefficient: (4a) Use the K nearest neighbor search algorithm to find out the 10 nearest neighbor training photo blocks that are most similar to each test photo block in the training photo sample block set, and at the same time select one of the nearest neighbor training photo blocks from the training image sample block set. There are 10 adjacent training image blocks corresponding to each other, and the coefficient of each adjacent training image block is w _i,k , where, k represents the sequence number of the training image block, k=1,...,10; (4b) Using the Markov graph model formula, the depth features of all test photo blocks, the depth features of all neighboring training photo blocks, all the neighboring training image blocks, the coefficients u _i,l of the depth feature map, and the coefficients of the neighboring training image blocks w _i,k modeling; (4c) solving the Markov graph model formula to obtain the face portrait block reconstruction coefficient w _i,k ; (5) Reconstruct the face portrait block: Multiply the 10 nearest neighbor training portrait blocks corresponding to each test photo block with their respective coefficients w _{i, k} , and sum the results after multiplication, as the reconstructed face portrait block corresponding to each test photo block; (6) Synthetic face portrait: The reconstructed face portrait blocks corresponding to all test photo blocks are stitched together to obtain a synthetic face portrait. 2. the method for synthesizing human face portraits based on depth map model feature learning according to claim 1, is characterized in that: the degree of overlap described in step (2a), step (2b), step (2c) refers to, relative The overlapping area between two adjacent image blocks accounts for 1/2 of each other. 3. the method for synthesizing face portraits based on depth map model feature learning according to claim 1, characterized in that: the intermediate layer described in step (3b) refers to the activation function Relu2_2 layer of deep convolutional network VGG. 4. the method for synthesizing human face images based on depth map model feature learning according to claim 1, is characterized in that: the concrete steps of K nearest neighbor search algorithm described in step (4a) are as follows: The first step is to calculate the Euclidean distance between the depth feature vectors of each test photo block and the depth feature vectors of all training photo blocks; The second step is to sort all training photo blocks according to the order of Euclidean distance value from small to large; The third step is to select the first 10 training photo blocks as neighbor training photo blocks. 5. the method for synthesizing face portraits based on depth map model feature learning according to claim 1, is characterized in that: the Markov map model formula described in step (4b) is as follows: <mrow><mi>min</mi><munderover><mo>&amp;Sigma;</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mo>mn></mrow><mi>N</mi></munderover><mo>|</mo><mo>|</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>k</mi></mrow></msub><msub><mi>o</mi><mrow><mi>i</mi><mo>,</mo><mi>k</mi></mrow></msub><mo>-</mo><msub><mi>w</mi><mrow><mi>j</mi><mo>,</mo><mi>k</mi></mrow></msub><msub><mi>o</mi><mrow><mi>j</mi><mo>,</mo><mi>k</mi></mrow></msub><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup><mo>+</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><munderover><mo>&amp;Sigma;</mo><mrow><mi>l</mi><mo>=</mo><mn>1</mn></mrow><mn>128</mn></munderover><msub><mi>u</mi><mrow><mi>i</mi><mo>,</mo><mi>l</mi></mrow></msub><mo>|</mo><mo>|</mo><msub><mi>d</mi><mi>l</mi></msub><mrow><mo>(</mo><msub><mi>x</mi><mi>i</mi></msub><mo>)</mo></mrow><mo>-</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><msub><mi>w</mi><mrow><mi>i</mi><mo>,</mo><mi>k</mi></mrow></msub><msub><mi>d</mi><mi>l</mi></msub><mrow><mo>(</mo><msub><mi>x</mi><mrow><mi>i</mi><mo>,</mo><mi>k</mi></mrow></msub><mo>)</mo></mrow><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup><mo>+</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mo>|</mo><mo>|</mo><msub><mi>u</mi><mrow><mi>i</mi><mo>,</mo><mi>l</mi></mrow></msub><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup></mrow> Among them, min represents the minimum value operation, ∑ represents the sum operation, || || ² represents the modulo square operation, w _i,k represents the coefficient of the kth neighbor training image block of the i-th test photo block, o _{i , k} represents the pixel value vector of the overlapping part of the kth neighbor training image block of the i-th test photo block, w _{j, k} represents the coefficient of the j-th neighbor training image block of the j-th test photo block, o _{j, k} represents the pixel value vector of the overlapping part of the kth neighbor training image block of the jth test photo block, u _i,l represents the coefficient of the depth feature map of the lth layer of the depth feature of the ith test photo block, d _l (x _i ) represents the h-th layer feature map of the depth feature of the i-th test photo block, d _l ( _xi,k ) represents the u-th depth feature map of the k-th neighbor training photo block of the i-th test photo block The layer feature map, the values of l, h, and u are correspondingly equal. 6. the two-stage human face portrait generating method of joint local and global characteristics according to claim 1, is characterized in that: the method for the reconstructed portrait block corresponding to all test photo blocks of splicing described in step (6) is as follows: The first step is to place the reconstructed portrait blocks corresponding to all test photo blocks located in different positions of the portrait according to their positions; The second step is to get the average value of the pixel values of the overlapping parts between adjacent two reconstructed face portrait blocks; The third step is to replace the pixel values of the overlapping parts between two adjacent reconstructed face portrait blocks with the average value of the pixel values of the overlapping parts between two adjacent reconstructed face portrait blocks to obtain a synthetic face portrait.