CN106951840A - A kind of facial feature points detection method - Google Patents

Abstract

The present invention discloses a kind of facial feature points detection method, using attitude detection task as constraint, method of novel three-way road GEH (the Gray Edge Hog) mode images merged using multiclass feature figure as the facial feature points detection of input.Detection in view of face 3 d pose information to face global characteristic point, is especially detected in the case where attitude deflection is larger to features of human face images, with considerable influence；The Hog characteristic informations of reflection facial image local feature presentation and shape are added simultaneously and can effectively reduce the complexity of contour feature point detection for the edge image information of the Sobel operator extractions of rim detection, the present invention is by extracting image intensity value, marginal information and Hog features generate new GEH triple channels image and are used as input, the nonproductive task estimated simultaneously using 3 d pose is used as constraint information, progress facial feature points detection.

Description

Face characteristic point detection method

Technical Field

The invention belongs to the field of computer vision, and particularly relates to a human face characteristic point detection method taking human face three-dimensional posture information as auxiliary constraint in a novel image mode, which has important application in human face recognition, human face posture expression analysis and human face synthesis.

Background

In recent years, with the development of deep learning, Convolutional Neural Networks (CNNs) have achieved a good effect in detecting facial feature points. The CNN takes a face original image as input, and features obtained by using a local receptive field strategy have better expression capability; the weight sharing structure reduces the number of weights so as to reduce the complexity of a network model; meanwhile, downsampling the feature map by using the image local correlation principle effectively reduces the data processing amount while retaining useful structural information, so the CNN is widely applied to feature extraction of face images.

Yi Sun et al, 2013, proposed a three-level Deep Convolutional neural Network cascaded face feature point detection model (Deep Convolutional Network Cascade, DCNN). The first stage of the network takes three different regions (all face regions, eye and nose regions, nose and lip regions) of a face image as input, respectively trains three convolutional neural networks to predict the positions of feature points, and fuses the predicted values of the three networks to obtain a more stable primary feature point detection result. And secondly, extracting features near each feature point in the third stage, and training a convolution neural network for each feature point to correct the positioning result, so as to realize the detection of five feature points including the left eye center, the right eye center, the nose tip, the left mouth corner and the right mouth corner. However, the result obtained by the method only roughly calibrates the positions of eyes, nose and mouth, and the facial attributes cannot be well expressed, and the network model is too complex.

The same year Erijin Zhou et al proposed a four-stage convolutional neural network cascade model for detecting 68 facial feature points of a person. The model considers that the complexity of the positioning of the external outline of the human face and the internal feature points of the five sense organs is different, and the detection is respectively carried out. The multi-feature point detection can show attributes such as the pose and the expression of the face in more detail. However, the method is complex in operation process, and 10 different networks are involved to be trained respectively.

Zhanpen et al proposed a Deep Convolutional neural Network (TCDCN) Constrained by the auxiliary task in 2015. The model enhances the capability of extracting features by a network by taking 18 auxiliary tasks related to facial attributes as constraints while performing 5-person face feature point detection, and is beneficial to improving the accuracy of feature point detection. However, the method only considers the situation that the human face pose deflects in the horizontal dimension, and in most cases, the situation that the pose deflects in other dimensions has certain influence on the feature point detection accuracy.

Disclosure of Invention

The invention provides a method for detecting human face characteristic points by taking a novel three-channel GEH (Gray-Edge-Hog) mode image fused with multiple types of characteristic graphs as input and a three-dimensional posture auxiliary task as constraint information. The invention considers that the human face can obviously see the change of the image outline structure when the posture is changed, so that the human face three-dimensional posture information has considerable influence on the calibration of the global characteristic points of the human face; meanwhile, because the detection difficulty of the external contour feature points of the human face is different from that of the internal organ feature points, the edge information extracted from the human face is used as an image mode variable, so that the detection difficulty of the external contour points can be reduced; the Hog feature map extracted from the face image is used as an image mode variable, the image contour structure is clearly reflected, and meanwhile, the regional features of each organ of the face are more effectively highlighted, so that a convolutional neural network structure model which takes a novel GEH mode image formed by fusing an image gray value, edge information and the Hog feature as input and facial three-dimensional posture information as auxiliary constraint and jointly trains facial feature points and facial postures is provided, and 68 feature points of the face are accurately positioned.

In order to achieve the purpose, the invention adopts the following technical scheme:

a face characteristic point detection method comprises the following steps:

step (ii) of1. Carrying out face detection positioning and cutting and three-channel multi-feature image fusion on the original image to obtain a three-channel GEH mode image Picture_GEH；

Step 2, taking a three-channel GEH mode image obtained by fusing the three feature images as the input of a convolutional neural network, and extracting the facial features of the network, wherein the facial features comprise: the method comprises the following steps that characteristic points and three-dimensional postures of a human face are detected, and a double-task loss function is designed according to a least square function corresponding to a linear regression problem;

and 3, performing network training on the double-task loss function by adopting a gradient back propagation algorithm, finally learning the detection weight of the human face characteristic points and the detection weight of the posture, and extracting a network through the same human face characteristics in the test process so as to realize the detection of the human face characteristic points and the detection of the three-dimensional posture of the human face.

Preferably, the network feature extraction is alternately completed by 3 convolutional layers and 3 pooling layers, and 2 full-connection layers;

firstly, a three-channel GEH mode diagram Picture_GEHAs input to a first layer convolution operationOutput feature map y_jThe calculation formula of (a) is shown as the following formula:

where f denotes convolution operation, l denotes the current number of network layers, i denotes the number of input profiles, j denotes the number of output profiles, w_ijFor the convolution kernel parameter to be solved, b_jIs a bias parameter, w_ijAnd b_jAcquiring by adopting a random normal initialization mode at the beginning of an experiment;

then, according to the result obtained in the convolution stage, the characteristics are sent to the function corresponding to the linear regression problem, and the designed dual-task loss function expression is shown as the following formula:

wherein N represents the number of training images,a tag value indicating the ith image feature point detection task,respectively representing label values, x corresponding to the detection of the three-dimensional posture of the human face_iFeatures of the ith image, W, representing convolutional neural network extraction^fRepresenting the weight of the feature point detection task, W^yaw,W^pitch,W^rollRespectively representing weight lambda corresponding to three-pose detection task of human face_Yaw,λ_Pitch,λ_RollRepresenting a loss function loss weight;

performing network training through a back propagation algorithm to obtain the face characteristic point detection weight W^fAnd attitude detection weight W^yaw,W^pitch,W^roll(ii) a In the testing process, the same face feature extraction network is used to finally obtain the face feature point detection result (W)^f)^Tx_iAnd three-dimensional attitude detection result (W)^Yaw)^Tx_i,(W^Pitch)^Tx_i,(W^Roll)^Tx_i。

Preferably, in step 1, performing gray processing on the face subgraph subjected to face detection positioning and cutting to obtain a gray feature graph G; extracting the Hog features from the face subgraph to obtain a Hog feature graph H; finally, extracting edge features to obtain an edge feature graph E; using RGB (Red-Green-Blue) color space as substrate, respectively mapping the feature map G (Gray), the feature map E (edge), and the feature map H (Hog) onto RGB rectangular coordinate system color space to generate novel GEH modeImage Picture_GEHThe generation formula is as follows:

wherein,the Gray values representing the Gray feature map Gray are mapped to the r (red) color space in RGB,the representative Hog feature map feature values are mapped to the b (blue) color space,the feature values representing the Edge feature map are mapped to the g (green) color space.

Drawings

FIG. 1: the three-channel characteristic diagram generation process schematic diagram is disclosed;

FIG. 2 a: an original face image;

FIG. 2 b: the human face image is fused by multiple characteristic images;

FIG. 3: GEH-double-task convolution neural network structure model.

Detailed Description

The invention provides a human face characteristic point detection method, which takes a posture detection task as constraint and takes a novel three-channel GEH (Gray-Edge-Hog) mode image fused by multiple types of characteristic graphs as an input human face characteristic point detection method. The detection of the human face global feature points by considering the human face three-dimensional posture information has a considerable influence on the detection of the human face image feature points especially under the condition of large posture deflection; the invention can effectively reduce the complexity of contour feature point detection by adding the Hog feature information reflecting the local feature representation and shape of the face image and the edge image information extracted by the Sobel operator for edge detection.

The basic data used by the invention is from a 300-W face characteristic point detection competition platform, wherein four data sets of LFPW, AFW, HELEN and IBUG are included. The image labels in each dataset are 68 points, including the outer contour points and the inner five sense organs (eyebrows, eyes, nose, mouth). The corresponding three-dimensional attitude tags are generated by Interface software developed by a human perception laboratory and calculated according to feature point tags provided by a 300-W image, and represent the Yaw, Pitch and Roll three-dimensional information respectively.

The CNN structure for detecting the human face characteristic points comprises two tasks, namely a 68-point human face characteristic point detection task and a three-dimensional posture information detection task. By utilizing the related influence of the three-dimensional posture information on the global face characteristic point detection, the combined characteristics which can represent the position of the face characteristic point and reflect the face posture orientation are extracted through the CNN face characteristic point detection network, and finally the detection of the face characteristic point is realized.

1. Image pre-processing

The proper image preprocessing method can eliminate the environmental influences of weather, illumination and the like in the original image, so that the edge and color features of the image are more prominent, and the feature extraction of the convolutional neural network is facilitated. The method comprises the steps of firstly carrying out face detection positioning and cutting on an original image, carrying out normalization processing, then respectively carrying out graying processing, Hog feature extraction and visualization and Sobel operator edge feature extraction on the cut image, fusing the generated multi-class feature maps to form a novel GEH image mode, and taking a three-channel image in the mode as input.

1.1 face detection positioning and clipping

The invention aims to remove the interference of information such as image background, hair and clothes on a characteristic point detection task. According to the result of face detection and positioning, the distances between the detected image and the face image are expanded in a certain proportion, and then the expanded face image is cut and normalized in scale.

1.2 three-channel multiple feature map fusion

Firstly, carrying out gray level processing on the face subgraph obtained in the step 1.1 to obtain a gray level feature graph G; extracting the Hog Features from the face subgraph, wherein the Hog Features are extracted, and because the dimensions of the feature graph after the Hog Features are extracted are different from those of the face subgraph, an investing Visual Features method proposed by CarlVondrick in 2012 is adopted for carrying out feature visualization processing, and finally, a Hog feature graph H with the same size as the face subgraph is obtained; and finally, extracting edge features, and performing edge extraction on the RGB face sub-image subjected to scale normalization by using a 5-order Sobel operator to generate an edge feature image E.

Because the three characteristic images reflect the information of different attributes of the face image and are independent of each other, a new image mode can be formed by the comprehensive action of the three independent variables. Therefore, the invention uses RGB (Red-Green-Blue) color space as a substrate, and respectively maps the G (Gray), E (edge) and H (Hog) feature map variables to RGB rectangular coordinate system color space to generate a novel GEH mode image Picture_GEH. The generation formula is as follows:

wherein,the Gray values representing the Gray feature map Gray are mapped to the r (red) color space in RGB,the representative Hog feature map feature values are mapped to the b (blue) color space,the feature values representing the Edge feature map are mapped to the g (green) color space.

The GEH three channel image generation process is shown in fig. 1.

The GEH three-channel image effect is shown in fig. 2a and 2 b.

2. Face characteristic point detection and three-dimensional posture detection double-task network model

The CNN network structure jointly considers a face image characteristic point detection task and a three-dimensional posture detection task, wherein the face characteristic point detection task is a main task, the three-dimensional posture detection task is an auxiliary task, and the network structure is shown in figure 3.

2.1 face feature extraction

Three-channel GEH mode Picture fused by three characteristic pictures in network_GEHAnd extracting the human face features as the input of the convolutional neural network. The feature point detection and the gesture detection task both use a least square function corresponding to a linear regression problem as a loss function, adopt a gradient back propagation algorithm, train network parameters, and finally realize the detection of the feature points of the human face and the detection of the three-dimensional gesture of the human face.

In the invention, the network feature extraction is alternately completed by 3 convolution layers, 3 pooling layers and 2 full-connection layers. The convolutional layer performs convolution operation through local receptive fields to extract visual features. In order to more completely reserve the image characteristics after the original image is converted into the GEH image mode, the invention takes the size close to the face subgraph of the original image as the input size, and the input of the convolution operation of the first layer(i.e., Picture)_GEH) The image with the size of 224 × 224 is 2.3 times of the input sizes of DCNN and TCDCN, so that the integrity and the feature effectiveness of the image are ensured;the convolution kernel sizes are 7 × 7, 4 × 4 and 3 × 3 respectively, and the output characteristic diagram y_jThe formula (2) is shown as follows:

where f denotes convolution operation, l denotes the current number of network layers, i denotes the number of input profiles, j denotes the number of output profiles, w_ijFor the convolution kernel parameter to be solved, b_jIs a bias parameter, w_ijAnd b_jAnd acquiring by adopting a random normal initialization mode at the beginning of an experiment.

The pooling layer adopts a maximum pooling method, aiming at the characteristics of the input size of the GEH mode image, the pooling ranges of the first layer and the third layer are set to be 3 multiplied by 3, the step length is 3, the completeness of the extracted features is ensured, the feature dimension is effectively reduced, and the complexity of network training is reduced; the pooling range of the second layer was 2 x 2 with a step size of 2.

2.2 design of the Dual-tasking Objective function

And then, according to the result obtained in the convolution stage, the characteristics are sent to a function corresponding to the linear regression problem. The detection problem of the human face feature points and the detection problem of the human face three-dimensional posture are both linear regression problems, a least square function is adopted as a loss function, and the expression is shown as a formula (3):

f_loss＝||l-W^Tx_i|| (3)

wherein l represents a label of the regression problem, x_iRepresents the features extracted by the convolutional neural network, and W represents the weight coefficient corresponding to the linear regression problem.

The invention takes the detection of the human face characteristic points as a main task. The three-dimensional posture detection is an auxiliary task and is used for assisting the human face characteristic point detection task to extract characteristics capable of reflecting the three-dimensional posture more and accurately positioning the human face image with a larger posture; the three-dimensional attitude coordinates are respectively expressed by Pitch, Yaw and Roll, taking the right-hand cartesian coordinate system as an example in a three-dimensional rectangular coordinate system, Pitch representing rotation about the X axis, called Pitch angle, Yaw representing rotation about the Y axis, called Yaw angle, and Roll representing rotation about the Z axis, called Roll angle. In the three-dimensional posture detection task, according to statistics of experimental data, the variation amplitude of the Yaw posture in the three-dimensional posture is larger and is 5-6 times of the variation of Pitch posture and Roll posture, so that the influence of the Yaw is larger than that of Pitch and Roll, and the influence of the three-dimensional posture on the main task of face detection is set by setting the weight of a loss function. The expression of the dual task loss function designed by the invention is shown as a formula (4):

wherein N represents the number of training images,a label value (dimension 136) representing the ith image feature point detection task,respectively representing label values (the dimensionalities are all 1) corresponding to the detection of the three-dimensional posture of the human face, and x_iFeatures of the ith image, W, representing convolutional neural network extraction^fRepresenting the weight of the feature point detection task, W^yaw,W^pitch,W^rollAnd respectively representing weights corresponding to the three-pose detection tasks of the human face. Lambda [ alpha ]_Yaw,λ_Pitch,λ_RollAnd the loss weights of the loss functions are respectively 0.3, 0.1 and 0.1.

2.3 network learning

The invention adopts a back propagation algorithm to carry out network training. Finally learning the face characteristic point detection weight W^fAnd attitude detection weight W^yaw,W^pitch,W^roll. In the testing process, the same face feature extraction network is used to finally obtain the face feature pointsDetection result (W)^f)^Tx_iAnd three-dimensional attitude detection result (W)^Yaw)^Tx_i,(W^Pitch)^Tx_i,(W^Roll)^Tx_i。

The method is experimentally verified, and an obvious effect is achieved. The evaluation index measures the performance of the designed method by adopting an average estimation error index which is published on CVPR by Yi Sun et al in 2013 and is provided for detecting the characteristic points of the human face, and the index shows the accuracy and reliability of a characteristic point positioning algorithm.

The average estimation error calculation formula is as follows:

wherein, (x, y) and (x ', y') denote the feature point true value coordinates and the estimated coordinates, respectively, and l denotes the estimation error normalization factor. If the estimation error exceeds 10%, the estimation is considered to be invalid.

The experiment used a 300-W challenge platform of LFPW database, which is a multi-pose, multi-view database of faces. Most human face feature point detection methods are validated on the data set because the picture includes various aspects of pose, expression, illumination, etc. The data set contains 811 training images and 224 test images.

The first set of experiments was mainly performed: the performance of a double-task convolutional neural network (3 feature-D-CNN) taking a multi-feature fusion image as input, a double-task convolutional neural network (D-CNN) taking an original image as input and a traditional Convolutional Neural Network (CNN) taking the original image as input are compared with the performance of human face feature point detection. The structure of the convolutional layer, the pooling layer and the full-connection layer in the three networks are the same. The difference lies in the input image type, the output loss function and the output dimension of the network structure. The experimental comparison results are as follows:

table 1: comparison of three convolutional neural network models

Table 1 the columns represent the results of testing different network models on LFPW data. Lower values indicate better detection of the human face feature points. It can be seen that the average estimation error of the 3 feature-D-CNN network provided by the invention is respectively reduced by 14.02% and 11.6% compared with the original CNN network and D-CNN network. This shows that the method for detecting human face feature points is effective, wherein the method takes the posture detection task as constraint and takes the novel three-channel GEH (Gray-Edge-Hog) mode image fused by multiple types of feature maps as input.

The second set of experiments was mainly performed: the effects of the three networks on the detection of the outer contour points are compared. The results of the experiment are shown in table 2:

table 2: comparison of detection results of external contours of human faces detected by three network models

Table 2 each column of data represents the results of contour point detection on LFPW data for different network models. Lower values indicate better detection of the human face feature points. It can be seen that the 3 feature-D-CNN network provided by the invention has the average error respectively reduced by 21.52% and 4.95% in the aspect of external contour detection compared with the original CNN network and the D-CNN network. The detection effect of the invention on the contour points outside the human face is verified to be improved to a certain extent.

Claims (3)

1. A face feature point detection method is characterized by comprising the following steps:

step 1, carrying out face detection positioning and cutting and three-channel multi-feature image fusion on an original face image to obtain a three-channel GEH mode image Picture_GEH；

Step 2, taking a three-channel GEH mode image obtained by fusing the three feature images as the input of a convolutional neural network, and extracting the facial features of the network, wherein the facial features comprise: the method comprises the following steps that characteristic points and three-dimensional postures of a human face are detected, and a double-task loss function is designed according to a least square function corresponding to a linear regression problem;

and 3, performing network training on the double-task loss function by adopting a gradient back propagation algorithm, finally learning the detection weight of the human face characteristic points and the detection weight of the posture, and extracting a network through the same human face characteristics in the test process so as to realize the detection of the human face characteristic points and the detection of the three-dimensional posture of the human face.

2. The method of claim 1, wherein the network feature extraction is performed by alternating 3 convolutional layers and 3 pooling layers, with 2 fully-connected layers;

firstly, a three-channel GEH mode diagram Picture_GEHAs input to a first layer convolution operationOutput feature map y_jThe calculation formula of (a) is shown as the following formula:

$x_j^l=f(b_j+{\mathrm{\Σ}}_i{w}_{ij}*x_i^{l-1})$

where f denotes convolution operation, l denotes the current number of network layers, i denotes the number of input profiles, j denotes the number of output profiles, w_ijFor the convolution kernel parameter to be solved, b_jIs a bias parameter, w_ijAnd b_jAcquiring by adopting a random normal initialization mode at the beginning of an experiment;

then, according to the result obtained in the convolution stage, the characteristics are sent to the function corresponding to the linear regression problem, and the designed dual-task loss function expression is shown as the following formula:

$\begin{array}{c}\underset{W^f,W^{Yaw},W^{Pitch},W^{Roll}}{\arg \min }\left\{\underset{i=1}{\overset{N}{\mathrm{\Σ}}}\right||l_i^f-{\left(W^f\right)}^T{x}_i|{|}^2+{\mathrm{\λ}}_{Yaw}\underset{i=1}{\overset{N}{\mathrm{\Σ}}}\left|\right|l_i^{Yaw}-{\left(W^{Yaw}\right)}^T{x}_i|{|}^2\\ +{\mathrm{\λ}}_{Pitch}\underset{i=1}{\overset{N}{\mathrm{\Σ}}}\left|\right|l_i^{Pitch}-{\left(W^{Pitch}\right)}^T{x}_i|{|}^2+{\mathrm{\λ}}_{Roll}\underset{i=1}{\overset{N}{\mathrm{\Σ}}}||l_i^{Roll}-{\left(W^{Roll}\right)}^T{x}_i|{|}^2\}\end{array}$

wherein N represents the number of training images,a tag value indicating the ith image feature point detection task,respectively representing label values, x corresponding to the detection of the three-dimensional posture of the human face_iFeatures of the ith image, W, representing convolutional neural network extraction^fRepresenting the weight of the feature point detection task, W^yaw,W^pitch,W^rollRespectively representing weight lambda corresponding to three-pose detection task of human face_Yaw,λ_Pitch,λ_RollRepresenting a loss function loss weight;

performing network training through a back propagation algorithm to obtain the face characteristic point detection weight W^fAnd attitude detection weight W^yaw,W^pitch,W^roll(ii) a In the testing process, the same face feature extraction network is used to finally obtain the face feature point detection result (W)^f)^Tx_iAnd three-dimensional attitude detection result (W)^Yaw)^Tx_i,(W^Pitch)^Tx_i,(W^Roll)^Tx_i。

3. The method for detecting human face feature points as claimed in claim 1, wherein in step 1, the human face sub-image after human face detection positioning and clipping is processed with gray scale to obtain a gray scale feature image G; extracting the Hog features from the face subgraph to obtain a Hog feature graph H; finally, extracting edge features to obtain an edge feature graph E; respectively mapping the feature map variables of the feature map G (Gray), the feature map E (edge) and the feature map H (hog) onto the color space of an RGB rectangular coordinate system by using an RGB (Red-Green-Blue) color space as a substrate to generate a novel GEH mode image Picture_GEHThe generation formula is as follows:

wherein,the Gray values representing the Gray feature map Gray are mapped to the r (red) color space in RGB,the representative Hog feature map feature values are mapped to the b (blue) color space,the feature values representing the Edge feature map are mapped to the g (green) color space.