CN112270308A - A facial feature point localization method based on two-layer cascaded regression model - Google Patents

Abstract

The invention discloses a face feature point positioning method based on a double-layer cascade regression model, which is applied to face feature point positioning, and improves the accuracy of face feature point positioning to a certain extent, wherein the first layer of the model is used for positioning a simplified face shape containing part of key feature points, in order to enhance the robustness of the face feature point positioning method, a fusion subspace is used for dividing samples, and each part of the samples train a feature extraction model and a regressor independently; the second layer is used for positioning the complete face shape, 3D fitting is carried out on the regression result of the first layer model to obtain the roughly aligned complete face shape, and finer regression is realized on the basis of the shape. Experiments prove that compared with a single-layer cascade regression model, the model provided by the invention has 23.55% improved performance on a 300-W challenge set with large attitude change and has certain attitude robustness.

Description

A facial feature point localization method based on two-layer cascaded regression model

technical field

The invention relates to the technical field of computer image processing technology, in particular to a method for locating facial feature points based on a double-layer cascaded regression model.

Background technique

The goal of facial feature point location is to locate more specific facial shapes, such as eyebrows, eyes, nose, mouth, and contours, based on the completion of face detection. Facial feature point location is an important part of face image processing tasks, and plays a vital role in subsequent face recognition, facial expression analysis, and 3D face reconstruction. Most of the current facial feature point localization algorithms can achieve satisfactory results on frontal face images, but for unconstrained face images with huge changes in expressions and postures, it is still difficult to achieve high-precision facial feature point localization. challenge.

Algorithms related to facial feature point localization can be divided into two categories: (1) Methods based on generative models. The representative algorithm is the active appearance model (AAM) proposed by Cootes et al. First, the principal component analysis (PCA) is used to construct a parameter model in the shape and texture feature space, and then the parameters are optimized to realize the matching between the model and the face image. . While various improvements have been made to this type of approach, parametric models are ultimately limited in their expressive power to handle subtle shape changes. (2) Methods based on discriminant models. The goal of such methods is to map directly from the extracted image features to facial feature point coordinates. The cascade regression model is currently the most widely used model in the field of facial feature point location, and it first appeared in the cascade pose regression (CPR) method proposed by Dollár et al. Algorithms based on cascade regression regard facial feature point localization as solving the nonlinear optimization problem between image texture features and face shape, and gradually update the initial shape to the final shape by learning a series of feature-to-shape mappings. In addition to cascaded regression models, methods based on deep networks have also received extensive attention in recent years. The earliest application of deep network to facial feature point location is the facial feature point location algorithm based on deep convolutional neural network (DCNN) proposed by Sun et al. In recent years, algorithms for locating facial feature points using 3D face models have also appeared. Such methods fit the model to the face image by adjusting the parameters, and then project the fitting result to a 2D plane to obtain the localization result. Due to the large number of parameters in the 3D face model, most of these algorithms also require the help of deep networks. Although the addition of the deep network has greatly improved the performance of the facial feature point localization algorithm, there are still many excellent algorithms based on cascaded regression emerging. On the basis of relatively low training cost, in some cases It also has performance that surpasses deep learning-based algorithms.

Shape initialization is the first step of the face feature point localization algorithm based on cascade regression. Many traditional cascade regression algorithms use the average value of the true shape of all samples in the training set, or a positive neutral face shape as the initial shape, and then based on the initial shape, iteratively updated to reach the final prediction result. When using this initial shape, it is usually difficult for faces with large changes in pose and expression to achieve accurate positioning. This is because when the initial shape differs greatly from the real shape, the cascaded regression model may be limited due to the limited number of iterations, or Stuck in a local optimum, resulting in an unsatisfactory final result. Xiong et al. proposed to divide the samples according to domains of homogeneous descent (DHD), and then build regression models separately to avoid regression from entering the local optimum. Real face shape, which was not realistic during testing. Coarse-to-fine shape searching (CFSS) explores the true shape of all samples as a solution space, which solves the limitation of the initial shape, but also reduces the computational speed.

SUMMARY OF THE INVENTION

The purpose of this section is to outline some aspects of embodiments of the invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the application to avoid obscuring the purpose of this section, abstract and title, and such simplifications or omissions may not be used to limit the scope of the invention.

The present invention has been proposed in view of the above-mentioned existing problems.

Therefore, the technical problems solved by the present invention are: the problems existing in the process of locating the facial feature points in the unconstrained face image pose a large change, and the initial shape limits the regression result.

In order to solve the above-mentioned technical problems, the present invention provides the following technical solutions: including, expanding the sample by randomly selecting the face shape from the training sample set as the initial shape;

From the complete face shape of 68 points, 14 points that play a key role in the positioning of face feature points are selected to form a simplified face shape;

Using the subspace division method, the samples are divided into multiple subsets, and each subset trains a regressor, and all the regressors together form the first-level cascaded regression model, which is used to predict the simplified face shape;

Using 3D fitting, realize the projection from the simplified face shape predicted in the first layer to the complete face shape, and generate a rough initial complete face shape;

The rough and complete face shape is used as the initial shape, and all training samples are used to train the regressor to form a second-level cascaded regression model to form a two-layer cascaded regression model for predicting the complete face shape.

As a preferred solution of the method for locating facial feature points based on the double-layer cascaded regression model of the present invention, wherein: the step of constructing the double-layered cascaded regression model includes:

First, extract the shape index feature based on PHOG, use the projection matrix P obtained in the training process to project it into the fusion subspace, and divide K sample subsets according to it;

和回归器

经过T次迭代回归得到简化人脸形状

Then, the feature mapping function is trained separately according to different subsets

and regressor

After T iterations of regression, the simplified face shape is obtained

Then use the 3D fitting method to obtain affine transformation parameters, and project to obtain a rough complete face shape s ⁰ ;

Finally, according to all the training samples and the rough and complete face shape obtained in the previous step, the feature extraction function Φ and the regressor W are trained, and refined regression is achieved after T iterations.

As a preferred solution of the method for locating facial feature points based on the double-layer cascaded regression model of the present invention, wherein: the step of selecting the optimal simplified face shape includes:

Based on the most representative 5 contour feature points, add some feature points to the eyes, nose, mouth and contour respectively, and select 6 simplified face shapes;

The five most representative contour feature points include the center of the eyes, the tip of the nose and the corner of the mouth.

As a preferred solution of the method for locating facial feature points based on the double-layer cascaded regression model of the present invention, wherein: the step of selecting the optimal simplified face shape further includes:

Pick the most suitable simplified face shape, including calculating the positioning error and fitting error;

The positioning error and the fitting error are compared, and the 14-point shape with the minimum positioning error and the fitting error is selected as the simplified face shape.

As a preferred solution of the method for locating facial feature points based on a double-layer cascaded regression model according to the present invention, wherein: the method for calculating the positioning error is to train a single-layer cascaded regression based on each simplified face shape. Model, calculate the normalized standard error of each model on the test set samples;

The fitting error calculation method is, based on the positioning result obtained by the positioning error and the complete face shape generated by the 3D fitting method, calculate the average normalized error between the generated complete face shape and the real face shape of the sample. .

As a preferred solution of the method for locating facial feature points based on the double-layer cascaded regression model of the present invention, wherein: the fusion subspace includes:

Among them, r is the column dimension of the fusion subspace, and z _i,j represents the projection result of the shape index feature of the ith sample on the jth dimensional subspace.

As a preferred solution of the method for locating facial feature points based on the double-layer cascaded regression model of the present invention, wherein: the step of using the fusion subspace to divide the sample into K sample subsets includes:

Define the face images in the training set and the corresponding set of real shapes,

分别为第i个人脸图像和对应的真实形状；where N is the number of samples, I _i and

are the ith face image and the corresponding real shape, respectively;

Match the average simplified face shape to the bounding box of all training set samples, and the matching result is defined as

compute the shape residuals,

and PHOG-based shape index features

Use canonical correlation analysis to analyze the shape residual and shape index features, and obtain their corresponding projection matrices P and Q;

The subsets are divided according to the following formula,

As a preferred solution of the method for locating facial feature points based on the double-layer cascaded regression model of the present invention, wherein: if the i-th and g-th samples have the same sign in each dimension in the fusion subspace, Then the two samples belong to the same subset U _k .

As a preferred solution of the method for locating facial feature points based on the double-layer cascaded regression model of the present invention, wherein: the step of using the 3D fitting method to generate an initial shape for the second-layer model includes:

和3D完整人脸形状

Using the 3DMM model composed of dense 3D coordinate point sets, 14 coordinate point sets and 68 coordinate point sets are extracted respectively, which are respectively recorded as 3D simplified face shape

and 3D full face shape

Use weak perspective projection to project the 3D simplified face shape to the 2D plane, and fit the s _3d to the face image, including:

R(α,β,γ)为一个由俯仰角α，偏航角β和翻滚角γ构成的3×3的旋转矩阵，t_3d为位移向量。

为

在2D平面上的投影；Among them, f is the scaling factor, P _o is the orthographic projection matrix

R(α,β,γ) is a 3×3 rotation matrix composed of pitch angle α, yaw angle β and roll angle γ, and t _3d is the displacement vector.

for

Projection on a 2D plane;

通过最小化

与

之间的欧氏距离，则确定参数f，R和t_3d的合适值，实现人脸图像与3D人脸形状的拟合，包括：If the 2D simplified face shape output by the first-level cascaded regression model is recorded as

by minimizing

and

The Euclidean distance between, then determine the appropriate values of the parameters f, R and t _3d to realize the fitting of the face image and the 3D face shape, including:

替换成3D完整人脸形状

即可得到与人脸图像拟合后的3D完整人脸形状在2D平面上的投影

After calculating the parameters of the radiation transformation by the least squares method, the 3D simplified face shape in the above weak perspective projection formula

Replaced with 3D full face shape

The projection of the 3D complete face shape after fitting with the face image on the 2D plane can be obtained.

When the face pose changes, the feature points of the contour parts marked on the 3D face shape will be occluded by the cheeks, while the real contour feature points should be at the border of the cheeks.

Use the yaw angle β obtained in the above fitting process to determine whether the face is biased to the left or the right.

As a preferred solution of the method for locating facial feature points based on the double-layer cascaded regression model of the present invention, wherein: the step of using the 3D fitting method to generate an initial shape for the second-layer model further includes:

When the face is biased to the left, the 8 contour feature points on the left search for the minimum x-coordinate point in its corresponding coordinate point subset, and use it as a new contour feature point;

When the face is biased to the right, the 8 contour feature points on the right side search for the maximum x-coordinate point in the subset of corresponding coordinate points, and mark it as a new contour feature point;

The eight contour feature points are three contour feature points added on the basis of the five contour feature points.

Beneficial effects of the present invention: The present invention provides a method for locating facial feature points based on a double-layer cascaded regression model. The pose robustness of the facial feature point localization method is improved, which in turn improves the overall localization accuracy.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort. in:

1 is a schematic flowchart of a method for locating facial feature points based on a double-layer cascaded regression model;

2 is a schematic diagram of the results of the positioning errors and fitting errors of different simplified face shapes based on the facial feature point positioning method based on the double-layer cascaded regression model;

Fig. 3 is the face image distribution diagram in the two-dimensional subspace of the facial feature point location method based on the double-layer cascaded regression model;

4 is a schematic diagram showing the influence of the K value of the facial feature point localization method based on the double-layer cascaded regression model on the localization effect of the simplified face shape on different data sets;

Fig. 5 is a schematic diagram showing the influence of two subspace division methods of the facial feature point localization method based on the double-layer cascaded regression model on the localization effect of the simplified face shape on the 300-W data set;

6 is a schematic diagram showing the influence of the K value of the facial feature point localization method based on the double-layer cascaded regression model on the localization effect of the complete face shape on different data sets;

7 is a schematic diagram of the CED curve of each algorithm based on the double-layer cascaded regression model-based facial feature point location method on the 300-W ensemble;

8 is a schematic diagram of the localization results of different algorithms in the prior art based on a double-layer cascaded regression model-based facial feature point localization method on a 300-W test set;

Fig. 9 is a partial localization result of the algorithm of the facial feature point localization method based on the double-layer cascaded regression model on the 300-W test set.

Detailed ways

In order to make the above objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are a part of the embodiments of the present invention, not all of them. Example. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and those skilled in the art can do it without departing from the connotation of the present invention. Similar promotion, therefore, the present invention is not limited by the specific embodiments disclosed below.

Second, reference herein to "one embodiment" or "an embodiment" refers to a particular feature, structure, or characteristic that may be included in at least one implementation of the present invention. The appearances of "in one embodiment" in various places in this specification are not all referring to the same embodiment, nor are they separate or selectively mutually exclusive from other embodiments.

The present invention is described in detail with reference to the schematic diagrams. When describing the embodiments of the present invention in detail, for the convenience of explanation, the cross-sectional views showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the present invention. scope of protection. In addition, the three-dimensional spatial dimensions of length, width and depth should be included in the actual production.

At the same time, in the description of the present invention, it should be noted that the orientation or positional relationship indicated in terms such as "upper, lower, inner and outer" is based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention. The invention and simplified description do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operate in a particular orientation, and therefore should not be construed as limiting the invention. Furthermore, the terms "first, second or third" are used for descriptive purposes only and should not be construed to indicate or imply relative importance.

Unless otherwise expressly specified and limited in the present invention, the term "installation, connection, connection" should be understood in a broad sense, for example: it may be a fixed connection, a detachable connection or an integral connection; it may also be a mechanical connection, an electrical connection or a direct connection. The connection can also be indirectly connected through an intermediate medium, or it can be the internal communication between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in specific situations.

Example 1

Referring to Fig. 1, it is a first embodiment of the present invention, which provides a schematic flowchart of a method for locating facial feature points based on a double-layer cascaded regression model. As shown in Fig. 1, a method based on a double-layer cascaded regression model The facial feature point localization method includes: amplifying the sample by randomly selecting the face shape from the training sample set as the initial shape; selecting from the complete face shape of 68 points plays a key role in the localization of facial feature points The 14 points are composed of simplified face shape; a two-layer cascaded regression model is constructed, the first layer is used to predict the simplified face shape, and the second layer is used to predict the complete face shape; in order to improve the pose robustness, in the first In the first-level cascaded regression model, the subspace division method is used to divide the samples into multiple subsets, and each subset trains the regressor separately; the 3D fitting method is used between the two layers to realize the transformation from simplified face shape to complete face shape. Projection to generate a rough initial full face shape; the first layer model uses the neutral simplified face shape as the initial shape, and the second layer model uses the result of 3D fitting as the initial shape.

其中N为样本数量，I_i和

分别为第i个人脸图像和对应的真实形状。首先将平均简化人脸形状与所有训练集样本的边界框匹配，匹配结果定义为

接着计算形状残差

以及基于PHOG的形状索引特征

利用典型相关分析(CCA)对形状残差和形状索引特征进行分析，获得二者对应的投影矩阵P和Q；由于在测试过程中形状残差不可知，因此仅使用形状索引特征的投影结果，并且将该投影结果的子空间命名为融合子空间

其中r为融合子空间的列维数，z_i,j代表第i个样本的形状索引特征在第j维子空间的投影结果；如果第i个和第g个样本在融合子空间上的每一维符号都相同，那么这两个样本属于同一子集U_k。Specifically, a method for locating facial feature points based on a two-layer cascaded regression model includes a process of dividing a sample subset by using a fusion subspace, and defining a face image in a training set and a corresponding set of real shapes

where N is the number of samples, I _i and

are the i-th face image and the corresponding real shape, respectively. First, the average simplified face shape is matched with the bounding box of all training set samples, and the matching result is defined as

Then calculate the shape residual

and PHOG-based shape index features

Use canonical correlation analysis (CCA) to analyze the shape residuals and shape index features, and obtain their corresponding projection matrices P and Q; since the shape residuals are unknown during the testing process, only the projection results of the shape index features are used, And the subspace of the projection result is named the fusion subspace

where r is the column dimension of the fusion subspace, z _i,j represents the projection result of the shape index feature of the ith sample on the jth dimensional subspace; The one-dimensional symbols are the same, then the two samples belong to the same subset U _k .

和3D完整人脸形状

为了使s_3d与人脸图像拟合，本实施例先利用弱透视投影将3D简化人脸形状投影

到2D平面上。如果将第一层级联回归模型输出的2D简化人脸形状记作

那么通过最小化

与

之间的欧氏距离，就可以确定对

执行弱透视投影的仿射变换参数，实现人脸图像与3D人脸形状的拟合。本实施例使用最小二乘法计算得到放射变换的参数，对3D完整人脸形状

执行相同的仿射变换，即可得到与人脸图像拟合后的3D完整人脸形状在2D平面上的投影

当人脸姿态发生变化时，3D人脸形状上标记的轮廓部位的特征点会受到脸颊的遮挡，而真正的轮廓特征点应该在脸颊的边界处。本实施例为16个轮廓特征点(排除了最底部的轮廓特征点)建立了与它们自身平行的坐标点子集。使用上述拟合过程中得到的偏航角β，可以判断人脸是偏向左侧还是右侧。当人脸偏向左侧时，左侧的8个轮廓特征点搜索其对应坐标点子集中x坐标最小点，将其作为新的轮廓特征点；而当人脸偏向右侧时，就是右侧的8个轮廓特征点搜索其对应坐标点子集中x坐标最大点，标记成为新的轮廓特征点。A face identification method for sparse hybrid dictionary learning also includes generating an initial shape for the second layer model using 3D fitting. This embodiment uses the neutral face shape in the 3DMM model, and also locates the required coordinate points according to the facial feature points, and extracts 14 coordinate point sets and 68 coordinate point sets respectively from the dense 3D coordinate point set, Respectively recorded as 3D simplified face shape

and 3D full face shape

In order to make s _3d fit with the face image, this embodiment first uses weak perspective projection to project the 3D simplified face shape.

onto the 2D plane. If the 2D simplified face shape output by the first-level cascaded regression model is written as

Then by minimizing

and

The Euclidean distance between

The affine transformation parameters of the weak perspective projection are performed to realize the fitting of the face image and the 3D face shape. This embodiment uses the least squares method to calculate the parameters of the radiation transformation, and the 3D complete face shape

Perform the same affine transformation to get the projection of the 3D complete face shape fitted with the face image on the 2D plane

When the face pose changes, the feature points of the contour parts marked on the 3D face shape will be occluded by the cheeks, while the real contour feature points should be at the border of the cheeks. This embodiment establishes a subset of coordinate points parallel to themselves for the 16 contour feature points (excluding the bottommost contour feature point). Using the yaw angle β obtained in the above fitting process, it can be determined whether the face is biased to the left or the right. When the face is biased to the left, the 8 contour feature points on the left search for the minimum x-coordinate point in the subset of corresponding coordinate points, and use it as the new contour feature point; and when the face is biased to the right, it is the 8 on the right. Each contour feature point searches for the maximum x-coordinate point in its corresponding coordinate point subset, and marks it as a new contour feature point.

和回归器

经过T次迭代回归得到简化人脸形状

再利用上述的3D拟合法得到仿射变换参数，投影得到粗略的完整人脸形状s⁰；最终，根据所有的训练样本和上个步骤得到的粗略完整人脸形状，训练特征提取函数Φ和回归器W，经过T次迭代实现精回归。Preferably, in this embodiment, the above two methods are combined, and a two-layer cascaded regression model is constructed on the basis of the traditional cascaded regression model. Firstly, the shape index feature based on PHOG is extracted, and the projection matrix P obtained in the training process is used to project it into the fusion subspace, and K sample subsets are divided according to it; then the feature mapping function is trained according to different subsets.

and regressor

After T iterations of regression, the simplified face shape is obtained

Then use the above-mentioned 3D fitting method to obtain affine transformation parameters, and project to obtain a rough and complete face shape s ⁰ ; finally, according to all training samples and the rough and complete face shape obtained in the previous step, the training feature extraction function Φ and regression The device W, after T iterations to achieve precise regression.

Example 2

Referring to FIG. 2 to FIG. 9 , it is a second embodiment of the present invention. This embodiment is different from the first embodiment in that the experimental samples used are from two widely used facial feature point positioning datasets: HELEN dataset and the 300-W dataset. These datasets contain unconstrained face images with pose deflection, occlusion, and illumination changes, which are challenging. In the training process, all samples are flipped horizontally, and then 10 face shapes are randomly sampled from other training samples for each sample as the initial shape, and the total number of samples is expanded by 20 times. During the testing process, the first-level cascaded regression model still uses the average face shape as the initial shape, and the second-level cascaded regression model uses the result of 3D fitting as the initial shape.

Parameter setting: The first-layer and second-layer cascaded regression models use basically the same parameters, the number of decision trees in each random forest is G=10, the depth of the tree is D=5, and the number of iterations is T=7. Since the shape change space of the second-level cascaded regression model is smaller than that of the first-level, the first-level and second-level cascaded regression models use different ranges when randomly selecting pixel difference features. Descending from 0.4 to 0.08, the radius of the second layer decreases from 0.3 to 0.06 with the number of iterations (all distances normalized by the face bounding box).

Evaluation standard: Same as most of the current facial feature point localization algorithms, the normalized mean error (NME) is used to measure the accuracy of the algorithm. The normalized standard error is calculated as follows:

分别是第i个样本的预测人脸形状和真实人脸形状，d_ipd则是第i个样本双眼中心之间的欧氏距离。where N is the total number of samples, L is the number of feature points of the face shape, s _i and

are the predicted face shape and the real face shape of the ith sample, respectively, and d _ipd is the Euclidean distance between the centers of the eyes of the ith sample.

Based on the above, first simplify the evaluation of the fitting effect of the face shape, and before locating the simplified face shape, first determine which feature points to use as the simplified face shape. This embodiment is based on the 5-point shape including the center of the eyes, the tip of the nose, and the corner of the mouth. Some feature points are added to the parts of the eyes, nose, mouth, and contour, respectively, and a total of 6 simplified face shapes are selected and recorded. In order to select the most suitable simplified face shape from these shapes, this embodiment evaluates from two aspects: (1) trains a single-layer cascaded regression model based on each simplified face shape, and calculates The normalized standard error on the set samples is recorded as the positioning error; (2) using the positioning results obtained in (1), generate a complete face shape according to the 3D fitting method proposed in this embodiment, and calculate the generated complete face shape The average normalized error between the real face shape of the sample and the fitting error. Figure 2 shows the localization error and fitting error calculated on the HELEN dataset for six simplified face shapes according to the above criteria. At the same time, in order to prove the advantage of simplifying the face shape, the error of directly locating the complete face shape and the error of the newly generated complete face shape after 3D fitting according to the positioning result are also marked in the figure.

According to the positioning error curve and fitting error curve in Figure 2, it can be seen that the 8-point shape and the 16-point shape have an increase of 3 points and 2 points in the contour part compared with the 5-point and 14-point shapes, respectively, and the positioning error also increases. , which proves that the features of contour parts cannot be well reflected by texture features. However, the fitting error of the 8-point shape is much lower than that of the 5-point shape, which proves that adding a small number of feature points of the contour can constrain the overall shape and reduce the fitting error. Compared with the previous shape, the 8 o'clock, 10 o'clock, 12 o'clock and 14 o'clock shapes are made to add 3 contour feature points, change 2 eye center feature points to 4 double eye corner feature points, and add 2 nose wing features. Point and the improvement of adding two upper and lower lip feature points, the fitting error also gradually decreases, which proves that adding the feature points of the facial features can get a better fitting effect. However, considering the computational cost, this embodiment does not continue to try to add more feature points to the facial features of the simplified face shape.

It can also be seen from Figure 2 that the positioning error of the complete face shape containing 68 points is higher than that of all simplified face shapes, which shows that a large number of non-key feature points will increase the difficulty of positioning. Among the selected 6 simplified face shapes, the 14-point shape with the smallest positioning error and fitting error is the 14-point shape, and its specific fitting error is 9.22%, and the positioning and fitting operations are directly performed on the complete face shape to obtain The fitting error is 9.63%. It can be seen that the simplified face shape can not only improve the positioning accuracy, but also prevent too many feature points from constraining each other to cause underfitting of the 3D model. In the following experiments, the 14-point shape is used as the simplified face shape.

Preferably, the effect evaluation of the subspace division, in order to prove the effectiveness of the fusion subspace division method proposed in this embodiment, will be evaluated from two aspects: (1) the correlation between the fusion subspace and the face pose; ( 2) The result of subspace division affects the positioning effect of the first-level cascaded regression model.

Since face images have the characteristics of manifold distribution, an effective feature subspace should also satisfy this characteristic. According to the subspace division method proposed in this embodiment, features are extracted from the samples in the 300-W data set and projected into the fusion subspace. At this time, the distribution of some face images in the first two dimensions of the fusion subspace is shown in Figure 3. It is not difficult to see that along the horizontal direction in the figure, the yaw angle of the sample changes, that is, it gradually transitions from the left to the right; and along the vertical direction in the figure, the roll angle of the sample changes, and It is a gradual transition from leaning to the left to leaning to the right. The samples distributed in the center of the image have a small attitude deflection angle, mainly frontal faces; the samples around the image have a gradually larger attitude deflection angle according to the change trend in the horizontal and vertical directions. This proves that the fusion subspace satisfies the manifold distribution characteristics of faces and can be used to divide sample subsets with different poses.

In order to evaluate the influence of the number K of sample subsets on the positioning effect of the first-level cascaded regression model, this embodiment calculates the normalized average error of the simplified face shape positioning on different data sets when different K values are taken. The specific results are as follows: shown in Figure 4. Where K=1 means that no subspace division is performed, and all samples are used to train a single regression model. It can be seen that on the HELEN dataset, the error shows a downward trend from K=1 to K=4, from 4.46% to 4.32%, which proves that the subspace division method can effectively reduce the positioning error. However, in the case of K=8 and K=16, the error increases again, and even exceeds the case of no subspace division at K=16. The reason for this phenomenon is that, as the number of subsets increases, the number of training samples in each subset is also decreasing, which reduces the robustness of the model; the second is that the pose of the HELEN dataset changes less, and the number of subsets is too large. It is easy to cause confusing division results. On the 300-W challenge set with rich pose changes, the subspace division method can greatly reduce the positioning error, from K=1 to K=8, the error is reduced from 13.83% to 11.48%. The error picks up slightly at K=16 for similar reasons to the HELEN dataset. According to the experimental results, when the algorithm of this embodiment is compared with other algorithms, the values of K in the HELEN data set and the 300-W data set are respectively 4 and 8.

In order to prove that the fusion subspace division method is more advantageous than the PCA subspace division, this embodiment compares the performance of the two subspace division methods on the simplified face shape on the 300-W challenge set when K=2, 4, and 8, respectively. The effect of positioning error is shown in Figure 5. The position of the dotted line in the figure is the positioning error when subspace division is not used. It can be seen that both subspace division methods can reduce the positioning error, but the effect of the division method based on fusion subspace is obvious because it is based on PCA subspace. Since the division method based on PCA subspace cannot be applied to static faces, only shape index features are used to implement the method in this experiment. This method ignores the correlation between the shape index feature and the shape residual, and the obtained division results are also relatively chaotic; while the method of this embodiment combines the characteristics of the two, so the division of the pose is more accurate. A lower alignment error can be achieved in each value.

Further, the effect evaluation of the two-layer cascaded regression model is based on the unimproved LBF algorithm, the two-layer cascaded regression model (K=1) without subspace division, and the two-layer cascaded regression model when K takes different values. Localization results of the regression model on the complete face shape on the HELEN dataset and the 300-W dataset.

Since some parameters used in this embodiment are different from the original article of the LBF algorithm, the experimental data about the LBF algorithm in Table 1 is different from the data in the article. According to Table 1, it can be seen that the double-layer cascaded regression model (K=1) without subspace division performs better than the LBF algorithm before improvement on all data sets, which proves that the double-layer model is better than the single-layer model. layer model. After continuing to improve the subspace division, the algorithm in the HELEN data set and the 300-W challenge set increases with the K value, and the error gradually decreases, and the error is reduced to the lowest when K=4 and K=8 respectively. The original algorithm reduces by 4% and 23.55%, respectively. This corresponds to the positioning effect of simplifying the shape of the face, so the range of the K value is only 4 and 8 respectively. On the 300-W common set, the errors of the two-layer cascaded regression models with different K values are very small. The reason for this phenomenon may be that the common set samples with small attitude changes are densely distributed in the center of the subspace. location, and close to the boundary of the division. After subspace division, some frontal face samples may be assigned to inappropriate subsets, so this improvement is not very obvious for the optimization of the localization results of frontal face images. Nevertheless, after averaging the localization results with the challenge set, the localization error of the 300-W ensemble will decrease with the increase of the K value, which is 11.24% lower than the original algorithm.

As shown in Figure 6, the results of the seventh iteration correspond to the data in Table 1. It can be seen that the accurate initial shape can give the second-level cascaded regression model an advantage in the first iteration, and can maintain this advantage to the last iteration to achieve better localization effect.

Table 1. Complete face shape localization effects on different datasets (%)

Based on the above, in order to compare with the current state-of-the-art methods, this embodiment will use the normalized average error, CED curve and their specific positioning results of each algorithm on different data sets as evaluation criteria.

Table 2 shows the normalized average errors of different algorithms on the HELEN and 300-W datasets, which are all from the original paper of the algorithm or its related papers. Among them, ESR, SDM, LBF, CFSS and MCO are algorithms based on cascaded regression, and CFAN, 3DDFA and PIFA-S are algorithms based on deep learning.

According to the data in the table, it can be seen that the algorithm of this embodiment has the smallest error among all the comparison algorithms on the HELEN data set and the 300-W common set with small attitude changes, and is even better than the three algorithms based on deep learning . The reason for this result should be that CFAN only changes the feature mapping function to a deep autoencoder network, which is still a single-level cascaded regression in essence; 3DDFA and PIFA-S are based on the 3DMM model, and the results of 3D face alignment are in 2D people. It is not dominant under the evaluation criteria of face alignment. This result also proves that the regression process of the double-layer cascaded regression model from coarse to fine can improve the accuracy of the algorithm. In the 300-W challenge set with rich posture changes, the algorithm of this embodiment ranks second among the algorithms based on cascade regression, second only to the CFSS algorithm, and also has a certain degree of competition compared with the two algorithms based on deep learning. force. Among them, CFSS selects candidate shapes by calculating the probability matrix, and fuses the update results of multiple candidate shapes to generate new shapes, which improves the accuracy of positioning at the expense of huge computational load; 3DDFA and PIFA-S both use convolutional neural networks. To optimize 3DMM parameters, achieve 3D face alignment, and have strong robustness to pose and occlusion changes. After averaging the results of the 300-W normal set and the challenge set, the results of the algorithm in this embodiment are second only to CFSS, and surpass all other algorithms based on cascaded regression and two algorithms based on deep learning. Compared with the current advanced algorithms, the algorithm has a good competitive advantage.

Table 2 Normalized mean error (%) on the 300-W dataset

As shown in Figure 7, the normalized average error corresponding to the reproduced SDM algorithm in this embodiment is 7.04%, which is better than the 7.50% of the original text; the normalized average error corresponding to the reproduced LBF algorithm is 6.59 %, which is slightly worse than the original 6.32%, because in order to prevent the running memory consumption from being too high due to the high feature dimension, this embodiment reduces the number of decision trees from 1200 in the original to 680 when reproducing; the reproduced CFSS The normalized average error corresponding to the algorithm is 6.00%, and the MATLAB code provided by the original author on Github is used. It can be seen that the algorithm of this embodiment has certain advantages compared with other algorithms.

As shown in FIG. 8 , the first row in the figure is the real shape of the sample, and the second to fifth rows are the positioning results of the algorithm of this embodiment, CFSS, LBF, and SDM, respectively. It can be seen from the first to fourth columns that the algorithm of this embodiment can obtain results that are very close to the real shape, while errors are visible in the other three algorithms; in the fifth to eighth columns, the results of the algorithm of this embodiment are There is a certain error in the results. The basic positioning of SDM and LBF fails. The results of CFSS in the fifth and sixth columns are better than that of the algorithm in this embodiment, but the positioning in the seventh and eighth columns with larger pitch angles fails, and the results are not as good as the algorithm in this embodiment. . According to these results, it can be seen that the algorithm of this embodiment has certain robustness in attitude.

As shown in Figure 9, the samples marked with feature points represent easy cases, difficult but successfully predicted cases, and failed cases, respectively. The main reason for the failure is the exaggerated facial expressions of the samples. From these results, it is not difficult to see that the proposed method is pose robust.

It should be appreciated that embodiments of the present invention may be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in non-transitory computer readable memory. The methods can be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the specific Methods and figures described in the Examples. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose.

Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes (or variations and/or combinations thereof) described herein can be performed under the control of one or more computer systems configured with executable instructions, and as code that executes collectively on one or more processors (eg, , executable instructions, one or more computer programs or one or more applications), implemented in hardware, or a combination thereof. The computer program includes a plurality of instructions executable by one or more processors.

Further, the methods may be implemented in any type of computing platform operably connected to a suitable, including but not limited to personal computer, minicomputer, mainframe, workstation, network or distributed computing environment, stand-alone or integrated computer platform, or communicate with charged particle tools or other imaging devices, etc. Aspects of the invention may be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optically read and/or written storage medium, RAM, ROM, etc., such that it can be read by a programmable computer, when a storage medium or device is read by a computer, it can be used to configure and operate the computer to perform the processes described herein. Additionally, the machine-readable code, or portions thereof, may be transmitted over wired or wireless networks. The invention described herein includes these and other various types of non-transitory computer-readable storage media when such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein. A computer program can be applied to input data to perform the functions described herein, transforming the input data to generate output data for storage to non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

As used in this application, the terms "component," "module," "system," etc. are intended to refer to a computer-related entity, which may be hardware, firmware, a combination of hardware and software, software, or running software. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread in execution, a program, and/or a computer. As an example, both an application running on a computing device and the computing device may be components. One or more components can exist in a process and/or thread of execution, and a component can be localized on one computer and/or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures thereon. These components can be implemented by, for example, having one or more data groupings (eg, data from one component interacting with another component in a local system, a distributed system, and/or in a signaling manner such as the Internet network to interact with other systems) to communicate locally and/or as remote processes.

It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be Modifications or equivalent substitutions without departing from the spirit and scope of the technical solutions of the present invention should be included in the scope of the claims of the present invention.

Claims (10)

1. a facial feature point location method based on double-layer cascade regression model, is characterized in that: comprise, Amplify the sample by randomly selecting the face shape from the training sample set as the initial shape; From the complete face shape of 68 points, 14 points that play a key role in the positioning of face feature points are selected to form a simplified face shape; Using the subspace division method, the samples are divided into multiple subsets, and each subset trains a regressor, and all the regressors together form the first-level cascaded regression model, which is used to predict the simplified face shape; Using 3D fitting, realize the projection from the simplified face shape predicted in the first layer to the complete face shape, and generate a rough initial complete face shape; The rough and complete face shape is used as the initial shape, and all training samples are used to train the regressor to form a second-level cascaded regression model to form a two-layer cascaded regression model for predicting the complete face shape. 2. The method for locating facial feature points based on a double-layer cascaded regression model according to claim 1, wherein the step of constructing the double-layered cascaded regression model comprises, First, extract the shape index feature based on PHOG, use the projection matrix P obtained in the training process to project it into the fusion subspace, and divide K sample subsets according to it; Then, the feature mapping function is trained separately according to different subsets

and regressor

After T iterations of regression, the simplified face shape is obtained

Then use the 3D fitting method to obtain affine transformation parameters, and project to obtain a rough complete face shape s ⁰ ; Finally, according to all the training samples and the rough and complete face shape obtained in the previous step, the feature extraction function Φ and the regressor W are trained, and refined regression is achieved after T iterations. 3. the facial feature point locating method based on double-layer cascade regression model according to claim 1 and 2, is characterized in that: the step of selecting optimal simplified facial shape comprises, Based on the most representative 5 contour feature points, add some feature points to the eyes, nose, mouth and contour respectively, and select 6 simplified face shapes; The five most representative contour feature points include the center of the eyes, the tip of the nose and the corner of the mouth. 4. the facial feature point location method based on the double-layer cascade regression model according to claim 3, is characterized in that: the step of selecting optimal simplified facial shape also comprises, Pick the most suitable simplified face shape, including calculating the positioning error and fitting error; The positioning error and the fitting error are compared, and the 14-point shape with the minimum positioning error and the fitting error is selected as the simplified face shape. 5. The method for locating facial feature points based on a double-layer cascaded regression model according to claim 4, wherein the method for calculating the positioning error is to train a single-layer cascaded regression based on each simplified face shape. Model, calculate the normalized standard error of each model on the test set samples; The fitting error calculation method is, based on the positioning result obtained by the positioning error and the complete face shape generated by the 3D fitting method, calculate the average normalized error between the generated complete face shape and the real face shape of the sample. . 6. The method for locating facial feature points based on a double-layer cascaded regression model according to claim 2, wherein the fusion subspace comprises:

Among them, r is the column dimension of the fusion subspace, and z _i,j represents the projection result of the shape index feature of the ith sample on the jth dimensional subspace. 7. The method for locating facial feature points based on a double-layer cascaded regression model according to claim 6, wherein the step of dividing the sample into K sample subsets by utilizing the fusion subspace comprises, Define the face images in the training set and the corresponding set of real shapes,

where N is the number of samples, I _i and

are the ith face image and the corresponding real shape, respectively; Match the average simplified face shape to the bounding box of all training set samples, and the matching result is defined as

compute the shape residuals,

and PHOG-based shape index features

Use canonical correlation analysis to analyze the shape residual and shape index features, and obtain their corresponding projection matrices P and Q; The subsets are divided according to the following formula,

8. the facial feature point location method based on the double-layer cascade regression model according to claim 7, is characterized in that: if the i-th and g-th samples have the same each dimension symbol on the fusion subspace, Then the two samples belong to the same subset U _k . 9. The method for locating facial feature points based on a double-layer cascaded regression model according to claim 8, wherein the step of utilizing the 3D fitting method to generate an initial shape for the second-layer model comprises, Using the 3DMM model composed of dense 3D coordinate point sets, 14 coordinate point sets and 68 coordinate point sets are extracted respectively, which are respectively recorded as 3D simplified face shape

and 3D full face shape

Use weak perspective projection to project the 3D simplified face shape to the 2D plane, and fit the s _3d to the face image, including:

Among them, f is the scaling factor, P _o is the orthographic projection matrix

R(α,β,γ) is a 3×3 rotation matrix composed of pitch angle α, yaw angle β and roll angle γ, and t _3d is the displacement vector.

for

Projection on a 2D plane; If the 2D simplified face shape output by the first-level cascaded regression model is recorded as

by minimizing

and

The Euclidean distance between, then determine the appropriate values of the parameters f, R and t _3d to realize the fitting of the face image and the 3D face shape, including:

After calculating the parameters of the radiation transformation by the least squares method, the 3D simplified face shape in the above weak perspective projection formula

Replaced with 3D full face shape

The projection of the 3D complete face shape after fitting with the face image on the 2D plane can be obtained.

When the face pose changes, the feature points of the contour parts marked on the 3D face shape will be occluded by the cheeks, while the real contour feature points should be at the border of the cheeks. Use the yaw angle β obtained in the above fitting process to determine whether the face is biased to the left or the right. 10. The method for locating facial feature points based on a double-layer cascaded regression model according to claim 9, wherein the step of utilizing the 3D fitting method to generate an initial shape for the second-layer model further comprises: When the face is biased to the left, the 8 contour feature points on the left search for the minimum x-coordinate point in its corresponding coordinate point subset, and use it as a new contour feature point; When the face is biased to the right, the 8 contour feature points on the right side search for the maximum x-coordinate point in the subset of corresponding coordinate points, and mark it as a new contour feature point; The eight contour feature points are three contour feature points added on the basis of the five contour feature points.

Images