KR20080068988A - Progressive aam based robust face alignment - Google Patents

Abstract

A face outline detection method using a gradual AAM(Active Appearance Model) is provided to improve a detection rate as being robust against a pose, a lighting and a face background. A face outline detection process comprises the following steps. From a model input image, a face region is detected. The detected face region is normalized. Fitting with a face inner AAM is performed. Through a relation matrix, an initial value for the entire face AAM parameter vector is estimated. By using the estimated initial value and performing entire face AAM fitting, the entire face outline is detected.

Description

Face contour detection using progressive active shape model {Progressive AAM based Robust Face Alignment}

1) Figure 1 is a configuration and operation flow chart of the facial contour detection method using a progressive AAM provided by the present invention

2) Figure 2 is a face region detection and size normalization that is one of the processes in the facial contour detection method using a progressive AAM provided by the present invention

3) Figure 3 is a facial posture normalization that is one of the processes in the facial contour detection method using a progressive AAM provided by the present invention

4) Figure 4 shows the internal facial feature points and the entire face features used in the facial contour detection method using the progressive AAM provided by the present invention.

5 is a shape change obtained in the configuration of the internal face AAM model, which is one of the processes in the face contour detection method using the progressive AAM provided by the present invention

6 is a texture change obtained from the configuration of the internal AAM model that is one of the processes in the facial contour detection method using the progressive AAM provided by the present invention

7 is a shape change obtained from the configuration of the entire face AAM model, which is one of the processes in the face contour detection method using the progressive AAM provided by the present invention

8 is a texture change obtained from the configuration of the entire face AAM model, which is one of the processes in the face contour detection method using the progressive AAM provided by the present invention.

The present invention is an improvement over the existing AAM method for face contour detection, which is essential for many applications requiring face recognition, face-based user interface, driver observation, animation, and other face modeling. It is robust against values, postures, lighting, and face backgrounds, further improving the detection rate. Of course, the detection time is increased compared to the conventional AAM-based face contour detection method, but the real-time detection is sufficiently possible because the increase of the calculation amount is not large. The configuration and operation flowchart of the face contour detection method of the present invention are shown in FIG.

There is no domestic patent registration of the invention similar to the present invention, but it is investigated that there are the following two pending applications. 'Face feature detection for real-time situation recognition' (Application No. 10-2006-0026016), which is pending, provides a method for detecting a face feature point using a template matching method that can be modified. The present invention is fundamentally different from the template matching method which can be modified in that it is basically based on AAM. Facial feature point detection by the deformable template takes a relatively short time, but there is a disadvantage in that it is not easy to construct a deformable template suitable for various faces.

In addition, the pending facial feature detection method, medium and apparatus using a stereo coupling mechanism (Application No. 10-2004-0049380) provides a stereo coupling mechanism using two face images for face feature detection. The present invention is different from the above invention because it provides a method for effectively extracting facial feature points from only one face image. The studies on face contour detection or facial feature point detection published in the previous papers are as follows.

One of the most effective methods for detecting deformable 2D objects is the AAM (Active Appearance Model). AAM is described by Edwards et al. (G. Edwards, CJ Taylor, and TF Cootes, “Interpreting Face Images using Active Appearance Models,” in Proc. IEEE Int. Conf. Automatic Face and Gesture Recognition, pp. 300-30, 1998). First proposed, Cootes et al. (TF Cootes, DJ Edwards, and SJ Taylor, “Active Appearance Models,” IEEE Trans.Pattern Anal.Mach. Intell., Vol. 23, No. 6, pp. 681-685, Jun. 2001). Since then, much research has been done on AAM improvement and expansion. Facial contour detection using AAM has been studied to be relatively stable, but is well known to be sensitive to initial values. Much work has been done to improve the speed and performance issues of AAM.

Li et al. (SZ Li, YS Cheng, HJ Zhang, QS Cheng “Multi-view face alignment using direct appearance models,” 5th Automatic Face and Gesture Recognition, pp. 324-329, 2002) introduced the DAM (Direct Appearance Model). And it is extended to multi-image face contour detection. Unlike AAM, DAM does not combine shape and texture to directly predict shape using texture during optimal parameter vector search. This DAM method is reported to improve the search speed and accuracy compared to the AAM. Matthews and Baker (I. Matthews and S. Baker, “Active Appearance Models Revisited,” International Journal of Computer Vision, Vol. 60, No. 2, pp. 135–164, Nov. 2004) speed up the AAM registration process. Inverse compositional approach has been proposed to improve, and Batur and Hayes (AU Batur and MH Hayes, “Adaptive Active Appearance Models,” IEEE Transactions on Image Processing, Vol. 14, Issue 11, pp. 1707–1721 , Nov. 2005) proposed an adaptive AAM that improves the matching speed and accuracy of AAM. The basic difference between the above-described research on face contour detection and the proposed method is a gradual AAM-based face contour detection method. The proposed method differs from the basic AAM-based face contour detection methods because the initial AAM-based face contour detection is sensitive to the initial value, so the initial value should be set well. In this paper, the gradual AAM-based face detection method performs the contour detection of the face by applying AAM only to the inside of the face with little change. Since the initial value for the entire face AAM is estimated by using the information, it provides a more stable initial value. A study suggesting a multi-step facial contour detection method as in the present invention is described by Cristinacce et al. (D. Cristinancce, T. Cootes, and I. Scott, “A Multi-Stage Approach to Facial Feature Detection,” Proc. British Machine Vision Conference 2004, 1, pp. 277-286, 2004), where the facial feature points are roughly detected using the Pairwise Reinforcement of Feature Responses (PRFR) method, and then refined using AAM matching. This study differs from the progressive face detection of the present invention by first roughly detecting facial features and then elaborating them.

Face contour detection is the task of finding facial feature points such as eye, eyebrow, nose, mouth, and chin contours on the face, which is required for many applications requiring face recognition, face-based user interface, driver observation, animation, and other face modeling. Technology. In the face recognition system, when face contours, that is, feature points of a face, are detected in an input image, feature vector information is extracted from these feature points and compared with registered feature vector information to perform face recognition. Therefore, accurate facial contour extraction is needed whenever possible. Face shapes are not fixed shapes, but are deformable objects that vary from person to person. Deformable objects are well known to be harder to detect than fixed-type objects, and thus, much research has been conducted on face contour detection. One of the most effective methods for detecting deformable 2D objects is the AAM (Active Appearance Model). Facial contour detection using AAM has been studied to be relatively stable, but is well known to be sensitive to initial values. Usually AAM takes the initial value as the average form of the model (i.e. takes the AAM parameter initial value as 0).

  When face posture and lighting are varied, the change in the contour of the jaw of the face is more severe than the change in the contours of the eye, nose, mouth and the like. Also, the jaw parts are more affected by the face background because they are closer to the face background than the face contour. Therefore, for a new face image with a complex background and a different facial pose than the average form of the AAM model, the initial values of the features of the jaw contours will not start at a value close enough to the final global minimum (the exact jaw contour feature point value). It may not be possible to find the exact location of the feature of the correct jaw contour.

The present invention focuses on the fact that the feature points of the area including only the main part of the face are relatively less changed and less affected by the background screen, and then perform AAM on only the inside of the face, and then extract the feature points inside the face. A progressive AAM-based face contour detection method is provided for extracting feature points for an entire face using this information. The progressive AAM-based face contour detection method of the present invention finds a stable facial feature point first, and utilizes this information to detect feature points of less stable jaw parts by using the information as much as possible. It provides the advantage of better detecting the whole face feature points.

 The object of the present invention summarized based on the above description is as follows.

1) Robust AAM-based face contour detection algorithm including posture, lighting, and face background

2) Provides more robust face contour detection algorithm than posture AAM based face contour detection method

3) Provides a method for detecting face contours with improved detection rate without increasing the computational complexity compared to current AAM-based face contour detection methods.

The present invention improves the existing AAM method for face contour detection, which is essential for many applications requiring face recognition, face-based user interface, driver observation, animation, and other face modeling. An object of the present invention is to provide a face contour detection algorithm that is robust against a background and has an improved detection rate. The technique based on the present invention is a method using an AAM (Active Apperance Model).

A configuration and operation flowchart of the face contour detection method using the progressive AAM provided by the present invention are shown in FIG. The facial contour detection method provided by the present invention is composed of two steps such as 'face model composition and relationship extraction' and 'face contour detection'.

The procedure of 'Face Model Construction and Relationship Extraction' is performed by 1) face region detection 2) normalization of detected face region 3) construction of face AAM model 4) construction of full face AAM model 5) construction of face AAM model parameter vector And extracting a relation matrix between the entire face AAM model parameter vector (see FIG. 1 (a)),

  The procedure of the 'face contour detection' step of detecting face contours (feature points of a face) in an input image is performed by 1) detecting a face region in a model input image 2) normalizing detected face regions 3) matching with an internal AAM model 4). Initial value estimation for the entire face AAM model parameter vector through the relation matrix 5) Using the estimated initial value and total face contour detection through full face AAM matching, etc. (see FIG. 1 (b)).

end. Model Construction and Relationship Extraction Steps

1) Face area detection

   In order to find facial feature points, a face region must first be detected in an input image. Ada-Boosting algorithm was used to detect facial area.

2) Detected face region normalization

   The normalization procedure normalizes the face size and straightens the face out of position.

 Size normalization

The size of the face image detected by the Ada-Boosting method in the input image is not constant. In order to increase the efficiency of feature detection, a normalization process for making the size of the detected face image constant is required. In the present invention, the size of the face image is normalized to 256 (pixel) * 256 (pixel) size. Figure 2 shows the result of changing the color to black and white in the input image, the face detected in the black and white image and normalized to 256 * 256 size. Posture normalization

-When the face is rotated in plane, the tilt angle is obtained by using the valley and edge information. When the face is rotated in plane, the face posture is straightened.

3 shows the result of normalizing the face posture.

3) Construct AAM model inside face

The number of internal facial feature points used in the present invention is 45, and the positions of internal facial feature points are shown in FIG. 4.

 A) Basic Active Appearance Model (AAM) Overview

 Here is a brief description of AAM to understand the AAM model configuration. This explanation is given in the paper on AAM (TF Cootes, DJ Edwards, and SJ Taylor, “Active Appearance Models,” IEEE Trans.Pattern Anal.Mach.Intel., Vol. 23, No. 6, pp. 681-685, Jun 200). As described in this paper, AAM consists of two stages. It consists of a modeling step and a fitting step.

① AAM Modeling

 Assume the number of feature points of the face contour and the number of face model images used for modeling are M. In this case, the shape of each facial contour AAM is defined as a set of feature point coordinates. In other words,

(1)

(One)

는 i 번째 얼굴 특징점의 좌표를,'t'는 벡터와 행렬의 전치(transpose)를 의미한다.)(here

Is the coordinate of the i th facial feature point, and 't' is the transpose of the vector and the matrix.)

, n개의 형태 고유벡터(모드)

을 구할 수 있으며, 따라서 모델 얼굴 또는 새로운 입력 이미지의 얼굴의 형태 X는 다음과 같이 이들의 선형 결합 으로 나타낼 수 있다. At this time, it is assumed that the shape X vector of each face has already been normalized by procrustes analysis. PCA about the M model image shape, the average shape

, n shape eigenvectors (mode)

Therefore, the shape X of the model face or the face of the new input image can be represented by their linear combination as follows.

(2)

(2)

)(here,

)

에 정합되도록 들로니 삼각법 알고리즘(Delaunay triangulation algorithm) 을 이용하여 모델 이미지들을 워핑한다. 이후, 형태 정규화된 모델 이미지들로부터 그레이 레벨 정보를 샘플링 하여

을 얻는다. 다시 조명 변화의 영향을 줄이기 위해, 다음과 같이 정규화 한다. The feature points of each model are averaged to build a statistical model of gray level texture.

The model images are warped using a Delaunay triangulation algorithm to match. Then, the gray level information is sampled from the shape normalized model images.

Get To reduce the effects of lighting changes again, normalize as follows.

(3)

(3)

는 스케일링 상수,

는 오프셋상수로

가 정규화된 평균에 정합되도록 결정된 상수)(here

Is the scaling constant,

Is an offset constant

Constant determined to match the normalized mean)

을 다음과 같이 평균 정규화된 그레이 레벨 벡터

와

개의 고유 텍스쳐 모드 벡터

의 선형 모델로 표현할 수 있다. Now apply the PCA to the textures as in the shape vector so that each texture face is a texture vector of the face of the new input image.

Mean normalized gray level vector as

Wow

Unique texture mode vectors

It can be expressed as a linear model of.

(4)

(4)

)(here,

)

로 기술할 수 있다.Equations (2) and (4) combine to form a common parameter vector as follows:

It can be described as

(5a)

(5a)

(5b)

(5b)

Modeling the shape and texture of an object (face in our case), as shown in (5), is called (combined) AAM.

② AAM Fitting

를 찾고 이때의 파라미터 벡터

를 가지고 (5a)로 표현되는 얼굴 윤곽을 새로 입력된 이미지에서 찾고자 하는 얼굴 윤곽으로 판정한다. 에러 제곱을 최소화 하는 파라미터 벡터

를 어떻게 찾아 나아가느냐 하는 것과 최적화 절차 수행에 필요한 파라미터 벡터의 초기값 설정은 각 AAM 구현 알고리즘에 달려 있다. 보통 초기값 설정은 평균 형태 즉,

을 취한다. Matching the AAM model (5) to a new image, that is, applying the AAM model to find face contours in the new image is an optimization problem. In AAM, the parameter vector of model (5) minimizes the square of the error between the texture of the face image modeled as (5) and the texture of the newly input face image.

Find the parameter vector

The face outline represented by (5a) is determined as the face outline to be found in the newly input image. Parameter Vector Minimizing Error Squares

How to find and set the initial value of the parameter vector required to perform the optimization procedure is up to each AAM implementation algorithm. Usually the initial setting is in average form,

Take

B) Construct AAM model inside face

  Based on the AAM described in the above 3-A), 45 face feature points are manually detected for each of the given model face images, and the face face AAM model is trained using the manually detected face feature points. Configure The average shape and first two mode shape changes by intra-face modeling are shown in FIG. 5 and the average texture and first two mode texture changes are shown in FIG. 6.

4) Full Face AAM Model

  The total number of facial feature points is 58, including all 45 internal feature points and 13 chin parts. Based on the AAM described in the above 3-A), each of the 58 facial full feature points is manually detected for the given model face images, and the entire face AAM model is trained using the manually detected face total feature points to complete the full face AAM. Construct the model. Average shape and first two mode shape changes by full face modeling are shown in FIG. 7, and average texture and first two mode texture changes are shown in FIG. 8.

5) Extraction of the relationship matrix between the face AAM model parameter vector and the face AAM model parameter vector

In the progressive AAM-based face contour detection method described above, the most important process is how to estimate the initial value of the entire face AAM model parameter vector from the face AAM model parameter vector.

In the present invention, two model parameter vectors are correlated, and the correlation is solved by an optimization problem.

Assume that there are p parameters in the face AAM of the model face image and q parameters for the entire face AAM (these parameters were determined to include 95% energy).

Now let's express the inner AAM as follows:

(6)

(6)

라 하자. 또, 각 모델 이미지(총 M개)에 대해 얼굴 전체 AAM이 식 (5)로 모델링 될 때 각 얼굴 전체 AAM 파라미터 벡터를

이라 하자.

와

는 서로 관계가 있음을 알고 있다. 이제

와

에 대해 다음의 선형 관계Now, when modeled with Eq. (6) for each model image (total M totals), a vector of AAM parameters inside each face

Let's do it. In addition, for each model image (M total), when the full face AAM is modeled by Eq.

Let's say

Wow

Know that they are related to each other. now

Wow

The following linear relationship for

(7)

(7)

가 거의

에 대해 변화가 별로 없다고 가정하자, 즉

. 식 (7)을 모두 만족시키는 최적해

는

을 행렬

로 표현하고,

을 행

렬 로 표현하기로 하면, 즉Assume that is true. However, in addition

Almost

Suppose there is not much change for, i.e.

. Optimum solution to satisfy all of equation (7)

Is

Matrix

Expressed as

Row

If we choose to represent it as a column,

(8)

(8)

을 구할 수 있다.At this time, solve the following equation (9)

Can be obtained.

(9)

(9)

는

을 최소화하는

이다).(In other words,

Is

To minimize

to be).

에 대해 얼굴 전체 AAM 모델 기반 얼굴 윤곽 검출을 위한 초기 파라미터 벡터 값

는

로 추정한다.Intra-face parameter vector obtained after detecting intra-face AAM model-based intra feature points in the input image

Initial Parameter Vector Values for Face Contour Detection Based on Full Face AAM Model

Is

Estimate as

I. Facial contour detection step

Procedures 1) and 2) of the face contour detection step are the same as 1) and 2) of the face model configuration and relationship extraction step, and are omitted from 3).

3) Matching with the internal AAM model 4) Estimation of the initial value of the entire face AAM model parameter vector through the relation matrix 5) Using the estimated initial value and full face contour detection using full face AAM matching

3) Match with the face AAM model

In the input face image, first, an intra-face AAM model parameter vector that best matches the inside of a face is obtained for the inside of the face using the intra-face AAM model (Equation 6).

4) Estimation of initial values for the full face AAM model parameter vector using the relation matrix.

와 가-5)에서 구한 관계 행렬

을 이용하여

로 얼굴 전체 AAM 모델 파라미터 벡터에 대한 초기값

을 추정한다.Face AAM parameter vector obtained in the previous 3)

Relation matrix obtained from and A-5)

Using

Initial value for full face AAM model parameter vector

Estimate

5) Full face contour detection using estimated initial value and full face AAM matching

을 이용하여 AAM 정합 단계에서 얼굴 윤곽 (얼굴 전체 특징점들) 검출 수행Now, the initial value for the full face AAM model matching obtained in 4)

Face detection (Face Total Feature Points) in AAM Registration

The facial contour detection method using the progressive AAM provided by the present invention is more robust to the initial condition, posture, lighting, complex background, and the like than the conventional AAM-based facial contour detection method, thereby providing an advantage that the success rate is further improved. These results enable better development in many applications that require face recognition, face-based user interfaces, driver observation, animation, and other face modeling.

Claims (6)

Flowchart of facial contour model construction and relationship extraction and facial contour detection The method of claim 1, wherein a two-step facial AAM model is constructed, such as a face inside and a face AAM. 2. The method of claim 1, wherein the relationship matrix between the intra-face AAM model parameter vector and the entire face AAM model parameter vector is extracted. The method of claim 1, wherein an internal parameter vector is obtained by matching with an internal AAM model. The method of claim 1, wherein the initial value estimation method is performed for matching the entire face AAM model using a relation matrix. The method of claim 1, wherein the face full feature points are extracted by matching with the model through full face AAM matching.

Images