KR20130121360A - Optimal gradient pursuit for image alignment - Google Patents

Abstract

An image alignment method is disclosed. On that embodiment, the method comprises; a step which obtains a face image of a person; a step which uses a discriminative face alignment model in order to fit a generic alignment model on the face image for easily performing position determination of face characteristics. The discriminative face alignment model is able to comprise a producing shape model component and a discriminative configuration model component. Furthermore, the discriminative configuration model component is able to be trained in order to estimate a score function capable of minimizing an angle between vectors instructing a gradient direction and a ground truth shape parameter. An additional method, a system, and manufactures are also disclosed.

Description

Optimal Gradient Tracking for Image Alignment {OPTIMAL GRADIENT PURSUIT FOR IMAGE ALIGNMENT}

The present invention was made with government support under license number 2007-DE-BX-K191 awarded by the National Institute of Justice. The Government has certain rights in the invention.

This disclosure relates generally to image alignment and, in some embodiments, to techniques for aligning face images.

Model-based image registration / alignment is an important topic in computer vision, and the model is modified to minimize the distance to the image. In particular, face alignment is important because it enables various practical capabilities (eg, facial feature detection, posture correction, and facial animation) and presents scientific challenges due to facial appearance changes in posture, illumination, facial expression, and occlusion. Previous techniques include an Active Shape Model (ASM) that fits a statistical shape model to a target class. ASM extends to the Active Appearance Model (AAM) used for face alignment. During the base model fitting, the mean squared error between the contour example synthesized from the contour model and the warped contour from the input image is minimized by iteratively updating the shape and / or contour parameters. Although AAM may perform reasonably well during acquisition and fitting of a small set of subjects, performance degrades rapidly when it is trained on a large dataset and / or fitted to a subject that is not seen during model acquisition.

In addition to generation model-based approaches such as AAM, there is also a differential model-based sorting approach. The Boosted Appearance Model (BAM) uses the same shape model as AAM, but also uses a whole other appearance model that is originally a two-class classifier and is discriminated from accurately and incorrectly warped sets of images. During model fitting, BAM aims to maximize the classifier score by updating shape parameters along the gradient direction. Although BAM represents a better generalization of fitting for invisible images compared to AAM, one potential problem is that acquired binary classifiers cannot guarantee concave score surfaces while perturbing shape parameters. In other words, moving along the gradient direction does not always improve alignment. The Boosted Ranking Model (BRM) mitigates this problem by convexing through learning. Using a warped image pair where one side is better aligned than the other side, the BRM learns a score function that attempts to accurately place the two warped images within every training pair. Although BRM may provide some benefit over previous techniques, other improvements in image alignment may be achieved as described below.

Certain aspects of the same scope as the originally claimed invention are described below. It is to be understood that this aspect is only introduced to provide the reader with some form of a brief summary that the various embodiments of the presently disclosed subject matter may take, and it is to be understood that this aspect is not intended to limit the scope of the invention. Indeed, the invention may encompass a variety of aspects that may not be described below.

Embodiments of the presently disclosed subject matter may relate generally to image alignment. In one embodiment, the method includes obtaining a face image of a person and using a differential face alignment model that aligns a generic face mesh to the face image to facilitate positioning of facial features of the face image. It includes. The differential face alignment model may include a generated shape model component and a differential appearance model component. The differential appearance model component may be trained with training data to estimate a score function that is a function of the shape parameter of a given image and minimizes the angle between the gradient direction of the score function for the shape parameter and the ideal alignment movement direction for the shape parameter. .

In another embodiment, a system includes a memory device having a plurality of stored routines and a processor configured to execute the plurality of stored routines. The plurality of stored routines may be configured to access a set of training images, and the appearance model using the training image set to learn an alignment score function that minimizes the angle between the gradient direction of the alignment score function and the ideal direction of movement for the desired alignment. It may include a routine configured to train.

In a further embodiment, the article of manufacture comprises one or more non-transitory computer readable media having executable instructions stored thereon. Executable instructions may include instructions adapted to access an image comprising a human face and instructions adapted to align the human face using a differential face alignment model. The differential face alignment model may include a differential appearance model trained to estimate an alignment score function that minimizes the angle between the gradient direction of the alignment score function and the vector indicated in the maximum direction of the alignment score function.

Various improvements of the aforementioned features may be in various aspects of the subject matter described herein. In addition, other features may likewise be incorporated into these various aspects. These improvements and additional features can be present individually or in any combination. For example, various features discussed below with respect to one or more of the illustrated embodiments may be incorporated into any of the embodiments described herein, alone or in any combination. Moreover, the brief summary introduced above is merely intended to familiarize the reader with certain aspects and contexts of the subject matter disclosed herein without limitation to the claimed subject matter.

These, other features, aspects, and advantages of the present invention will be better understood when the following detailed description is read with reference to the accompanying drawings, in which like characters indicate like parts throughout the figures.
1 is a face shape template according to an embodiment of the present disclosure.
2 shows an example of a concave alignment score function learned through BRM.
3 illustrates an alignment score function in which the gradient direction is closely aligned with the ideal direction of movement according to embodiments herein.
4 is an example of a warped observation image and a face image using a face shape template according to an embodiment of the present disclosure.
5 is an example of a warped face image with feature parameterization according to an embodiment of the present disclosure.
6 shows an example of a rectangular feature type that may be used by an appearance model according to an embodiment of the present disclosure.
7 is an example of a feature template according to an embodiment of the present disclosure.
8 depicts the overall process of estimating an alignment score function in accordance with an embodiment herein.
9 and 10 illustrate the top 15 Haar features selected by a learning algorithm according to embodiments herein.
FIG. 11 is a spatial density map of the top 100 Har features selected by the acquisition algorithms of FIGS. 8 and 9 according to embodiments herein.
12-14 are exemplary images from three datasets according to embodiments herein.
15 is a graph comparing the ranking performance of the acquisition algorithm of the embodiments herein with that of the BRM.
16 is a graph comparing the angular estimation performance of the acquisition algorithm of the embodiments herein with that of the BRM.
17 is a graph comparing the alignment speed performance of the acquisition algorithm of the embodiments herein with that of the BRM.
18 is an example of a face analysis process according to an embodiment of the present disclosure.
19 is a block diagram of a processor-based device or system described herein and providing functionality in accordance with embodiments herein.

One or more specific embodiments of the presently disclosed subject matter will be described below. In an effort to provide a concise description of this embodiment, not all features of an actual implementation can be described herein. In the development of any such real implementation, as in any engineering or design project, a number of implementation specific decisions achieve the developer's specific goals, such as compliance with system-related and business-related constraints that may change from one implementation to the other. It must be understood what must be done to do so. Moreover, while such development efforts can be complex and time consuming, it should nevertheless be understood that it is a procedure to undertake design, fabrication, and manufacture for those skilled in the art having the benefit of this specification. When introducing elements of the various embodiments of the present technology, the articles “uh”, “un”, “the” and “the” are intended to mean that one or more of the elements are present. The terms "comprising", "including", and "having" are intended to include and mean that there may be additional elements other than the listed elements.

Image alignment is the process of moving and transforming a landmark-based generic mesh into an image (eg, a face image) in order for the image feature (eg, face feature) to be accurately positioned. Some alignment models include shape model components and appearance model components. Given an image, landmark points can be located to quantize the shape of the image. In facial image alignment, for example, the shape model may include landmark points corresponding to facial features (eg, nose tip, mouth, etc.). Exemplary mean shape 10 may include a number of triangles 12 defined by landmark points 14 and line segments 16, as shown in FIG. 1.

The appearance model may typically include a learned alignment score function, as shown overall in FIGS. 2 and 3. An example of an alignment score function acquired through BRM is generally illustrated, such as graph 20 of FIG. 2. In this concave function, the ground-truth shape parameter 22 represents the maximum value 24 of the function (ie the required alignment), while each line 26 is at a different point on each line 26. For points of the same size. The scores for the various perturbed shape parameters 28 are shown by elements 30 having a gradient direction 32. However, the gradient direction 32 in the BRM is relatively large angle 36 relative to the vector 34 indicating the ground truss shape parameter 22 (ie, the value 24) starting from the current shape parameter element 30. Can still have). Thus, although the shape parameters in the BRM can be updated along the gradient direction 32, the alignment process in the BRM can take a convolutional path during optimization due to the relatively large angle 36. This not only increases the chance of divergence but also slows alignment.

To address this issue, one embodiment of the present technology instead uses the Optimal Gradient Tracking Model (OGPM) described below to learn a differential alignment model that also includes shape and appearance model components. Using the same shape representation as BAM and BRM, the acquisition of the OGPM appearance model component, which is also an alignment score function, is formulated for a very different purpose. In particular, as represented entirely by the graph 40 in FIG. 3, the appearance model has a gradient 32 at various perturbed shape parameters 28 (represented graphically by reference numeral 30) in which the ideal direction of movement ( In other words, it is directed to learning an alignment score function having a minimum angle 36 with respect to the vector 34 directly indicating the ground truss shape parameter. The score function may comprise or consist of a set of weak functions, each of which acts on one local feature in the warped image domain. The objective function is formulated such that each weak function can be estimated in incremental fashion from a large pool of feature candidates. During model fitting, given an image with initial shape parameters, gradient rise is performed by updating the shape parameters in the gradient direction, which is due to the optimization of the angle 36 between the gradient 32 and the vector 34 in OGPM. It is expected to be more similar to the ideal direction of travel. Further details of the currently disclosed alignment model are provided below. Although certain embodiments regarding face models and face alignment are described below for explanation, it is again noted that the use of model and alignment techniques in other image backgrounds (ie non-faces) is also faced.

Face  Model

Similar to BAM and BRM, an embodiment face model consists of or includes a generated shape model component and a differential appearance model component. For shape models, it is noted that landmark-based shape representation is a popular way of describing the face shape of an image. That is, a set of 2D landmarks {x _i , y _i } _{i = 1, ..., v} may be placed on top of important facial features such as, for example, the corners of the eyes, the corners of the mouth, and the tip of the nose. This landmark linkage forms the shape observation of the image s = [x ₁ , y ₁ , x ₂ , y ₂ , ... x _v , y _v ] ^T. Given a facial database where each image is manually classified as a landmark, the entire shape observation set can be treated as training data for the shape model. In one embodiment, the shape model may be a point variance model (PDM) learned through principal component analysis (PCA) on the observation set. Thus, the learned production PDM can represent a particular shape case in the following equation:

Where s ₀ and s _i are the average shape and i ^th shape basis resulting from the PDM acquisition, respectively. The shape parameter may be provided by p = [p ₁ , p ₂ , ..., p _n ] ^T. Similar to the shape components of the AAM, the first four shape bases can be trained to exhibit overall translation and rotation, while the remaining shape bases can exhibit non-rigid deformation of the facial shape.

As shown in FIG. 4, the warping function 48 from the average shape coordinate system to the coordinates in the image observation 52 is defined as a distinct affine warp:

(2)

(2)

Where (x ⁰ , y ⁰ ) is the pixel coordinate 46 in the average shape domain, and a (p) = [a ₁ (p) a ₂ (p)] is the pair of triangles in s ₀ and s (p), respectively Is the unique 3x2 affine transformation matrix that relates Given the shape parameter p, a (p) can be calculated for each triangle 12. However, since the knowledge of the triangle to which each pixel (x ⁰ , y ⁰ ) belongs is known a priori, the warp can be efficiently performed through a simple table lookup. Using this warping function 48, a certain facial image 52 can be warped to an average shape (expressed globally for one pixel with reference numbers 50 and 56) and a reference number from which the appearance model is learned (54). This results in a shape normalized face image I (W (x; p)) expressed overall at 58.

One embodiment of the appearance model may be better understood with reference to FIGS. 5-7. In particular, FIG. 5 shows an example of a warped image 70 having a parameterized feature 72. 6 shows five feature types 74 (classified separately as feature types 76, 78, 80, 82, and 84) that may be used by the appearance model. In addition, FIG. 7 collectively shows conceptual image template A (reference numeral 92).

)의 수집에 의해 설명된다. 일실시예의 국부 특징은 하르 라이크(Haar-like) 직사각형 특징(예를 들어, 특징(72))일 수 있으며,　그것은 계산 효율에 대해(예를 들어, 통합 이미지 기술로 인해) 이득을 제공할 수 있다. 직사각형 특징은 다음 식으로 계산될 수 있다:In one embodiment, the appearance model is composed of m local features (

) Is explained by the collection. The local feature of one embodiment may be a Haar-like rectangular feature (eg, feature 72), which may provide a gain in computational efficiency (eg, due to integrated image technology). have. Rectangular features can be calculated by the following equation:

(3)

(3)

Where A is an image template 92. The dot product between the template and the warped image is like calculating a rectangular feature using an integrated image. As shown in Figure 5, the image template A can be parameterized by (α, β, γ, δ, τ), where (α, β) is the upper left corner, and γ and δ are the widths. And height, τ is a feature type 74.

Learning to align

By introducing an appearance model representation, we now begin to train the appearance model of the present technology. In one embodiment, the appearance model may include or consist of an alignment score function used during the model fitting step. First, p can be represented by the shape parameter of a given image representing the current alignment of the shape model of equation (1). In one embodiment, the goal of appearance model learning can be presented as follows: From the classified training data, the score function F (p) is adjusted such that when maximized for p, it is a shape parameter of the correct alignment. We aim to learn. In particular, using this goal, if p ₀ is a shape parameter corresponding to the correct alignment of the image, then F should be:

(4)

(4)

Given this equation, F (p) can be optimized through gradient rise. That is, by assuming that F is differential, the shape parameter can be updated repeatedly with each alignment iteration starting from the initial parameter p ⁽⁰⁾ ,

(5)

(5)

)가 미리 정의된 임계값보다 작으면 성공으로 고려된다.Is the step size. When the alignment process converges after k iterations, the alignment is at the Euclidean distance (

) Is considered successful if it is less than a predefined threshold.

가 형상 파라미터(p)의 이동 방향을 지시하는 것이 명백하다. 그러한 이동의 최종 목적지가 p₀이기 때문에, 이상적인 이동 방향은 p로부터 시작되는 p₀를 지적하는 벡터이어야 하며, 그것은

로 표시된다:From equation (5),

Is obviously indicative of the direction of movement of the shape parameter p. Since the final destination of such movement is p ₀ , the ideal direction of movement should be a vector pointing to p ₀ starting from p, which is

Is indicated by:

(6)

(6)

의 반대 방햐이며, 즉

이다. 따라서, 스코어 함수(F)의 습득 동안,

는 가능한 한 이상적인 이동 방향(

)과 같은, 또는 등가적으로 가능한 한 최악의 이동 방향(

)과 같지 않은 방향을 갖는 것이 바람직하다. 특히, 분류자를 정의하면Similarly, the worst direction of travel

Is the opposite of

to be. Thus, during the acquisition of the score function F ,

Is the ideal direction of movement (

), Or equivalently, the worst possible direction of movement (

It is desirable to have a direction that is not equal to). In particular, if you define a classifier

(7)

(7)

It is the cosine response of the inner product between two unit vectors and also the angle between these two vectors, where

(8)

(8)

Indeed, it is difficult to predict that H ( p ) can always be equal to 1 or -1 as shown in the above formula. Thus, the objective function for learning the H classifier can be formulated as

(9)

(9)

)은 그것이 또한

로부터의 제약을 나타낼 수 있으므로 사용된다. 이제부터,

는 명확화를 위해

로 단순화될 것이다. 이 목적 함수는 본래 그 그래디언트 방향이 모든 트레이닝 데이터에 대한 모든 가능한 형상 파라미터(p)에서 이상적인 이동 방향에 대해 최소 각도를 갖도록 함수(F)를 추정하는 것을 지향한다.Where only the ideal direction of travel (

) It also

Used because it can represent constraints from From now on,

For clarity

Will be simplified. This objective function originally aims to estimate the function F such that the gradient direction has a minimum angle with respect to the ideal direction of movement at all possible shape parameters p for all training data.

In one embodiment, a solution for minimizing the objective function 9 may be provided in the manner shown in FIG. 8 and as described below. First, suppose that the sort score function uses a simple additional model:

(10)

(10)

)에 작용하는 위크 함수이다. 그러므로, F의 그래디언트는 또한 추가적인 형태이다:

. 이것을 식　(7)로 결합함으로써, 다음 식을 갖는다:Where f _i (p) is a rectangular feature (

Is a wick function. Therefore, the gradient of F is also an additional form:

. By combining this with Formula (7), it has the following formula:

(11)

(11)

Given the fact that the H function can be recorded in a circular fashion, incremental estimation can be used to minimize the objective function 9. That is, by defining a training sample set and hypothesis space from which rectangular features can be selected, each weak function f _i can be estimated continuously and added incrementally to the target function F. Further details of exemplary portions of the learning process of one embodiment are described below.

In one embodiment, the training sample is an N-dimensional warped image I (W (x; p)). Given a face database {I _i } _{i∈ [1, K]} with manually classified landmarks {s _i } for each face image I _i , equation (1) is a ground truss shape parameter. (p _{0 , i} ) can then be used to synthesize a number of "inaccurate" shape parameters {p _{j , i} } _{j ∈ [1, U]} by random perturbation. Equation (12) below describes an example of perturbation, where v is an n-dimensional vector with each element uniformly distributed within [-1,1], and μ is the vectorized eigenvalue of all shape bases in the PDM. The perturbation index (σ) is a constant ratio that controls the range of perturbation, and ° represents the entrywise product of two equal length vectors.

(12)

(12)

The warped image I _i (W (x; p _{j , i} )) set may then be treated as a defined training sample (y _i = 1) for acquisition. Along with the ideal direction of movement, this can constitute a training set:

(13)

(13)

In one embodiment, the weak function f _i is defined as follows:

(14)

(14)

)에만 작용하는 것이 바람직할 수 있다. 평균 형상 공간 내에서, 직사각형 특징의 모든 가능한 위치, 사이즈, 및 타입은 가설 공간(F={α, β, γ, δ, τ})을 형성하며, 그것으로부터 최상의 특징은 각 반복으로 선택될 수 있다.Where g _i = ± 1 and the normalization constant ensures that f _i is in the range of [-1,1]. This choice can be based on several considerations. First, f _i must be differentiable because it assumes F is a differentiable function. Second, each function f _i has one rectangular feature (

It may be desirable to act only). Within the average shape space, all possible positions, sizes, and types of rectangular features form the hypothesis space F = {α, β, γ, δ, τ}, from which the best feature can be selected for each iteration. have.

One procedure for learning the alignment score function 10 is provided by Algorithm 1 in the table below:

Also, this algorithm is shown overall in FIG. 8 according to one embodiment, where process 96 estimates an alignment score function based on the set of samples 98 from equation (13) above.

In particular, at process 96, alignment score function F may be initialized at block 100 (corresponding to step 1 in the algorithm above). The weak function f _t may be fitted to block 102 in the manner described in step 3 of the algorithm. It is noted that step 3 in the algorithm is the most computationally intensive step since the entire hypothesis space is thoroughly searched. In step 3, the best feature is selected based on the L ² distance of H relative to 1 rather than the weak classifier in boosting based learning. Classifier function (H) may be updated by f _t in block 104 (corresponding to step 4 in the above algorithm), f _t corresponds to step 5 in alignment score function (F) (the algorithm in block 106 Can be added). Steps 3-5 of the algorithm may be repeated for each t, as represented entirely by blocks 108 and 110 and return loop 112 of FIG. 8 (corresponding to step 2 above). At the conclusion, process 96 may return an estimate of the alignment score function, such as the sum of the weak function set at block 114.

} 세트, 임계값{t_i}, 및 특징 사인{g_i}을 습득하는 것에 대응한다. 실제 구현에서는, g_i=+1, 및 g_i=-1를 각각 설정하고 양쪽 경우에 대한 최적 임계값을 추정할 수 있다. 결국 g_i는 작은 에러를 갖는 경우에 기초해서 설정될 것이다(식　15). 최적 임계값은 에러가 최소화되도록 특징 값의 범위에서 이진 검색에 의해 추정될 수 있다.In essence, learning the score function F is a feature {

} Corresponds to learning a set, a threshold {t _i }, and a feature sine {g _i }. In a practical implementation, g _i = + 1 and g _i = −1 can be set respectively and the optimal threshold values for both cases can be estimated. Eventually g _i will be set based on the case with small error (Eq. 15). The optimal threshold may be estimated by binary search over a range of feature values to minimize errors.

, g_i,t_i)}_i=1,...,m) 세트는, 형상 모델({s_i}_i=1,...,n)과 함께, 여기서 최적 그래디언트 추적 모델(OGPM)로 지칭된다. 일실시예에서 습득 알고리즘에 의해 선택되는 상부 15개의 특징은 도 9 및 10에 도시되어 있다. 특히, 도 9는 습득 알고리즘에 의해 선택되는 상부 5개의 하르 특징(120)의 초상(118)을 제공하고, 도 10은 습득 알고리즘에 의해 선택된 다음 10개의 하르 특징(126)의 초상(124)을 제공한다. 동일 실시예에서의 습득 알고리즘에 의해 선택되는 상부 100개의 하르 특징의 공간 밀도 맵(130)은 도 11에도 제공되어 있다.　다수의 선택된 특징은 얼굴 특징의 경계와 정렬되는 것이 주목된다.Final triple ({(

, g _i , t _i )} _{i = 1, ..., m} ) The set, together with the shape model ({s _i } _{i = 1, ..., n} ), is the optimal gradient tracking model (OGPM) It is referred to. The top 15 features selected by the acquisition algorithm in one embodiment are shown in FIGS. 9 and 10. In particular, FIG. 9 provides a portrait 118 of the top five har features 120 selected by the learning algorithm, and FIG. 10 shows a portrait 124 of the next ten har features 126 selected by the learning algorithm. to provide. A spatial density map 130 of the top 100 Har features selected by the acquisition algorithm in the same embodiment is also provided in FIG. It is noted that the number of selected features is aligned with the boundary of the facial feature.

Face alignment

In one embodiment, the OGPM may be fitted to the initial shape parameter p ⁽⁰ ) (in the 0th iteration) and the face of the given image I in the manner described below. As shown in equation (5), the alignment can be performed repeatedly by using a gradient rising approach. From equations (3), (10), and (14), it can be seen that the differential coefficient of F with respect to p is

(16)

(16)

는 p에서 평가된 워프의 야코비안이다. BAM에 대한

의 정렬 절차, 계산 복잡성, 및 효율적인 구현에 관한 논의는 명칭이 "Discriminative Face Alignment"(IEEE Trans. On Pattern Analysis and Machine Intelligence, 31(11):1941-1954, November 2009)인 Xioaming Liu에 의한 간행물에서 발견될 수 있다. 그러나 BAM 기반 피팅과 대조적으로, 본 기술은 간단한 고정 상수라기보다는 오히려 라인 검색을 통해 동적으로 결정되는 스텝 사이즈(λ)를 사용한다. 즉, 각각의 반복에서, 어떤 범위 내의 최적 λ는 업데이트된 형상 파라미터가 현재의 스코어 함수 값(F(p))을 최대로 증가시킬 수 있도록 요구된다.Where ▽ I is the gradient of the image evaluated at W (x; p),

Is the Jacobian of warp evaluated at p. For BAM

A discussion of the alignment procedure, computational complexity, and efficient implementation of the system is published by Xioaming Liu, entitled "Discriminative Face Alignment" (IEEE Trans.On Pattern Analysis and Machine Intelligence, 31 (11): 1941-1954, November 2009). Can be found in However, in contrast to BAM-based fitting, the technique uses a step size λ that is determined dynamically through line search rather than a simple fixed constant. That is, in each iteration, an optimal lambda within a range is required so that the updated shape parameter can increase the current score function value F (p) to the maximum.

Experiment result

The following experimental results were obtained using an experimental dataset comprising 964 images from three public databases, ND1, FERET, and BioID database. Each of the 964 images contains 33 manual classification landmarks. To speed up the training process for this experiment, the image set is down sampled so that the face width is approximately 40 pixels across this set. Sample images 134 of the ND1, FERET, and BioID databases are illustrated in FIGS. 12, 13, and 14, respectively. As shown in Table 1 below, all images are divided into three non-overlapping datasets. Set 1 included 400 images (one image per subject) from two databases. Set 2 included 334 images from the same subject but included different images from the ND1 database in set 1. Set 3 included 230 images from 23 subjects in a BioID database that was never used in training. Set 1 was used as a training set for model acquisition and all three sets were used to test model fitting. The motivation for such a division was to experiment with various levels of generalization capability. For example, set 2 can be tested with unseen data of the visible subject; Set 3 was tested with invisible data—more challenging cases of invisible subjects and more similar to the outline in practical applications.

In this experiment, the OGPM algorithm described above was compared with BRM based on two considerations. First, the OGPM algorithm can be considered as an extension of BRM. Secondly, BRM has surpassed other discriminatory image alignment techniques such as BAM. During model acquisition, both BRM and OGPM were trained from 400 images in set 1. The BRM used 24000 (= 400 × 10 × 6) training samples synthesized from set 1, where each image synthesized 10 profile lines and each line had 6 regularly spaced samples. In comparison, OGPM used 12000 training samples, where each image synthesized 30 samples according to equation (12). Fewer samples can be used for OGPM because all synthesized samples are randomly developed rather than multiple samples selected from one profile line as in BRM, resulting in good performance achieved with less training samples. Manual classification landmarks in Set 1 images are described in Xiaoming Liu, entitled "Face Model Fitting on Low Resolution Images" (In Proc. Of the British Machine Vision Conference (BMVC), vol. 3, pp. 1079-1088, 2006). Improvements were made using the automated model improvement approach described in the publication. After model acquisition, the shape model component of both BRM and OGPM was PDM with 9 shape bases, and the appearance model (ie alignment score function) had 100 weak classifiers / functions.

The BRM aims to improve the convexity of the score function learned by the correct ranking pairs of warped images. OGPM extends BRM in the sense that the score function should not only be concave, but also have a minimum angle between the vector and the gradient direction pointing to the ground truss shape parameter. Thus, convex is a good metric for evaluating the score function for both BRM and OGPM. Similar to BRM, the convex in the experiment was measured by calculating the percentage of the exact ranking pair of warped images. Given Set 1 and Set 2, two respective pairs of sets were synthesized and the ranking performance of BRM and OGPM was tested. As shown by graph 140 of FIG. 15, the perturbation index σ controls the amount of perturbation of the image pair (see Equation 12). For both sets, it can be seen that, unlike BRM, OGPM has achieved ranking performance very similar to BRM despite the fact that OGPM does not directly use ranking in its objective function. BRM showed slightly better performance when the perturbation was very small (σ = 1). However, this can be largely attributed to the classification of errors in the training data as a small perturbation of the classified landmarks can be treated as a fairly good alignment, which makes the ranking more difficult.

In addition to the convex measurements, we also confirmed the estimation of the angle between the gradient direction and the vector indicating the ground truss shape parameters. Minimization of this angle is the objective function of the OGPM, as represented by the H (p) function. Similar to the ranking experiments described above, given set 1, six warped image sets were randomly synthesized using various perturbation indices σ. Then, for each image in the set, the H (p) score was calculated, and the average score of each set was made in graph 150 of FIG. Similar experiments were done for set 2. Although OGPM and BRM have similar ranking performance, OGPM achieves large function scores and therefore small gradient angles for both sets 1 and 2. This can be used to obtain better alignment score function using the gradient angle directly with the objective function, as done by OGPM, using the ranking performance for the purpose as done by BRM, as done by OGPM. Prove it is.

In the alignment experiments, a model fitting algorithm was run on each image with multiple initial landmarks and the alignment results were evaluated. Initial landmarks were generated by using equation (12), ie, randomly perturbing the ground truss landmarks by a range that was independently constant such that the range was a large number (?) Of the eigenvalues of the shape basis during PDM training. Once the fitting on the image was terminated, the alignment performance was measured by the resulting mean square error (RMSE) between the aligned landmark and the ground truss landmark.

Alignment experiments were performed for all three sets using both OGPM and BRM. Table 2 above shows the RMSE results for the pixels, each element being the average of more than 2000 tests at one particular perturbation index σ. Thus, each image in sets 1, 2, and 3 was tested with 5, 6, and 9 random tests, respectively. OGPM and BRM were tested under the same conditions. For example, both algorithms were initialized with the same random test and the termination conditions were also the same. That is, the alignment iterations ended if the alignment score F (p) could not be further increased, or the landmark difference RMSE between successive iterations was less than a predefined threshold such as 0.05 pixels in the presently described experiment. All.

From Table 2, it can be seen that for all three sets, OGPM was able to achieve better alignment performance than BRM. Note that the performance gain is relatively large, with initial perturbations such as σ = 6 or 8, which is the most challenging case in practical applications. Given the fact that the test image was of very low resolution, this represents a substantial performance improvement. Comparing among the three data sets, the performance gain in the training set (set 1) was greater than the other two data sets.

One advantage of small gradient angles is the ability to converge with less iterations during alignment. In FIG. 17, a histogram 160 is provided that shows the number of iterations required for the experiment to converge to set 3 when OGPM and BRM are σ = 8. On average, it can be seen that OGPM can converge faster than BRM. In this experiment, the repeat average number of OGPM was 5.47, while that of BRM was 6.40. Similarly, for set 1, the repeat average number of OGPMs when σ = 8 was 5.08 and that of BRM was 6.09.

The image alignment techniques described herein can be used with a number of other processing techniques to achieve the desired result. For example, as shown entirely in FIG. 18, in accordance with one embodiment, the disclosed image alignment technique may be used in face analysis process 170. As one example, such process 170 can include receiving an image and detecting one or more faces in the image, as shown entirely by blocks 172 and 174. The detected faces may be aligned, such as by the presently disclosed technique, as shown entirely by block 176. The aligned face may then be analyzed at block 178, for example, for face recognition or pose estimation that identifies the person in the image by comparing the aligned face with reference data.

Finally, it is noted that the functionality described herein (eg, image detection, alignment, and analysis) may be performed by a processor based system such as a computer. An example of such a system is provided in FIG. 19 according to one embodiment. The processor-based system 184 shown may be a general purpose computer, such as a personal computer, configured to execute various software, including software that implements all or part of the functionality described herein. In the alternative, the processor-based system 184 may include a mainframe computer, distributed computing system, or application specific computer configured to implement all or a portion of the present technology based, among other things, on specialized software and / or hardware provided as part of the system. It can include a workstation. In addition, processor-based system 184 may include a single processor or multiple processors to facilitate the implementation of the presently disclosed functionality.

In general, processor-based system 184 may include a microcontroller or microprocessor 186, such as a central processing unit (CPU), that may execute various routines and processing functions of system 184. For example, the microprocessor 186 may execute various operating system instructions as well as software routines configured to accomplish certain processes. The routine may include memory 188 (e.g., random access memory (RAM) in a personal computer) or one or more mass storage devices 190 (e.g., internal or external hard drive, solid state storage, optical disk, Magnetic storage device, or any other suitable storage device), or may be stored in or provided by an article of manufacture including one or more non-transitory computer readable media. Microprocessor 186 also processes data provided as input to various routines or software programs, such as data provided as part of the present technology in computer-based implementations.

Such data may be stored in or provided by memory 188 or mass storage device 190. In the alternative, such data may be provided to the microprocessor 186 through one or more input devices 192. The input device 192 may include a manual input device such as a keyboard, a mouse, or the like. Input device 192 may also be a wired or wireless Ethernet card, wireless network adapter, or any of a variety of ports or devices configured to facilitate communication with other devices via any suitable communication network 198, such as a network local area network or the Internet. It may include a network device such as. Through such a network device, system 184 can exchange data and communicate with other networking electronic systems, whether in proximity to or away from system 184. The network 198 may include various components that facilitate communication with switches, routers, servers or other computers, network adapters, communication cables, and the like.

Results generated by the microprocessor 186, such as results obtained by processing data in accordance with one or more stored routines, may be provided to the operator through one or more output devices, such as display 194 or printer 196. . Based on the displayed or printed output, the operator may request additional or optional processing and provide additional or optional data, such as through input device 192. Communication between the various components of the processor-based system 184 may be accomplished through a chipset and one or more buses or interconnects that electrically connect the components of the system 184.

Technical effects of the present invention include improvements in speed, efficiency, and accuracy for face and non-face image alignment. Although only certain features of the invention have been illustrated and described herein, many modifications and variations will occur to those skilled in the art. Therefore, it is to be understood that the appended claims are intended to cover all such modifications and variations as fall within the true spirit of the invention.

Claims (20)

Obtaining a face image of a person,
Through software executed by a system processor, a differential face alignment model is used that aligns a generic face mesh to the face image to facilitate positioning of facial features of the face image, wherein the differential face alignment model is generated. A shape model component and a differential appearance model component, the differential appearance model component being a function of a shape parameter of a given image and determining the angle between the gradient direction of the score function for the shape parameter and the ideal alignment movement direction for the shape parameter. Training with training data to estimate a score function to minimize
Way.
The method of claim 1,
The differential appearance model component is trained with training data to estimate the score function through an objective function defined by the following equation,

For all shape parameters p of the training data, F is the score function,

Is a classifier such as a dot product between two unit vectors, each representing the gradient direction and the ideal alignment movement direction.
Way.
3. The method of claim 2,
Minimizing the objective function includes summing a weak function each acting on each single rectangular face feature
Way.
The method of claim 1,
Performing face recognition on the face image after alignment through additional software executed by the processor.
Way.
The method of claim 1,
Acquiring the face image of the person includes analyzing image data to detect the face of the person.
Way.
The method of claim 1,
Training the differential appearance model with the training data;
Way.
The method according to claim 6,
Optimizing the score function through gradient rise
Way.
The method according to claim 6,
Calculating ground truss shape parameters for each face image of the plurality of face images;
Synthesizing a plurality of modified shape parameters for each face image by random perturbation of the ground truss shape parameters.
The method of claim 8,
The training data includes a warped image set based on the modified shape parameter, and an ideal direction of movement for the warped image.
Way.
A memory device in which a plurality of routines are stored;
A processor configured to execute the plurality of routines stored in the memory device,
The plurality of routines,
A routine configured to access a set of training images,
A routine configured to train an appearance model using the set of training images to learn an alignment score function that minimizes the angle between the gradient direction of the alignment score function and the ideal direction of movement for the desired alignment.
system.
11. The method of claim 10,
The plurality of routines,
A routine configured to determine ground truss shape parameters for each image of the training image set;
A routine configured to synthesize a plurality of shape parameters deviating from the ground truss shape parameters;
system.
The method of claim 11,
The routine configured to synthesize the plurality of shape parameters includes a routine configured to synthesize the plurality of shape parameters via random perturbation.
system.
11. The method of claim 10,
A routine configured to train the appearance model initializes the alignment score function, iteratively estimates a plurality of weak functions that operate on a single rectangular feature, and incrementally adds an estimate of the plurality of weak functions to the alignment score function. A routine for learning the alignment score function by
system. The method of claim 13,
Iteratively estimating the plurality of weak functions includes fitting the weak functions of the plurality of weak functions based on the least square distance of the classifier function with respect to one.
system.
11. The method of claim 10,
The training image set includes a set of face images, and the routine configured to access the training image set includes a routine configured to access the face image set, and the routine configured to train the appearance model using the training image set. Includes a routine configured to train the appearance model using the face image set
system.
11. The method of claim 10,
The memory device includes at least one of an optical disk, a random access memory, or a hard drive.
system.
One or more non-transitory computer readable media having executable instructions stored thereon;
The executable instruction is
Instructions to access an image containing a human face,
The human face may be modified using a differential face alignment model that includes a differential appearance model trained to estimate an alignment score function that minimizes the angle between the gradient direction of the alignment score function and the vector indicated in the maximum direction of the alignment score function. With instructions to align
Manufactured goods.
The method of claim 17,
The one or more non-transitory computer readable media includes a plurality of non-transitory computer readable media having at least collectively stored therein the executable instructions.
Manufactured goods.
The method of claim 17,
The one or more non-transitory computer readable media includes an optical disk, magnetic disk, solid state device, or some combination thereof.
Manufactured goods.
The method of claim 17,
The one or more non-transitory computer readable media includes random access memory of a computer.
Manufactured goods.

Images