JP2003208619A - Face description method by use of primary and secondary inherent features and not affected by illumination and visual angle - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method usable in multimedia data base retrieval through the Internet, electronic library, video editing, monitoring/pursuit, and face recognition/collation and capable of interpreting and recognizing human face not affected by illumination. <P>SOLUTION: Secondary inherent feature is used in face description not affected by illumination, and both of primary and secondary inherent features are used in the face description not affected by a visual angle. After normalization and quantization, these features can describe the face effectively and efficiently. Furthermore, quantized inherent feature can be put into a code using a variable length code to reduce the size of face descriptor. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

FIELD OF THE INVENTION The present invention relates to facial description, recognition for content-based image retrieval, facial identification and verification of people for banks, security systems and videophones.
It can be used for surveillance and tracking, electronic libraries, and internet multimedia databases.

[0002]

BACKGROUND OF THE INVENTION Human facial recognition is an active area of computer vision research. Face recognition will play an important role in multimedia database search and many other applications. Significant progress has been made in recent years on the problem of face detection / recognition. Various specific technologies have been proposed. Among these techniques, neural networks, elastic template matching, Karhunen-Loeve expansion, algebraic moments and isopycnic lines are typical methods.

Of these methods, principal component analysis (PC
A) That is, Karhunen-Loeve expansion is an important method.
The eigenface method is derived from PCA, is convenient for calculation, and can be identified with consistent accuracy. According to previous studies, the PCA method naturally separates different kinds of information. The eigenvector with a large eigenvalue obtains information common to a subset of faces, and the eigenvector with a small eigenvalue obtains information unique to each face. Research shows that only information contained in eigenvectors with large eigenvalues can be used universally for new faces that have not yet been learned.

[0004]

The advantages of the eigenface method are:
The eigenvectors with large eigenvalues convey information about the basic shape and structure of the face. This means that the main features of the human face can be described using the features extracted from the eigenvectors with large eigenvalues. However, this is also a weakness of PCA. If we consider only the features extracted from the eigenvectors with large eigenvalues,
It is not possible to obtain facial details that correspond to individual faces. If these individual details of the face can be described together with the common features of the human face, the human face can be described more accurately.

Further, when the image of the face is taken under different illumination conditions and different viewing angles, it is difficult to obtain an appropriate eigenvector representing the human face. In our experience, the conventional eigenface method is not suitable for recognizing a face under different lighting conditions. However, many face images of the same person are taken under different lighting and viewing angle conditions. Facial description, which is independent of lighting and viewing angle, remains an important area to explore.

[0006]

The eigenface method is effective for extracting common facial features such as shape and structure.
A face reconstructed with features from the eigenvectors with high eigenvalues should be obtained in order to obtain the details of the face lost when truncating the eigenvectors with low eigenvalues. The reconstructed facial image can be used to obtain a residual image between the original image and the reconstructed image. These residual faces can be regarded as high pass face images that still contain a wealth of detailed information about the individual faces. To describe the residual face,
The eigenface method can again be applied to these residual faces. The resulting eigenvectors with large eigenvalues reveal common features of the residual face. Using this method, a quadratic eigenvector with a large eigenvalue can be obtained and the corresponding feature can be extracted. According to the experiments by the inventors,
Both primary and secondary eigenfeatures are useful for facial descriptions that are not affected by viewing angle, while secondary eigenfeatures are also useful for facial descriptions that are not affected by illumination. Therefore, the combination of the primary and secondary characteristic features can be used for the facial description that is not affected by the viewing angle, and the secondary characteristic feature can be used for the facial description that is not affected by the illumination. Another method of facial description is to use only primary eigenfeatures. For a face description that is not affected by illumination, the eigenvalue magnitudes are from the top kth to the Nth (0 <k <N) in order to avoid fluctuations in illumination that occur from the top k eigenfaces with eigenvalue magnitudes. Only the characteristic features extracted from the characteristic face in the range of are used. The facial feature that is not affected by the viewing angle may be a characteristic feature extracted from the characteristic face corresponding to the top N characteristic values in size.

Further, when considering the facial description, the symmetry of the face is an important feature. The faces of the majority of people are symmetrical. Therefore, it is possible to use a facial image in which left and right mirror images are formed to represent the same person. As a result, the unique features of the original image and the mirror imaged image can both represent the same person. Therefore, the original image can be represented using a linear combination of the characteristic features of the two images. The characteristic features may be combined with different weights to distinguish between the original image and the mirrored image.

According to a first aspect of the present invention, there is provided a step of obtaining an adjusted secondary eigenfeature of a facial image, a step of quantizing the adjusted secondary eigenfeature, and a step of quantizing the quantized secondary eigenfeature. And a step of selecting features constituting a facial descriptor for describing the face, the method including extracting features for the facial description that are not affected by illumination.

According to a second aspect of the present invention, there is provided a step of obtaining an adjusted secondary eigenfeature of a facial image, a step of quantizing the adjusted secondary eigenfeature, and a step of quantizing the quantized secondary eigenfeature. Illumination-insensitive facial description, comprising: selecting features that make up a facial descriptor for describing a face; and encoding the selected unique features into an illumination-insensitive facial descriptor. Is a method for extracting the features of.

According to a third aspect of the present invention, a step of obtaining an adjusted primary eigenfeature of the face image, a step of obtaining an adjusted secondary eigenfeature of the face image, and quantizing the adjusted primary eigenfeature. Quantizing the adjusted secondary eigenfeatures, and selecting features that make up a facial descriptor for describing a face from the quantized primary and secondary eigenfeatures. This is a method for extracting facial description features that are not affected by the viewing angle.

According to a fourth aspect of the present invention, a step of obtaining an adjusted primary eigenfeature, a step of obtaining an adjusted secondary eigenfeature, a step of quantizing the adjusted primary eigenfeature, and the adjustment Quantizing the secondary eigenfeatures, selecting features that make up a facial descriptor for describing a face from the quantized primary and secondary eigenfeatures, and selecting the selected eigenfeatures. Encoding a facial descriptor that is not affected by a visual angle, and extracting a feature for facial description that is not affected by a visual angle.

The invention according to claim 5 includes the step of obtaining an inner product of the face image and the adjusted secondary eigenface matrix, and the adjusted secondary image of the face image according to any one of claims 1 to 4. This is a method for obtaining the characteristic feature.

The invention of claim 6 includes the step of obtaining an inner product of the facial image and the adjusted primary eigenface matrix, and the adjusted primary eigenfeature of the facial image according to claim 3 or 4. Is a method of asking for.

The invention of claim 7 includes the step of calculating a primary eigenface matrix, and the step of adjusting the primary eigenface matrix, wherein the adjusted primary eigenface matrix of claim 6 is calculated. Is the way to do it.

The invention of claim 8 includes the step of calculating a quadratic eigenface matrix, and the step of adjusting the quadratic eigenface matrix, wherein the adjusted quadratic eigenface matrix of claim 5 is included. This is a method of calculating a matrix.

According to a ninth aspect of the present invention, a step of obtaining primary eigenfeatures of a training face image, a step of configuring the primary eigenface as a two-dimensional array of original images, and a mirror-imaged eigenface of the two-dimensional array. To get
Weighting the mirrored eigenface image, adding the weighted eigenface image to the primary eigenface, and re-priming the primary eigenface to obtain a one-dimensional adjusted primary eigenface. Configuring, and normalizing the primary eigenface,
Obtaining the adjusted primary eigenfeature weights for distance calculation, multiplying the adjusted primary eigenfeature weights for distance calculation to the adjusted primary eigenface matrix, and quantizing the primary eigenface matrix 8. The method of adjusting the primary eigenface of claim 7, including:

According to a tenth aspect of the present invention, a step of calculating a primary eigenface matrix, a step of obtaining primary eigenfeatures from a learning face image, a step of calculating a pseudo inverse matrix of the primary eigenface matrix, and the primary Calculating a reconstructed primary facial image by multiplying the eigenfeatures by the inverse of the primary eigenface matrix; and obtaining a secondary residual image by subtracting the reconstructed primary facial image from the original image. Calculating a eigenvector of the secondary residual image to obtain a quadratic eigenface. 9.

According to the invention of claim 11, the step of obtaining the secondary eigenfeatures of the training face image by calculating the inner product of the face image and the secondary eigenface, and reshaping the secondary eigenface into the shape of the original image. Then, a step of obtaining a mirror-formed eigenface image symmetrically, a step of weighting the mirror-imaged eigenface image and adding them to the corresponding eigenimage of the secondary eigenface image, the primary Reconstructing the shaped quadratic eigenfaces into the original image to obtain the original adjusted quadratic eigenfaces, normalizing the adjusted quadratic eigenfaces, and two for distance calculation. Obtaining the weights of the next eigenfeatures, multiplying the weights of the quadratic eigenfeatures for distance calculation to the adjusted quadratic eigenface matrix, and quantizing the quadratic eigenface matrix 9. The method of adjusting a secondary eigenface according to claim 8, further comprising the step of:

According to a twelfth aspect of the present invention, the step of obtaining the maximum value and the minimum value of the adjusted primary eigenface matrix and the quantization by dividing the maximum value and the minimum value into a plurality of quantization levels. 10. The adjustment according to claim 9, comprising: obtaining a step, dividing the adjusted quadratic eigenface matrix by the quantization step, and rounding the divided value to the nearest integer. This is a method of quantizing the generated primary eigenface matrix.

According to a thirteenth aspect of the present invention, there is provided a step of obtaining a maximum value and a minimum value of the adjusted quadratic eigenface matrix, and dividing the maximum value and the minimum value into a plurality of quantization levels. 12. The method of claim 11, comprising: obtaining a quantization step, dividing the adjusted quadratic eigenface matrix by the quantization step, and rounding the divided value to the nearest integer. Is a method for quantizing the adjusted quadratic eigenface matrix of.

According to a fourteenth aspect of the present invention, there is provided a step of obtaining a reconstructed adjusted quadratic eigenface by multiplying the quantized quadratic eigenface matrix by the quantization step of the eleventh aspect. Multiplying each column of the adjusted secondary eigenfaces by each row of the facial image to obtain secondary eigenfeatures.
It is a method of obtaining the adjusted quadratic characteristic feature described in the section.

According to a fifteenth aspect of the present invention, a step of obtaining a restored adjusted primary eigenface by multiplying the quantized primary eigenface matrix by the quantization step of the twelfth aspect, and the restored adjustment. Obtaining the adjusted primary eigenfeatures of any one of claims 3 or 4, comprising: multiplying each column of the completed primary eigenfaces by each row of the facial image to obtain the primary eigenfeatures. Is the way.

According to a sixteenth aspect of the present invention, the step of obtaining the maximum value and the minimum value of the adjusted secondary eigenfeatures of the training image, and dividing the maximum value and the minimum value into a plurality of quantization levels. Obtaining a quantized step by doing, dividing the adjusted quadratic eigenfeature by the quantized step, and rounding the divided value to the nearest integer. The method for quantizing the adjusted secondary eigenfeature according to any one of items 1 to 4.

According to a seventeenth aspect of the present invention, the step of obtaining the maximum value and the minimum value of the adjusted secondary eigenfeatures of the training image, and dividing the maximum value and the minimum value into a plurality of quantization levels. To obtain a quantizing step, dividing the adjusted quadratic eigenfeature by the quantizing step, rounding the divided value to the nearest integer, and the minimum standard deviation. Assigning a different number of bits to different eigenfeatures by calculating the rounded log of the standard deviation of the corresponding eigenfeatures in the divided training set, and quantizing said eigenfeatures according to the corresponding bit assignments A method for quantizing a tuned quadratic eigenfeature according to any one of claims 1 to 4, comprising:

According to a eighteenth aspect of the present invention, the step of obtaining the maximum value and the minimum value of the adjusted primary eigenfeatures of the training image, and dividing the maximum value and the minimum value into a plurality of quantization levels. Obtaining a quantized step by dividing the adjusted primary eigenfeature by the quantized step; and rounding the divided value to the nearest integer. The method for quantizing the adjusted primary eigenfeature according to any one of 1.

According to a nineteenth aspect of the present invention, the step of obtaining the maximum value and the minimum value of the adjusted primary eigenfeatures of the training image, and dividing the maximum value and the minimum value into a plurality of quantization levels. To obtain a quantization step, dividing the adjusted primary eigenfeature by the quantization step, rounding the divided value to the nearest integer, and dividing by the minimum standard deviation. Calculating a rounded logarithm of the standard deviation of the corresponding eigenfeatures in the learning set, allocating a different number of bits to the different eigenfeatures, and quantizing the eigenfeatures according to the corresponding bit allocation, A method of quantizing a tuned primary eigenfeature according to any one of claims 3 or 4 including.

The twentieth aspect of the invention includes the steps of: obtaining a standard deviation of the adjusted first-order eigenfeatures of a training face image; and obtaining a weight by extracting a square root of the standard deviation. The method of claim 9, wherein the adjusted primary eigenfeature weights for the calculation are obtained.

According to a twenty-first aspect of the present invention, a step of obtaining a variance of the adjusted secondary eigenfeatures of a training face image and a step of extracting a square root of the variances to obtain the weights of the adjusted secondary eigenfeatures. 12. The method of claim 11, further comprising: obtaining adjusted weights of the secondary eigenfeatures for distance calculation.

The invention of claim 22 is the same as claim 1 or 2.
The step of extracting the eigenfeatures of the face for the face description that is not affected by the illumination by the method described in 1, the step of obtaining the Euclidean distance of the eigenfeatures of the face, and the minimum Euclidean distance indicating the pair of faces that optimally match. And a step of selecting, which is a method of measuring the similarity between faces that are not affected by illumination.

The invention of claim 23 is the invention of claim 3 or 4.
The step of extracting the eigenfeatures of the face for the face description that is not affected by the viewing angle by the method described in 1), the step of obtaining the Euclidean distance of the face features, and the selection of the minimum Euclidean distance indicating the pair of the faces that optimally match. And a step of measuring the degree of similarity between the faces that are not affected by the viewing angle.

The invention of claim 24 is the invention of claim 1 or 2.
Obtaining the quantized eigenfeatures of the training set by the method according to claim 1, classifying the eigenfeatures into groups according to bit allocation, and using the entropy coding method the eigenfeatures having the same bit allocation. Configuring a code table for each of the groups of <RTIgt; of, </ RTI> a method for obtaining a code table for variable coding (VLC) of the illumination independent face descriptors.

The invention of claim 25 is the invention of claim 3 or 4.
Obtaining the quantized eigenfeatures of the training set by the method of claim 1, classifying the eigenfeatures into groups according to bit allocation, and using the entropy coding technique the eigenfeatures having the same bit allocation. Forming a code table for each of the groups, and obtaining a code table for variable coding (VLC) the facial descriptors that are not affected by the viewing angle.

In the twenty-sixth aspect of the present invention, the entropy coding method is a Huff based on a probability of a quantization level.
The code table construction method according to claim 24 or 25, which is a man coding method.

The invention of claim 27 is the code table construction method according to claim 24 or 25, wherein the entropy coding method is an arithmetic coding method based on a probability of a quantization level. .

The invention of claim 28 refers to the code table generated by the method of claim 24 for each quantized characteristic feature, and uses the corresponding code word to perform the quantized characteristic feature. A method for coding a lighting-independent facial descriptor according to claim 1 or 2, including the step of expressing

The invention of claim 29 refers to the code table generated by the method of claim 25 for each quantized eigenfeature, and uses the corresponding codeword to perform the quantized eigenfeature A method of encoding a viewing angle insensitive facial descriptor according to claim 3 or 4, including the step of expressing

According to a thirtieth aspect of the present invention, the step of obtaining the adjusted primary eigenfeatures of the facial image by the method of the seventh aspect, and the quantization of the adjusted primary eigenfeatures by the method of the nineteenth aspect. And a step of selecting a feature to construct a facial descriptor describing a face from the quantized primary eigenfeature, and extracting a feature for general facial description.

According to a thirty-first aspect of the present invention, the step of obtaining the adjusted primary eigenfeatures of the facial image by the method of the seventh aspect, and the quantization of the adjusted primary eigenfeatures by the method of the nineteenth aspect. Selecting a feature to construct a facial descriptor that describes a face from the quantized primary eigenfeatures, and encoding the selected eigenfeatures into a facial descriptor.
This is a method for extracting general facial description features including.

The invention of claim 32 includes the step of selecting eigenfeatures corresponding to the top N eigenvalues of size in order to perform a facial description that is not affected by the viewing angle.
A method of selecting a feature constituting a facial descriptor describing a face from the quantized primary eigenfeatures described in either 0 or 31.

According to a thirty-third aspect of the present invention, there is included a step of selecting eigenfeatures corresponding to the kth to Nth eigenvalues (0 <k <N) in order of magnitude in order to perform a face description that is not affected by illumination. 32. A method according to any of claims 30 or 31 for selecting features comprising a facial descriptor that describes a face from the quantized primary eigenfeatures.

[0041]

DETAILED DESCRIPTION OF THE INVENTION The present invention provides two methods of describing a person's face without being affected by lighting or viewing angles. These can be used for image retrieval (facial example query), personal identification and matching, monitoring and tracking, and other facial recognition applications. To describe facial features, we propose a concept of secondary eigenface based on our knowledge and inference. First, the primary and secondary eigenfaces are derived from the set of learning face images. The facial symmetry feature can then be used to adjust the eigenface. With these adjusted characteristic faces, adjusted primary and secondary characteristic features are obtained. A combination of these features can be used to describe the face. For this description, Euclidean distance can be used to measure similarity.

The image in the middle of FIG. 2 is an image of a digital photograph as it is taken, and is original face images 1a, 2a, 3a, 4a, 5a, 6a which have not been processed. If the original facial image 1a is represented by a one-dimensional vector, Φ
Displayed as ₁ . Similarly, the original facial images 2a, 3a, 4
When a, 5a, and 6a are represented by a one-dimensional vector, Φ ₂ , Φ ₃ ,
Displayed as Φ ₄ , Φ ₅ , and Φ ₆ . In general, the face image ia is represented by the vector Φi.

If the one-dimensional vectors of all six original facial images are added and the total is divided by 6, the average image Ψ
Is obtained. That is, Ψ = (Φ ₁ + Φ ₂ + Φ ₃ + Φ ₄ + Φ ₅ + Φ ₆ ) / 6 is obtained. The difference between the one-dimensional vector Φ ₁ of the image 1a and the average image Ψ is defined by Γ ₁ ⁽¹⁾ . Similarly, for images 2a, 3a, 4a, 5a, 6a,
₂ ^{( 1)} , Γ ₃ ⁽¹⁾ , Γ ₄ ⁽¹⁾ , Γ ₅ ⁽¹⁾ , Γ ₆
⁽¹⁾ is defined.

That is, Γ ₁ ⁽¹⁾ = Φ ₁ −Ψ Γ ₂ ⁽¹⁾ = Φ ₂ −Ψ Γ ₃ ⁽¹⁾ = Φ ₃ −Ψ Γ ₄ ⁽¹⁾ = Φ ₄ −Ψ Γ ₅ ⁽¹⁾ = Φ ₅ −Ψ Γ ₆ ⁽¹⁾ = Φ ₅ −Ψ.

Γ₁ ⁽¹⁾As I will explain later,
Primary feature W of page 1a₁ ⁽¹⁾To get a base
It Similarly, Γ_Two ⁽¹⁾, Γ_Three ⁽¹⁾, Γ_Four ⁽¹⁾, Γ
₅ ^{( 1)}, Γ₆ ⁽¹⁾Are images 2a, 3a,
Primary features W of 4a, 5a, 6a _Two ⁽¹⁾, W_Three ⁽¹⁾, W
_Four ⁽¹⁾, W₅ ⁽¹⁾, W₆ ⁽¹⁾Base for getting
Becomes (1) at the upper right of the symbolic expression is the value related to the primary feature
Is shown. Also, Γ₁ ⁽¹⁾, Γ_Two ⁽¹⁾, Γ
_Three ⁽¹⁾, Γ_Four ⁽¹⁾, Γ₅ ⁽¹⁾, Γ₆ ⁽¹⁾That of
Reconstruction matrix obtained by processing each with a low-pass filter Express as.

The images 1b, 2b, 3b in the upper part of FIG.
4b, 5b, and 6b are the average image Ψ and the reconstruction matrix, respectively. Sum with It is a reconstructed face image represented by. That is, Becomes

Since the reconstructed face image includes data processed by the low-pass filter, even if the face angle slightly changes, the face information does not change significantly. Figure 2
Lower images 1c, 2c, 3c, 4c, 5c, 6c
Is the reconstructed facial image from the original facial image Φ in the middle of FIG. Is the residual image Γ ^{(2) as} a result of subtracting. That is, Becomes

In the residual image, the brightness information due to the illumination included in the original facial image and the brightness information due to the illumination included in the reconstructed facial image are canceled by subtraction, so that the residual image has the brightness due to the illumination. It is a person who is not affected by the influence. Hereinafter, the present invention will be described with reference to the general formula.

Image in M (6 in the case of interruption in FIG. 2) image group (Step 110 in FIG. 1). here, Is a one-dimensional vector of the raster scan image, Is defined as the average image (FIG. 1, step 12).
0).

[Equation 1]

All images are vectors from the average image Only (step 130, FIG. 1). Therefore, the covariance matrix of the data is defined as follows.

[Equation 2] here, (FIG. 1, step 140).

Q is It is a dimension. Where w is the image width and h is the height. The size of this matrix is huge, but since it only sums a finite number of image vectors M, the rank of this matrix does not exceed M-1. But If the eigenvector of (i = 1,2, .., M) (FIG. 1, step 150), (here, Is Is the eigenvalue of As you can see by multiplying Is Is the eigenvector of. That is,

[Equation 3] Becomes

However, The size of Nothing more. therefore, Eigenvector of If is defined, it becomes as follows.

[Equation 4]

Eigenvalue Is the eigenvector Is the variance along the new coordinate space (step 160 in FIG. 1). After that, the eigenvalue Suppose i is ordered so that decreases. The eigenvalues are decreasing exponentially. Therefore, (here, and Facial image by calculating To M ₁ dimension only Can be projected onto. Is in the new coordinate system Is the k-th coordinate of. In this sense Is called a primary feature. vector Is actually an image and is called the primary eigenface (typically called eigenface in other documents). Then (FIG. 1, step 170),

[Equation 5] (FIG. 1, step 180).

Primary features in FIG. The procedure for calculating is shown. In the figure, Is a function that calculates the i-th largest eigenvalue and its corresponding eigenvector. An interesting by-product of equation (3) is The reconstruction matrix can be obtained from Is Since it is a matrix, its inverse cannot be obtained. But,
The pseudo matrix can be used to approximate the inverse matrix. To If we take the pseudo-inverse of

[Equation 6] here, Is and Is a matrix reconstructed from In Figure 2, several original facial images and their corresponding reconstructed facial images give. The upper image is a reconstructed facial image And the middle image is the corresponding original facial image, the lower image is the residual image. Is. The reconstruction matrix shows that the facial details are lost. That is, The contents described by can be regarded as a low-pass filter image. The residual image is the corresponding high pass filtered image. Looking at the reconstruction matrix, some images could not be reconstructed at a reasonable resolution, indicating that primary features do not describe these images well. The poorer the reconstruction matrix, the more information is left in the residual image. These residual images still contain a wealth of information about the individual images, so
Facial features should be extracted again from these residual faces.

[0055] , To Eigenvalue of, To Given the corresponding eigenvectors of Is. Based on the above description, The eigenvector of is Is. Therefore, Residual facial image by calculating For M ₂ dimensions only Can be projected onto. here,

[Equation 7] and Is. Is the eigenvector of the residual facial image, The secondary eigenface, Is called a secondary feature.

[0056] Then, equation (5) can be written as follows.

[Equation 8]

[0057] Then,

[Equation 9] Is.

[0058] Is a constant conversion matrix and does not affect the calculation efficiency because it is calculated only once. The facial image is (here, ). Compared with the feature calculation only from the eigenface U The calculation load does not increase by calculating. For convenience of explanation, the residual image is called a secondary residual image and the original facial image is called a primary residual image (although it seems better to call the original facial image a zero-order residual image).

The primary characteristic describes the characteristic of the original image, and the secondary characteristic represents the characteristic of the high-pass image. These features can be used in different applications.

In the case of being unaffected by illumination, the viewing angles of the images are almost the same, but the illumination conditions are different. Illumination light can come from the left, right, or both directions. Also, the face may be bright or dark. In this situation, the image features have to be extracted after filtering. That is, the data of the original image includes the characteristic of illumination, and the data of the primary characteristic also includes the characteristic of illumination. Therefore, the primary characteristic is not suitable for this case. Therefore, the secondary feature obtained by subtracting the primary feature image from the original image image has the effect of illumination removed and can be used as data having a feature that is not affected by illumination. Only secondary eigenfeatures are suitable. On the other hand, in the case of being unaffected by the viewing angle, the images of the same person are shot under substantially the same lighting conditions, but the viewing angles or poses are not the same. In this situation, primary and secondary eigenfeatures are important to describe the face. The primary feature is an image obtained by processing the original image image with a low-pass filter, and shows the global feature of the facial image. Therefore, almost the same feature is extracted even in images with different viewing angles.
Therefore, when the viewing angle is different, the primary feature is used in addition to the secondary feature, and thus the descriptor can be used as a descriptor that is not affected by the viewing angle.

Therefore, the secondary eigenfeatures will be used as the facial descriptors that describe the situation unaffected by illumination, and the primary and secondary eigenfeatures will be used as the face descriptors that describe the situation unaffected by the viewing angle.

The symmetry of the face is an important feature when considering the face description. The faces of the majority of people are symmetrical.
Therefore, it is possible to use a facial image formed as a mirror image to represent the same person. As a result, the unique features of the original image and the mirror imaged image can both represent the same person. Therefore, the original image can be represented using a linear combination of the characteristic features of the two images. Features may be calculated using different weights to distinguish between the original image and the mirrored image.

[0063] The original image, To The adjusted intrinsic features are given below, given the mirror image formation of

[Equation 10] and

[Equation 11] Here, 0 ≦ c ≦ 1 is an adjustment weight. The calculation order may be rearranged to simplify the calculation.

[0064] Therefore, the following holds.

[Equation 12] here, Is It is a mirror-image formed peculiar face. In Figure 3 From Shows the procedure for obtaining. Using the same method the following is obtained:

[Equation 13] and

[Equation 14] Where mirror () is the left / right mirror image forming function for the mirror surface matrix. Using equations (8) and (9), the primary and secondary eigenfaces are When Can be adjusted to

A human face can be effectively described by utilizing the above-mentioned characteristic features. However, if we consider the characteristic features as those for facial recognition and related applications, the characteristic features should be normalized because their range is not the same.

One rational normalization method is to divide the eigenfeature by the corresponding standard deviation of the eigenfeature in the learning set. In the learning set The standard deviation of Here, j = 1,2 indicates a primary or secondary characteristic feature. Then the image The normalized eigenfeatures of are given below.

[Equation 15]

The facial image similarity is simply defined as the weighted distance between the normalized projections.

[Equation 16]

[0068] In the case of, the similarity of the facial image for the facial description that is not affected by illumination is measured using only the secondary features. For face-independent facial descriptions, both primary and secondary features are needed. The weights are chosen as follows.

[Equation 17] and

Other weights may be selected depending on the application. In fact, weights may be embedded in the adjusted eigenfaces to reduce the amount of computation, and the normalization parameters may also be moved to the eigenface matrix. The eigenface matrix and eigenfeatures should be quantized from floating point to integers to reduce storage requirements.

[0070] And here, Round () is a rounding function and j = 1,2. Intrinsic features can be quantized using:

[Equation 18] here, Is the image in the learning set. The feature set is and Can be written. Therefore, each unique feature can be quantized to 8 bits.

Furthermore, in order to reduce the overall size of the characteristic features, the characteristic features can be requantized according to the standard deviation of the training set. This is considered a trade-off between facial descriptors and search accuracy. According to the inventors' experiments, the bits for each unique feature can be assigned as follows.

[Formula 19] here, Is the minimum standard deviation of the jth characteristic feature. Different unique features allocate different bits, but the range remains [-128,127]. This quantisation strategy reduces the size of the facial descriptors by about 40%, whereas the search accuracy drops by only 1-2%. If the original quantization strategy is acceptable, no further quantization is needed.
Here is an example of normalization and clipping. As a result, further quantization is performed, and a more compact one is obtained than that obtained from the equation (10-1), and each eigenfeature can be uniformly represented by 5 bits. Where Z = 16384 * 8, which is an empirically determined normalization constant. From the above formula, two facial images in the case of a description that is not affected by the viewing angle The distance between is calculated as follows.

[Equation 20] In addition, two facial images in the case of a description that is not affected by lighting The distance between is calculated as follows.

[Equation 21]

Variable length coding can be applied to the quantized eigenfeatures to further compress the facial descriptors. Huffman and arithmetic coding are available for this task. The code table is calculated as follows. 1) Compute unique features for images in the learning set. 2) Quantize the characteristic features by the above measures. 3) Compute the probability of each quantization level for quantized eigenfeatures with the same bit allocation. 4) Compute the variable length code for all quantization levels. 5) Step 3) for all bit allocation cases
And 4) are used to calculate all chord tables.

With this method, the facial descriptor can be further compressed. Correspondingly, the coded face descriptors need to be decoded before the distance between the two face descriptors can be calculated.

According to our observation, the primary features themselves can also be used to describe facial features that are independent of viewing angle and illumination. The facial feature that is not affected by the viewing angle may be a characteristic feature extracted from the characteristic face corresponding to the top N characteristic values in size. The facial features that are not affected by illumination may be the characteristic features extracted from the characteristic face corresponding to the characteristic values in the range from the top k to N in size. Here 0 <k <N, but generally 4 ≦ k ≦ 10 and 40
≦ N ≦ 60. Preferably, with k = 8, N = 40,
The total eigenface value is 48. 4 and 5 are flowcharts showing a method of extracting a facial feature that is not affected by illumination and a method of extracting a facial feature that is not affected by a viewing angle.

The method for extracting the facial features that are not affected by illumination in FIG. 4 comprises the following steps. Step 410: Compute the adjusted quadratic eigenface matrix. Step 420: Obtain adjusted secondary eigenfeatures. Step 430: Select features that describe the face from the adjusted secondary eigenfeatures. Step 440: Encode the characteristic features quantized by VLC. This step may be omitted.

The method for extracting facial features that are not affected by the viewing angle in FIG. 5 comprises the following steps. Step 510: Compute the adjusted primary eigenface matrix. Step 520: Compute the adjusted quadratic eigenface matrix. Step 530: Obtain adjusted primary characteristic features. Step 540: Obtain adjusted secondary eigenfeatures. Step 550: Quantize the adjusted primary and secondary eigenfeatures. Step 560: Select facial description features from the adjusted primary and secondary eigenfeatures. Step 570: Encode the characteristic features quantized by VLC. This step may be omitted.

[0077]

INDUSTRIAL APPLICABILITY The present invention is extremely effective for describing the human face. Since the secondary eigenface is obtained from only one calculation using the learning face image, the secondary feature can be obtained as efficiently as the primary feature. Since the detailed information can be revealed by using the higher-order eigenfeatures, the performance of the combination of the primary and secondary features is superior to that of the same number of primary eigenfeatures in the facial description that is not affected by the viewing angle. Secondary eigenfeatures are far superior to primary eigenfeatures in their ability to describe facial features independent of lighting. The present invention is extremely effective and efficient in describing the human face. The present description can be used in Internet multimedia database searching, video editing, electronic libraries, surveillance / tracking, and other applications that make extensive use of face recognition / matching.

[Brief description of drawings]

FIG. 1 i-th order feature It is a flow chart which shows the procedure of calculating.

FIG. 2 is a diagram showing a reconstructed face, a corresponding original face and a secondary residual image.

FIG. 3 is a schematic diagram showing a procedure for obtaining a mirror image formation of a proper face.

FIG. 4 is a flowchart of generating a face descriptor that is not affected by lighting.

FIG. 5 is a flowchart of generating a facial descriptor that is not affected by a viewing angle.

[Explanation of symbols] 320 Reshaping 340 Mirror image formation 360 reshaping

   ─────────────────────────────────────────────────── ─── Continued front page    F term (reference) 5B075 ND08 NK06 NR12 PR06 QM08                 5L096 BA18 DA02 EA07 FA33 FA81                       GA06 JA03 JA05 JA11 KA04                       KA09 MA05

Claims (33)

[Claims] 1. Obtaining adjusted secondary eigenfeatures of a facial image, quantizing the adjusted secondary eigenfeatures, and describing a face from the quantized secondary eigenfeatures. Selecting the features that make up the facial descriptor of the method, and extracting the features for lighting-independent facial description. 2. Obtaining adjusted secondary eigenfeatures of a facial image, quantizing the adjusted secondary eigenfeatures, and describing a face from the quantized secondary eigenfeatures. And selecting the features that make up the facial descriptor of the above, and encoding the selected unique features into a facial descriptor that is not affected by lighting, and extracting features for facial description that is not affected by lighting. Method. 3. Obtaining adjusted primary eigenfeatures of the facial image; obtaining adjusted secondary eigenfeatures of the facial image; quantizing the adjusted primary eigenfeatures; Affecting the viewing angle, the method comprising: quantizing a quantized secondary eigenfeature; and selecting features that make up a facial descriptor for describing a face from the quantized primary and secondary eigenfeatures. Method for extracting features for facial description that are not processed. 4. Obtaining the adjusted primary eigenfeatures, obtaining the adjusted secondary eigenfeatures, quantizing the adjusted primary eigenfeatures, and the adjusted secondary eigenfeatures. A step of selecting a feature that constitutes a facial descriptor for describing a face from the quantized primary and secondary eigenfeatures, and a face that is not affected by a viewing angle of the selected eigenfeature. A method of extracting facial description-independent facial description features, the method comprising: encoding into a descriptor. 5. A method for determining an adjusted quadratic eigenfeature of a face image according to any one of claims 1 to 4, including a step of obtaining an inner product of the face image and the adjusted quadratic eigenface matrix. . 6. The method according to claim 3, further comprising a step of obtaining an inner product of the face image and the adjusted primary eigenface matrix.
The method for obtaining the adjusted primary eigenfeature of the facial image according to any one of 1. 7. The method of calculating the adjusted primary eigenface matrix of claim 6, comprising: calculating a primary eigenface matrix; and adjusting the primary eigenface matrix. 8. The method of calculating the adjusted quadratic eigenface matrix of claim 5, comprising: calculating a quadratic eigenface matrix; adjusting the quadratic eigenface matrix. . 9. A step of obtaining primary eigenfeatures of a training face image, a step of configuring the primary eigenfaces as a two-dimensional array of original images, a step of obtaining a mirror-imaged eigenface of the two-dimensional array, Weighting the mirrored eigenface image, adding the weighted eigenface image to the primary eigenface, and re-priming the primary eigenface to obtain a one-dimensional adjusted primary eigenface. The steps of constructing, normalizing the primary eigenfaces, obtaining the weights of the adjusted primary eigenfeatures for distance calculation, and adjusting the weights of the primary eigenfeatures for distance calculation Adjusting the primary eigenfaces of claim 7, comprising multiplying a matrix and quantizing a primary eigenface matrix. Method. 10. A step of calculating a primary eigenface matrix, a step of obtaining primary eigenfeatures from a training facial image, a step of calculating a pseudo inverse matrix of the primary eigenface matrix, and a step of calculating the primary eigenfeatures to primary eigenfeatures. Calculating a reconstructed primary facial image by multiplying the inverse of the facial matrix, obtaining a secondary residual image by subtracting the reconstructed primary facial image from the original image, and the secondary residual Obtaining a quadratic eigenface by calculating the eigenvectors of the image.
The method for calculating the secondary eigenface described in. 11. A step of obtaining secondary eigenfeatures of a learning face image by calculating an inner product of the facial image and the secondary eigenface, and reshaping the secondary eigenface into the shape of the original image to obtain bilateral symmetry. Obtaining a mirror-imaged eigenface image, weighting the mirror-imaged eigenface image,
Adding them to the shaped secondary eigenface image in the corresponding original image, and reconstructing the shaped secondary eigenface to obtain a one-dimensional adjusted secondary eigenface. , Normalizing the adjusted secondary eigenfaces, obtaining secondary eigenfeature weights for distance calculation, and adjusting secondary eigenfeature weighted secondary eigenfaces for distance calculation 9. A method for adjusting a quadratic eigenface according to claim 8, comprising: multiplying a matrix and quantizing a quadratic eigenface matrix. 12. Obtaining maximum and minimum values of the adjusted primary eigenfacial matrix, and obtaining a quantization step by dividing the maximum value and the minimum value into a plurality of quantization levels. The adjusted primary eigenfaces of claim 9, comprising: dividing the adjusted quadratic eigenface matrix by the quantization step; and rounding the divided value to the nearest integer. How to quantize a matrix. 13. Obtaining a maximum and minimum value of the adjusted quadratic eigenface matrix, and obtaining a quantization step by dividing between the maximum value and the minimum value into a plurality of quantization levels. A step of dividing the adjusted quadratic eigenface matrix by the quantization step, and rounding the divided value to the nearest integer,
A method for quantizing a conditioned quadratic eigenface matrix according to claim 11, comprising: 14. Obtaining a reconstructed adjusted quadratic eigenface by multiplying the quantized quadratic eigenface matrix by the quantization step of claim 11, wherein the reconstructed adjusted quadratic eigenface is obtained. 5. Obtaining a quadratic eigenfeature by multiplying each column of the eigenface by each row of the facial image, the method comprising: obtaining the adjusted quadratic eigenfeature according to any one of claims 1 to 4. 15. Obtaining a reconstructed adjusted primary eigenface by multiplying the quantized primary eigenface matrix by the quantized step of claim 12. 5. Obtaining the primary eigenfeatures by multiplying each column by each row of the facial image, the method comprising: obtaining the adjusted primary eigenfeatures according to claim 3 or 4. 16. Obtaining maximum and minimum values of the adjusted quadratic eigenfeatures of the training image, and quantizing by dividing the maximum value and the minimum value into a plurality of quantization levels. 5. A step of obtaining a step, a step of dividing the adjusted quadratic eigenfeature by the quantization step, and a step of rounding the divided value to the nearest integer. A method for quantizing a tuned quadratic eigenfeature according to item 1. 17. Obtaining maximum and minimum values of the adjusted quadratic eigenfeatures of the training image, and quantizing by dividing the maximum value and the minimum value into a plurality of quantization levels. Obtaining a step, dividing the adjusted quadratic eigenfeature by the quantization step, rounding the divided value to the nearest integer, and learning for dividing by the minimum standard deviation. Allocating a different number of bits to different eigenfeatures by calculating the rounded logarithm of the standard deviation of the corresponding eigenfeatures in the set, and quantizing the eigenfeatures according to the corresponding bit allocation. A method of quantizing a tuned quadratic eigenfeature according to any one of claims 1 to 4. 18. Obtaining maximum and minimum values of the adjusted primary eigenfeatures of a training image; quantizing steps by dividing the maximum value and the minimum value into a plurality of quantization levels. 5. The method according to claim 3, further comprising: obtaining the adjusted primary eigenfeatures, dividing the adjusted primary eigenfeatures by the quantization step, and rounding the divided value to the nearest integer. A method for quantizing the adjusted primary eigenfeatures according to claim 1. 19. Obtaining maximum and minimum values of the adjusted primary eigenfeatures of the training image; quantizing steps by dividing the maximum value and the minimum value into a plurality of quantization levels. A step of dividing the adjusted first-order eigenfeatures by the quantization step, a step of rounding the divided value to the nearest integer, and a learning set divided by the minimum standard deviation. Allocating a different number of bits to different eigenfeatures by calculating the rounded logarithm of the standard deviation of the corresponding eigenfeatures of, and quantizing the eigenfeatures according to the corresponding bit allocation. Method for quantizing the adjusted primary eigenfeatures according to any one of clauses 3 or 4. 20. An adjustment for distance calculation, comprising: obtaining a standard deviation of the adjusted primary eigenfeatures of a training facial image; obtaining a weight by extracting a square root of the standard deviation. The method of claim 9, wherein the weights of the selected primary eigenfeatures are obtained. 21. Obtaining a variance of the adjusted secondary eigenfeatures of a training face image; and obtaining a weight of the adjusted secondary eigenfeatures by extracting a square root of the variance. 12. The method of claim 11, including obtaining adjusted secondary eigenfeature weights for distance calculation. 22. Extracting the eigenfeatures of the face for lighting-independent face description by the method of claim 1 or 2, obtaining the Euclidean distance of the eigenfeatures of the face, and optimally matching Selecting a minimum Euclidean distance indicating a pair of faces to be faced, and measuring the similarity between the faces that are not affected by illumination. 23. A method of extracting a characteristic feature of a face for a face description which is not affected by a visual angle by the method according to claim 3; a step of obtaining a Euclidean distance of the feature of the face; and an optimally matching face. Selecting a minimum Euclidean distance that indicates a pair of F., and measuring the degree of similarity between faces that are not affected by the viewing angle. 24. Obtaining said quantized eigenfeatures of a training set by the method of claim 1 or 2, classifying said eigenfeatures into groups according to bit allocation, and entropy coding method. Constructing a code table for each of said groups of unique features having the same bit allocation using, and obtaining a code table for variably coding (VLC) a lighting-independent face descriptor. . 25. Obtaining the quantized eigenfeatures of a training set by the method of claim 3 or 4, classifying the eigenfeatures into groups according to bit allocation, and entropy coding techniques. Constructing a code table for each of said groups of unique features having the same bit allocation using a method for obtaining a code table for variable coding (VLC) a facial descriptor independent of viewing angle. . 26. The code table constructing method according to claim 24, wherein the entropy coding method is a Huffman coding method based on a probability of a quantization level. 27. The code table construction method according to claim 24, wherein the entropy coding method is an arithmetic coding method based on a probability of a quantization level. 28. Referencing the code table generated by the method of claim 24 for each quantized eigenfeature, and expressing the quantized eigenfeature using corresponding codewords. A method of encoding a lighting-independent facial descriptor according to any one of claims 1 or 2, comprising. 29. Referring to the code table generated by the method of claim 25 for each quantized eigenfeature, and expressing the quantized eigenfeature with corresponding codewords. A method of encoding a viewing angle insensitive facial descriptor according to any one of claims 3 or 4, comprising. 30. Obtaining the adjusted primary eigenfeatures of a facial image according to the method of claim 7, quantizing the adjusted primary eigenfeatures according to the method of claim 19, Selecting features to construct a facial descriptor that describes the face from the quantized primary eigenfeatures, and extracting features for general facial description. 31. Obtaining adjusted primary eigenfeatures of a facial image by the method of claim 7; quantizing the adjusted primary eigenfeatures by the method of claim 19; Selecting a feature to construct a facial descriptor that describes a face from the quantized primary eigenfeature; and encoding the selected intrinsic feature into a facial descriptor. A method for extracting features for facial description. 32. The quantization of claim 30 or 31, including the step of selecting eigenfeatures corresponding in magnitude to the top N eigenvalues to provide a facial description that is independent of viewing angle. A method for selecting features that form a facial descriptor that describes a face from the generated primary eigenfeatures. 33. The kth to Nth eigenvalues (0 <k) whose magnitudes are high in order to perform a face description that is not affected by illumination.
<N) corresponding characteristic step is selected,
32. A method according to any of claims 30 or 31 for selecting from the quantized primary eigenfeatures the features that make up a facial descriptor describing a face.