JP2001266150A - Method for portraying, identifying and matching face surface by using high-order face surface characteristic value - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for portraying and recognizing a human face surface effectively and efficiently. SOLUTION: In order to normalize the image of a face surface, the upper/ lower positions of eyes are promptly obtained. In order to portray detailed information which cannot be portrayed by a first-order face surface characteristic value a high-order face characteristic is used. In order to simplify calculation, an expression is given to calculate high-order transformation matrix for photographing. The face surface is portrayed individually by using high-order feature or combining it with a first-order feature.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to facial depiction and recognition for content-based image retrieval, human face identification and verification, monitoring and tracking for banks, security systems and videophones, digital libraries and Internet multimedia databases. Can be used for

[0002]

2. Description of the Related Art Human face recognition is an active field in the computer vision industry. Facial recognition plays an important role in multimedia database searching and many other applications. In recent years, considerable progress has been made on the problem of face detection and recognition. Different specific techniques have been proposed. Among these techniques are neural nets, stretch template matching, and Karhunen-L.
oeve) Expansions, algebraic moments and isopycnic lines are typical methods.

Among these methods, principal component analysis (PC
A) and Karhunen-Loeve expansion are important methods. The facial eigenvalue calculation method is derived from PCA, is convenient for calculation, and has consistent accuracy in identification. According to the conventional work, the PCA method spontaneously separates information of different types. Eigenvectors with large eigenvalues collect information common to a subset of faces, and eigenvectors with small eigenvalues collect information specific to individual faces. Studies have shown that only information surrounded by eigenvectors with large eigenvalues can be generalized to unadjusted new faces.

[0004]

The advantage of the facial eigenvalue method is that eigenvectors with large eigenvalues convey information about the basic shape and structure of the face. This means that the main features of the human face can be described using the features extracted from the eigenvectors having large eigenvalues. However, this is also a weak point of PCA. If we only consider features extracted from eigenvectors with large eigenvalues, we cannot get facial details corresponding to individual faces. Being able to describe these details of individual faces together with the common features of a person's face allows a more accurate description of the person's face.

[0005] Furthermore, if the image of the face is not normalized to a uniform size and the face is not placed at an appropriate position, it is difficult to obtain an appropriate eigenvector representing the human face. In order to normalize the facial image, we need to enlarge or reduce the facial image to a standard size and locate the face in a suitable position during preprocessing of the facial recognition.

[0006]

A method of calculating a facial eigenvalue is as follows.
This is effective for extracting common facial features such as shape and structure. In order to obtain details of the face lost when truncating eigenvectors with small eigenvalues, a face reconstructed with features from eigenvectors with large eigenvalues should be obtained. Using the reconstructed facial image,
A residual image between the original image and the reconstruction matrix can be obtained. These residual faces can be viewed as high frequency facial images that still contain a wealth of detailed information about the individual faces. The facial eigenvalue method can be used again on the residual face to depict the residual face. The resulting eigenvectors with large eigenvalues reveal common features of the residual face. Using this method, higher-order eigenvectors with large eigenvalues are
It can be obtained to extract the corresponding features. The face can be effectively depicted using a combination of the features from different order facial eigenvalues.

[0007] To normalize the facial image, the facial image is resized to a standard size. Since the vertical position of the eyes is a parameter suitable for determining the position of the face, the position of the face is determined using the vertical position and moved to a predetermined position. The vertical position of the eyes is obtained using a method based on the discrete Fourier transform.

[0008]

The present invention provides a method for rendering a human face that can be used for image search (querying by face example), personal identification and confirmation, monitoring and tracking, and other facial recognition applications. To depict facial features, the concept of higher-order facial eigenvalues has been proposed by our opinion and guidance. Initially, all facial images are normalized to a standard size.
Next, the vertical position of the eyes is calculated, and the face is moved to an appropriate position. Upon completion of all these pre-processing procedures, facial eigenvalues and higher-order facial eigenvalues can be obtained from the set of adjusted facial images. In order to query the facial image to the facial database, the features of the image projected using the facial eigenvalue and the higher-order facial eigenvalue may be calculated using the selected facial eigenvalue and the higher-order facial eigenvalue. A face can be depicted using a combination of these features. For this depiction, the Euclidean distance can be used to measure similarity. The features should be weighted to improve the similarity accuracy.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention provides a method for extracting facial features.

First, it is necessary to determine the vertical position of the eyes for normalization of the face. Eye location can be obtained using the following steps. Projecting the face image along the vertical direction Obtaining the high frequency coefficient of the projection using the discrete Fourier transform Discarding the high frequency coefficient (5 to 10 low frequency coefficients can be held depending on the image size) Performing the inverse Fourier transform on the residual frequency coefficients (filling the corresponding high frequency coefficients with zeros); the first two whose two peak magnitudes are greater than a threshold.
Locating the minimum of the curve between two local peaks.
This minimum captures the eye position with a high probability. If there is no local minimum in the prediction area, the eye position is set to the most likely position.

FIG. 1 shows a flowchart for determining the vertical position of the eye. HorizontalProjection (X) is a function for accumulating the pixel values of the image X along the horizontal direction. F
FT (y) gives the Fourier transform of y. LocalPeak (y ', k)
Is a function that searches for the k-th local maximum of y ′, and returns its value and position. LocalValley (y ', Loc1, Loc2) is the minimum and L
and the position between oc1 and Loc2. D is a predetermined default eye position. If the program cannot find the eye position in the appropriate position, it uses D as the top and bottom eye position. After locating the top and bottom of the eye, the image is moved by placing the eye in the appropriate position. Using this method, an effective facial eigenvalue can be calculated. After this normalization, the facial eigenvalues are more reliable than before.

Using the normalized face image, the face unique value and the higher-order face unique value can be obtained as follows.

Consider an image Φ _i (where Φ _i is a one-dimensional vector of a raster scan image) of the collection of M images. Ψ is defined as the average image.

Every image differs from the averaged image by a vector Γ _i ⁽¹⁾ = Φ _i -Ψ. Therefore, the covariance matrix of the data is defined as follows. here, It is.

Note that Q has dimensions wh × wh (where w is the width of the image and h is the height). Although the size of this matrix is huge, the columns of this matrix cannot exceed M-1 because it only adds up a finite number of image vectors M. ν _i ⁽¹⁾ (i = 1,2, ..., M) (Where λ _i ⁽¹⁾ is Eigenvalue), and as can be seen by multiplying the left side of the previous equation by A ⁽¹⁾ Is Note that this is the eigenvector of.

However, Is only a size M × M. therefore, Defining the eigenvector u _i ⁽¹⁾ of

The eigenvalue λ _i ⁽¹⁾ is the variance along the new coordinate space spanned by the eigenvector u _i ⁽¹⁾ . From here on, it is assumed that the order of i is such that the eigenvalue λ _i ⁽¹⁾ decreases. The eigenvalues decrease exponentially. Therefore, (here, And 1 = k = M ₁ ) to calculate the facial image Γ
⁽¹⁾ can be projected onto only M ₁ << M dimensions. w _k
⁽¹⁾ is the k-th coordinate of Γ ^{(1) in} the new coordinate system. In this context, W ⁽¹⁾ is called the primary feature. vector
u _k ⁽¹⁾ is actually the image, the primary facial eigenvalue (eig
enfaces) (commonly called eigenfaces in other documents). Then Becomes

FIG. 3 shows a procedure for calculating the primary feature W ⁽¹⁾ . In the figure, eig (B, i) is a function for calculating the i-th largest eigenvalue and its corresponding matrix eigenvector. From equation (3), an interesting by-product is that a reconstruction matrix can be obtained from W ⁽¹⁾ . Since U ⁽¹⁾ is an M ₁ × P matrix, the inverse cannot be obtained. However, the pseudo inverse can be used to approach the opposite. But Is the inverse of here, Is the reconstructed matrix from W ⁽¹⁾ and U ⁽¹⁾ . FIG. 2 gives several original facial images and their corresponding reconstructed facial images (M ₁ = 25). The upper image is a reconstructed facial image Where the middle image is the corresponding original face image and the lower image is the residual image It is. From the reconstruction matrix, it can be seen that facial details have been lost. This means that the content described by W ⁽¹⁾ can be noticed as a low-pass filter image.
The residual image is the corresponding high pass filter image. When observing the reconstruction matrix, some images cannot be reconstructed with suitable solutions indicating that the primary features do not render the image well. The worse the reconstruction matrix, the more information is stored in the residual image. Since these residual images still contain a wealth of information about the individual images, facial features should be extracted again from these residual faces.

[0019]Then λ_i ⁽²⁾IsEigenvalue of ν_i ⁽²⁾IsIs the corresponding eigenvector. Also, It is. Based on the above description,The eigenvector of isIt is. Therefore,By calculating the residual facial image Γ⁽²⁾M_Two<< M
It can be projected onto dimensions only. here, And 1 = k = M_TwoIt is. u_k ⁽²⁾Is the residual facial image
U _k ⁽²⁾Is the secondary facial eigenvalue, w_k ⁽²⁾
Are called secondary features.

[0020] Then, equation (5) can be written as follows.

[0021] Then It is.

U ₂ is a constant transformation matrix and is calculated only once, so that it does not affect the calculation efficiency. The facial image is (Where 1 = M ₁ ′ = M ₁ ). Computing Ω (Φ) does not increase the computational burden compared to computing features only from the facial eigenvalue U. Facial image similarity is simply defined as the weighted distance between projections.

If a ₁ = 0, the similarity of the facial images is measured using only the secondary features. For convenience of explanation, the residual image is called a secondary residual image, and the original facial image is called a primary residual image (although it seems better to call the original facial image a zero-order residual image).

Using the same method, tertiary, quaternary,..., N
The next facial eigenvalue can also be derived. By projecting the residual image of the corresponding order, the third, fourth,…,
An nth order feature can be obtained. Using these higher-order features, the similarity of facial images can be defined as a weighted Euclidean distance between projections. FIG. 4 shows a procedure for calculating the _i- ^th facial eigenvalue U ⁽ⁱ⁾ and the corresponding transformation matrix Ui. In the figure, Pseudo_Inv (B) is a function for calculating a pseudo inverse of the matrix B.

In order to efficiently represent the features, the features may be compressed and represented. The simplest way is to truncate the features to integers. In our opinion, one possible way is as follows. Here, round () is a function for rounding a real number to an integer. Therefore, Is the coded representation of Ω (Φ).

A method of drawing a facial image using the above method can be considered as follows. 1) Normalizing the facial image to a standard size 2) Quickly locating the top and bottom of the eyes 3) Moving the head to an appropriate position 4) Scanning the facial image into a one-dimensional array of pixels 5) Subtracting the one-dimensional array of the pixels using an average image 6) Multiplying the one-dimensional array of the subtracted pixels using the primary and higher order facial eigenvalues 7) Depicting the resulting features in the face 8) encoding the features in a coded display

Using this method, a human face can be effectively and efficiently described in space, and can be calculated with reduced temporal complexity.

[0028]

The present invention is very effective for depicting a person's face. Since the higher-order face eigenvalue can be calculated only once using the adjusted face image, the higher-order features can be obtained as efficiently as the primary features. However, because the detailed information can be revealed using higher-order features, the combination of primary features and higher-order features allows for better facial depiction than the same number of primary features.
According to our experiments, when the number of the primary feature M ₁ is less than 20, the secondary feature has a superior image quality than the primary features. That even means simply using the secondary features to depict the face, which improves the recognition rate over using the same number of primary features.

The present invention is very effective and efficient in depicting human faces and makes extensive use of Internet multimedia database searching, video editing, digital libraries, surveillance and tracking, and facial recognition and verification. It can be used in other applications.

[Brief description of the drawings]

FIG. 1 is a flowchart for determining the vertical position of an eye.

FIG. 2 is an explanatory diagram of a reconstructed face, a corresponding original face, and a secondary residual image.

FIG. 3 is a flowchart showing a procedure for calculating a primary feature W ^(1).

FIG. 4 i-th face eigenvalue U ⁽ⁱ⁾ and corresponding transformation matrix U _i
Flowchart showing the procedure for calculating

Continuing on the front page (72) Inventor Tio Ken Tan Singapore 534415 Singapore, Thai Sen Avenue, Blocks 1022, 04-3530, Thai Sen Industrial Estate, Panasonic Singapore Institute F-term ( Reference) 5C054 AA02 CA04 CC03 EA05 FC12 HA05 HA18 HA23 HA24 5L096 BA02 BA18 CA02 EA13 EA14 FA23 FA69 FA81 JA11

Claims (19)

[Claims] Calculating a primary facial eigenvalue using the adjusted facial image; calculating a secondary facial eigenvalue using the adjusted facial image; and a facial image to be drawn from the secondary facial eigenvalue. A method for extracting features for facial depiction, comprising: obtaining a secondary feature of: and selecting a feature that describes the facial image from the secondary feature. Calculating a primary facial eigenvalue using the adjusted facial image; obtaining a primary feature for the facial image to be drawn from the primary facial eigenvalue; and a secondary facial using the adjusted facial image. Calculating eigenvalues; obtaining secondary features for the facial image to be rendered from the secondary facial eigenvalues; selecting features from the primary and secondary features to describe the facial image; A method for extracting features for facial depiction, including: Calculating a primary facial eigenvalue using the adjusted facial image; calculating a higher facial eigenvalue using the adjusted facial image; and a facial image to be drawn from the higher facial eigenvalue. , I-th,..., N-th features of the following, and selecting a feature that describes the facial image from the higher-order features: Method. Calculating a primary facial eigenvalue using the adjusted facial image; obtaining a primary feature for the facial image to be rendered using the primary facial eigenvalue; Calculating a secondary facial eigenvalue; and obtaining secondary, tertiary,..., I-order,..., N-order features (higher-order features) for the facial image to be rendered using the higher-order facial eigenvalue. Selecting a feature for describing the facial image from the primary feature and the higher-order feature. 5. The method according to claim 1, further comprising: calculating a pseudo-inverse of the primary facial eigenvalue; obtaining a secondary residual image; and calculating an eigenvector of the secondary residual image. How to calculate the secondary facial eigenvalues. 6. calculating a pseudo-inverse of said primary facial eigenvalue; obtaining a secondary residual image; secondary, tertiary,..., I-order,. Is referred to as a higher-order facial eigenvalue).
How to calculate the higher-order facial eigenvalues described. 7. The method includes: (i-1) calculating a pseudo-inverse of an eigenvector matrix, obtaining an i-th residual image, and calculating an i-th eigenvector matrix of the i-th residual image. 7. The method for calculating an i-th eigenvector according to claim 6. 8. The method according to claim 5, further comprising: obtaining a reconstruction matrix of an original image; and subtracting the reconstruction matrix of the original image from the original image to obtain a quadratic residual image. How to get the next residual image. 9. An image processing apparatus comprising: (i-1) obtaining a next-order residual image; (i-1) obtaining a reconstruction matrix of the (i-1) -th order residual image; Subtracting the reconstruction matrix of the (i-1) -th order residual image from the residual image (original face image).
A method for obtaining the described i-th residual image. 10. The method for obtaining a reconstruction matrix of an original image according to claim 8, comprising multiplying a first-order feature by using a pseudo-inverse of a first-order eigenvector matrix. 11. The method of claim 9, further comprising the step of: (i-1) multiplying the characteristic of the (i-1) -th order feature using a pseudo inverse matrix of the (i-1) -th order eigenvector matrix. How to get the constituent matrix. 12. The method of claim 1, further comprising the step of multiplying the quadratic residual image with a quadratic eigenvector matrix.
A method for obtaining the described secondary features. 13. The method of claim 3, further comprising the step of multiplying the i-th residual image with the i-th eigenvector matrix.
A method for obtaining the described i-th feature. 14. A method of normalizing a head position, comprising: locating an upper and lower position of an eye; and moving a facial image by placing the eye in a uniform position. 15. Projecting a facial image along a vertical direction, obtaining a frequency coefficient of the projection using a discrete Fourier transform, discarding the high frequency coefficient, and an inverse Fourier transform with respect to a residual frequency coefficient. And determining the minimum of the curve between the first two local peaks where the magnitude of the two peaks is greater than a threshold. If the minimum is found in the expected area, the vertical position of the eye is reduced to the minimum The method of claim 14, further comprising: setting a position of the eye; and setting the upper and lower positions of the eye to the most probable position when there is no local minimum in the expected area. . 16. normalizing the facial image to a standard size; scanning the facial image into a one-dimensional array of pixels; subtracting the one-dimensional array of pixels using an average image; And multiplying the one-dimensional array of the drawn pixels using a higher-order facial eigenvalue and using the obtained features as a facial depiction;
How to render facial images, including. 17. normalizing the face image to a standard size; locating the top and bottom of the eyes; moving the head to an appropriate position; and scanning the face image into a one-dimensional array of pixels. Subtracting a one-dimensional array of the pixels using an average image; multiplying the one-dimensional array of the extracted pixels using the primary and higher-order facial eigenvalues; Steps to use as
How to render facial images, including. 18. normalizing the facial image to a standard size; scanning the facial image into a one-dimensional array of pixels; subtracting the one-dimensional array of pixels using an average image; And multiplying the one-dimensional array of the drawn pixels using a higher-order face eigenvalue, using the obtained feature as a facial depiction, and coding the feature into a coded display. How to depict facial images, including. 19. normalizing the facial image to a standard size; locating the upper and lower positions of the eyes; moving the head to an appropriate position; and scanning the facial image into a one-dimensional array of pixels. Subtracting a one-dimensional array of the pixels using an average image; multiplying the one-dimensional array of the extracted pixels using the primary and higher-order facial eigenvalues; And encoding the features into a coded representation.