JP2006338686A - Face similarity calculation method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a face metadata generation technique and a face similarity calculation technique capable of improving accuracy of face recognition, and to provide a technique for constructing a practical face matching system. <P>SOLUTION: Face feature quantities are extracted by a face feature extraction part 121, and a reliability index is extracted by a reliability index extraction part 122 and output as face metadata. Information such as a parameter or the like related to a posterior distribution when the reliability index is obtained is estimated by a distribution estimation part 141 by using the reliability index of the face metadata in matching, and the similarity among the feature quantities is calculated by a distance calculation part 142. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique that can be used for face identification, face identification, facial expression recognition, gender identification by face, age discrimination by face, and the like, and in particular, the degree of similarity of a face image displayed on a still image or a moving image. The present invention relates to a calculation method and apparatus.

  Metadata generally refers to data that describes or represents the meaning of data. In the case of face recognition, it mainly refers to data related to face data such as still face images and moving images.

  MPEG-7 (International standard for multimedia content description interface, ISO / IEC 15938 standardized by the Moving Pictures Experts Group (MPEG)) as a standardization activity for multimedia content such as video, images and audio Is widely known. Among them, a face recognition descriptor has been proposed as a metadata descriptor related to face recognition (Non-Patent Document 1).

  In this face recognition descriptor, a base matrix for extracting feature values of a face image is obtained from a cut and normalized face image using a kind of subspace method generally called an eigenface. A face feature amount is extracted from the image using a base matrix, and this is used as metadata. It has also been proposed to use a weighted absolute value distance as the similarity to the face feature amount.

  In addition, it is known that there are various methods in the technology related to face recognition, for example, a method using a unique face based on principal component analysis (Non-Patent Document 2) or discriminant analysis (Non-Patent Document 3). ing.

  In addition, when applying the subspace method to a feature amount obtained from a fingerprint image, there is a method of introducing a quality index and adaptively measuring a distance between patterns (Non-patent Document 4 and Patent Document 1).

A. Yamada et al., "MPEG-7 Visual part of eXperimental Model Version 9.0," ISO / IEC JTC1 / SC29 / WG11 N3914, 2001 Moghaddam et al., "Probalilistic Visual Learning for Object Detection", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 17, No. 7, pp. 696-710, 1997 W. Zhao et al., "Discriminant Analysis of Principal Components for Face Recognition," Proceedings of the IEEE Third International Conference on Automatic Face and Gesture Recognition, pp. 336-341, 1998 T. Kamei and M. Mizoguchi, “Fingerprint Preselection Using Eigenfeatures,” Proceedings of the 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp.918-923, 1998, C.M.Bishop, "Neural Networks for Pattern Recognition", Oxford University Express, 1995 Japanese Patent Laid-Open No. 10-177650

  However, the conventional technology described above cannot obtain sufficient face recognition accuracy. Accordingly, an object of the present invention is to provide a face metadata generation technique and a face similarity calculation technique capable of improving the accuracy of face recognition, and further provide a technique for constructing a practical face matching system. There is to do.

  According to the present invention, the accuracy of face recognition can be improved by extracting the reliability from the face image and adaptively calculating the similarity between the patterns according to the reliability.

  According to the present invention, the face similarity that calculates the similarity of the face image based on the face feature amount extracted from the face image and the reliability index that represents the reliability of the face recognition result using the face feature amount. In the calculation device, a statistic of a distribution related to a comparison amount between facial feature amounts is stored in advance for each reliability index value, and the statistic corresponding to the reliability index value is calculated as a posterior distribution of the comparison amount. A distance for outputting the distance between facial feature amounts calculated based on the posterior distribution of the comparison amount using the distribution estimation means that outputs the parameter as parameter (parameter information) and the comparison amount and the parameter as the similarity And calculating means.

  According to the first aspect of the present invention, the distribution estimation means stores in advance a statistical quantity of a difference vector distribution between facial feature quantities for each reliability index value, and uses the reliability index value as a value. The corresponding statistic is output as a parameter of the posterior distribution of the difference vector, and the distance calculation means is derived from the log likelihood of the posterior distribution of the difference vector using the difference vector and the parameter. The distance between the facial feature amounts calculated using a distance function is output as the similarity.

  According to the second aspect of the present invention, the distribution estimation means includes a first statistic of a difference vector distribution (intraclass distribution) between facial feature quantities in a class to be regarded as a match, and a mismatch. The second statistic of the difference vector distribution (class-class distribution) between the facial feature quantities between classes that should be considered as a class is stored in advance for each reliability index value, and corresponds to the reliability index value. The first statistic and the second statistic are respectively output as a parameter of the intraclass distribution and a parameter of the interclass distribution, and the distance calculation means uses the difference vector and the parameter. The distance between the face feature amounts calculated using a distance function derived from the log likelihood of the ratio between the intraclass distribution and the interclass distribution is output as the similarity.

  According to the present invention, a statistic corresponding to the value of the reliability index is output as a parameter of the posterior distribution of the comparison amount, and the facial feature amount calculated based on the posterior distribution of the comparison amount using the comparison amount and the parameter By outputting the inter-distance as the similarity, highly accurate face image matching can be achieved.

(Principle of the invention)
First, the principle of the present invention will be described. In general, when pattern recognition is performed, if it is possible to prepare a large amount of learning data for a class to be recognized, a pattern distribution function is estimated based on statistical analysis of the learning data. Can be built. However, in the face recognition application, only one registered image can be obtained for each individual, and only a few registered images are often allowed.

  Even in such a case, an index called a reliability index is given to the facial feature vector, and by considering a class based on that index, statistical analysis of the class for that reliability index is performed, and the pattern distribution function is calculated. By estimating, it is possible to construct a pattern recognition mechanism based on a distribution function through a reliability index even for face recognition applications in which only one registered image can be obtained.

  Hereinafter, the principle of face recognition regarding the Mahalanobis distance based on the error distribution and the discrimination distance based on the intraclass distribution and the interclass distribution will be described.

Consider the case of observing one person's face. It is assumed that the feature vector obtained from the face image when there is no error is v ₀ , and the actually observed vector v is superimposed with the error vector ε (Equation 1).

Here, if the feature vector v ₀ is observed twice, two observation vectors v ₁ and v ₂ are obtained (Equation 2).

Now, a reliability index θ ₁ and a reliability index θ ₂ correlated with the error vector ε ₁ and the error vector ε ₂ are obtained, respectively, and their posterior distributions are p (ε | θ ₁ ), p (ε | θ ₂ ).

If the posterior distribution of the difference vector s between v ₁ and v ₂ when such a distribution is obtained is expressed as p (s | θ ₁ , θ ₂ ), the similarity d (v ₁ between patterns) , v ₂ ), the following log likelihood can be used.

If the posterior distributions p (ε | θ ₁ ) and p (ε | θ ₂ ) are normal distributions, the distribution p (s | θ ₁ , θ ₂ ) of the difference vector s is also a normal distribution. Here, considering the normal distribution of the covariance matrix Σε (θ _i ) with an average of 0 as the posterior distribution p (ε | θ _i ) (i = 1, 2) of the error vector, the distribution of the difference vector s has an average of 0 Thus, the covariance matrix Σ _s (θ ₁ , θ ₂ ) is expressed by the following equation (Equation 4).

つまり、事後分布ｐ(ｓ|θ₁, θ₂)は次式（数５）で表される。

That is, the posterior distribution p (s | θ ₁ , θ ₂ ) is expressed by the following equation (Equation 5).

Therefore, (Equation 3) is adapted to the reliability indices θ ₁ and θ ₂ using the covariance matrix Σε (θ ₁ ) and Σε (θ ₂ ) as shown in the following equation (Equation 6). It can be expressed as a typical Mahalanobis distance.

誤差ベクトルの各要素間における独立性を仮定すれば、（数５）は次式（数７）となる。

Assuming the independence between the elements of the error vector, (Equation 5) becomes the following equation (Equation 7).

Here, σ _{s, k} (θ ₁ , θ ₂ ) ² is the k-th diagonal element of the covariance matrix Σ _s (θ ₁ , θ ₂ ), that is, the variance of the observation error. Also, σε _{, k} (θ ₁ ) ² and σε _{, k} (θ ₂ ) ² are the k-th diagonal elements of the covariance matrices Σε (θ ₁ ) and Σε (θ ₂ ), respectively. s _k is the k th element of the difference vector s.

Assuming a normal distribution in this way, (Equation 3) gives the variances σε _{, k} (θ ₁ ) and σε _{, k} (θ ₂ ) for each element of each feature vector as shown in the following equation (Equation 8). The similarity can be defined by the Mahalanobis distance adaptive to the reliability indices θ ₁ and θ ₂ .

ここで、ｖ_1,k、ｖ_2,kはそれぞれ特徴ベクトルｖ₁、ｖ₂のｋ番目の要素である。

_{Here, v 1, k, v 2} , k is the k th element of the feature vector v _1, v _2, respectively.

In the above description, a normal distribution is assumed as the posterior distribution p (s | θ ₁ , θ ₂ ). Hereinafter, a mixed normal distribution is assumed. As the posterior distribution p (s | θ ₁ , θ ₂ ), as shown by the following equation (Equation 9), the normal distribution p (s | θ ₁ , θ ₂ , j) (j = 1, 2,... ., M) suppose that the posterior distribution p (s | θ ₁ , θ ₂ ) can be expressed.

従って、次式（数１０）により適応的混合マハラノビス距離を定義することができる。

Therefore, the adaptive mixed Mahalanobis distance can be defined by the following equation (Equation 10).

For estimating the covariance matrices Σ _s (θ ₁ , θ ₂ , j) and P (j) of the posterior distribution p (s | θ ₁ , θ ₂ , j), a maximum likelihood estimation method which is a general estimation method Or EM algorithm (see Non-Patent Document 5).

  By assuming a mixed normal distribution, the distribution can be approximated more accurately and the matching performance is improved. However, a large amount of learning data is required and the amount of calculation is also greatly increased.

  The Mahalanobis distance based on the above error distribution is a distance that is excellent for problems such as face identification to find out which face to search is closest to among other registered face data. is there.

  On the other hand, in the face verification problem, whether to accept or reject is important in determining the identity of the input face with the registered image. The distance named “discrimination distance” described below is a better similarity measure than the above-described Mahalanobis distance for the face identification problem.

  Now, it should be determined that the two feature vectors v of the face match, that is, if the two feature vectors belong to the same class (for example, the two feature vectors are the face data of the same person), Assume that the combination of these feature vectors belongs to class W. Also, if the two feature vectors v should be determined to be inconsistent, that is, feature vectors between classes (for example, the two feature vectors are different person face data), the combination thereof is class B. Suppose it belongs.

Assume that reliability indices θ ₁ and θ ₂ are obtained for the two feature vectors v ₁ and v ₂ . Class W that imitates that the difference vector s and the two reliability indices θ ₁ and θ ₂ (hereinafter, the set of two reliability indices is represented as [θ _i ]) match each other. And the discriminant problem between the class B to be imitated and the discrimination rule of the following equation (Equation 11) is obtained.

  The left side of the above (Equation 11) can be rewritten as the following equation (Equation 12) by the Bayes Theorem.

Here, assuming that the probabilities of occurrence of W, B and [θ _i ] are independent, P (W, [θ _i ]) = P (W) P ([θ _i ]), P (B, [ θi]) = P (B) P ([θ _i ]).

By calculating the log likelihood of (Equation 12) as the distance d (v ₁ , v ₂ ) between patterns, a similarity suitable for the face identification problem can be obtained as in the following equation (Equation 13). .

  If the prior probabilities P (W) and P (B) are different for each individual matching and can be known, it is desirable to calculate the second term of (Equation 13). However, in many cases, since the prior probability cannot be known for each individual matching, assuming that the prior probability is constant, the second term is assumed to be constant and excluded from the calculation of the similarity.

The posterior probabilities P (s | W, [θ _i ]) and P (s | B, [θ _i ]) are respectively expressed as intra-class distribution p _W (s | [θ _i ]) and inter-class distribution p _B (s | When it is rewritten as [θ _i ]), the following equation (Expression 14) is obtained.

Next, it is assumed that the intraclass distribution p _W (s | [θ _i ]) and the interclass distribution p _B (s | [θ _i ]) are normal distributions, and the mean is 0 and the covariance matrix is Assuming that Σ _W ([θ _i ]) and Σ _B ([θ _i ]) respectively, the posterior distribution can be expressed by the following equation (Equation 15).

  Substituting the above expression into (Expression 14) (however, the second term of (Expression 14) is omitted), the distance expressed by the following expression (Expression 16) can be obtained. This is called “adaptive discrimination distance”.

差ベクトルｓの各要素間における独立性を仮定すれば、（数１５）は、次式となる。

Assuming the independence between the elements of the difference vector s, (Equation 15) becomes the following equation.

Here, σ _{W, k} (θ _i ) ² and σ _{B, k} (θ _i ) ² are the k-th diagonal elements of the covariance matrices Σ _W (θ _i ) and Σ _B (θ _i ), respectively. That is, it corresponds to intra-class variance and inter-class variance. s _k is the k th element of the difference vector s.

Assuming a normal distribution in this way, (Equation 16) can be obtained from the following equation (Equation 18): intra-class variance σ _{W, k} (θ _i ) ^{2 for} each element of each feature vector, inter-class variance σ _{Using B, k} (θ _i ) ² , it is possible to define a similarity based on an adaptive discrimination distance with respect to the reliability index [θ _i ].

In the above description, the normal distribution is assumed as the intra-class variance σ _{W, k} (θ _i ) ² and the inter-class variance σ _{B, k} (θ _i ) ² , but in the following, a mixed distribution is assumed.

As shown in the following equation (Equation 19), the intra-class distribution p _W (s | [θ _i ]) and the inter-class distribution p _B (s | [θ _i ]) are respectively expressed as normal distributions p _W (s | [θ _i]. ], j _W ) (j _W = 1,2, ..., M _W ), p _B (s | [θ _i ], j _B ) (j _B = 1,2, ..., M _B ) Assume that the posterior distribution can be expressed by the sum.

  Therefore, the adaptive mixed Mahalanobis distance of the following equation (Equation 20) can be derived using this log likelihood.

Covariance matrix Σ _W (s | [θ _i ], j _W ) of intraclass distribution p _W (s | [θ _i ], j _W ) and interclass distribution p _B (s | [θ _i ], j _B ) , Σ _B (s | [θ _i ], j _B ), P (j _W ), and P (j _B ) can be estimated using a maximum likelihood estimation method or an EM algorithm.

  By assuming a mixed distribution, the distribution can be approximated more accurately and the matching performance can be improved, but at the same time, a large amount of learning data is required and the amount of computation is greatly increased.

  Thus, by extracting a reliability index further from the face feature amount, an adaptive distance criterion can be derived for the reliability index, and a highly accurate face recognition mechanism can be constructed. In the above, the reliability index for the feature vector is not specified as a scalar quantity (only one component) or a vector quantity (having a plurality of components), but it is an argument that holds in either case. Use of elements can be expected to improve performance.

  As for the specific reliability index, it is necessary to find an experimentally effective reliability index. In the case of face recognition, the contrast index indicating the contrast of the image, and the front face recognition, Using an asymmetric index that represents the amount of distortion in the face image caused by posture changes from left and right symmetry distortion is highly effective, and combining these reliability indices into a vector quantity is expected to improve accuracy. it can.

(Embodiment)
FIG. 1 is a block diagram illustrating a face image matching system according to an embodiment of the present invention. Hereinafter, the face image matching system will be described in detail.

  As shown in FIG. 1, in the face image matching system according to the present invention, a face image input unit 11 for inputting a face image, and generating face metadata of the face feature amount and reliability index from the input face image. A face metadata generation unit 12, a face metadata storage unit 13 that stores the extracted face metadata, a face similarity calculation unit 14 that calculates a face similarity from the face metadata, and a face image. A face image database 15, a control unit 16 that controls input of images, generation of metadata, accumulation of metadata, and calculation of face similarity in response to an image registration request / search request; And a display unit 17 for displaying information.

  The face metadata generation unit 12 includes a face feature extraction unit 121 that extracts a face feature from an input face image and a reliability index extraction unit 122 that extracts a reliability index, and includes a face similarity calculation unit. 14 includes a distribution estimation unit 141 that estimates parameter information related to the posterior distribution from the reliability index, and a distance calculation unit 142 that calculates the distance between the facial feature amount and the posterior distribution information from the distribution estimation unit 141. The

  At the time of registration, the image input unit 11 inputs a face photograph or the like after matching the size and position of the face with a scanner or a video camera. Alternatively, a person's face may be input directly from a video camera or the like. In this case, it is better to detect the face position of the input image and automatically normalize the size of the face image using the face detection technique as shown in the above-mentioned Moghaddam document. Would be good.

  Further, the input face image is registered in the face image database 15 as necessary. Simultaneously with the registration of the face image, face metadata is generated by the face metadata generation unit 12 and stored in the face metadata storage unit 13.

  Similarly, at the time of retrieval, a face image is input by the face image input unit 11, and face metadata is generated by the face metadata generation unit 12. The generated face metadata is once registered in the face metadata accumulation unit 13 or directly sent to the face similarity calculation unit 14. In the search, when it is confirmed whether or not a face image inputted in advance is in the database (face identification), the similarity with each piece of data registered in the face metadata storage unit 13 is calculated. Based on the result with the highest similarity (small distance value), the control unit 16 selects a face image from the face image database 15, displays the face image on the display unit 17 and the like, and displays the face in the search image and the registered image. The operator confirms the identity of.

  On the other hand, when confirming whether the face image specified in advance by the ID number or the like matches the searched face image (face identification), it is determined whether or not the face image with the specified ID number matches. When the similarity is lower than the predetermined similarity (the distance value is large), it is determined not to match, and when the similarity is high, it is determined to match, and the result is calculated. It is displayed on the display unit 17. If this system is used for entry management, entry control can be performed by controlling the automatic door by sending an opening / closing control signal from the control unit 16 to the automatic door instead of displaying.

  As described above, the face image matching system operates, but such an operation can also be realized on a computer system. For example, a metadata generation program that executes metadata generation and a similarity calculation program that executes similarity calculation, which will be described in detail below, are stored in memory, respectively, and are executed by a program control processor. Face image matching can be realized.

  Next, the operation of the face image matching system, particularly the face metadata generation unit 12 and the face similarity calculation unit 14 will be described in detail.

(1) Face Metadata Generation The face metadata generation unit 12 extracts a face feature amount using an image I (x, y) whose position and size are normalized. For normalization of the position and size, for example, the image may be normalized so that the eye position is (16, 24), (31, 24), and the size is 46 × 56 pixels. Hereinafter, a case where an image is normalized to this size will be described.

As a face feature amount, feature extraction is performed using a so-called eigenface technique (Moghaddam's paper described above). That is, the basis matrix U specified by the partial basis vector selected from the basis vectors obtained by the principal component analysis of the face image sample set [Λ] with respect to the feature vector Λ having the pixel values in the image as elements. And a feature vector v = U ^T (Λ−ψ) is calculated using the average face Ψ which is an average vector in the face image sample set [Λ], and is used as a feature vector. In this way, the data amount of the input image can be reduced by using the partial basis vector instead of the input image. Reducing the amount of data in this way is an important factor for realizing high-speed matching as well as reducing the amount of storage in the meta database. As the number of dimensions of the feature vector, for example, a 48-dimensional feature amount may be used.

  In addition to the commonly used eigenface technique, partial basis vectors may be defined using a technique that combines discriminant analysis with principal component analysis (the above-mentioned W. Zhao paper). Absent.

  In addition, the base matrix Un specified by the partial base vector selected from the base vectors obtained by the principal component analysis or discriminant analysis of the feature vector set [Λ] in the face image sample as described above, and the face image are horizontally reversed. The matrix U (= aUn + bUm) obtained by linear summation of the basis matrix Um specified by the inverted partial basis vector with the basis vector elements replaced so as to correspond to the conversion of the pixel value to be A face feature vector may be extracted. For example, when a = b = 1, the obtained facial feature vector can extract only components that are symmetric with respect to the left and right transformations in the input image space. Since the face is essentially symmetric, the image component that has become asymmetric due to the effect of lighting and the asymmetric component that occurs because the orientation of the face is not directed to the front is essentially equivalent to noise. By extracting only symmetric components, it is possible to extract a facial feature quantity that is stable against changes in illumination and posture.

  Also, the image is subjected to Fourier transform, and a vector whose element is the size of each component of the obtained complex Fourier component is calculated as a feature vector Λ. The amount may be extracted. In this way, by performing Fourier transform on the image, it is possible to extract a facial feature amount that is strong against displacement. In this way, the facial feature extraction unit 121 extracts the facial feature quantity v.

The reliability index extraction unit 122 extracts a contrast index θ _contrast and an asymmetry index θ _asymmetric that are effective as the reliability index of the facial feature value v. As the contrast index θ _contrast , the standard deviation of the pixel value of the face image I (x, y) is calculated by the following equation (Equation 21).

Here, round () means rounding of numerical values. Values exceeding the range are limited so that the reliability index θ _contrast thus extracted falls within the 4-bit range of [0,1,2, ..., 15]. In the above description, the standard deviation of the image is calculated as the contrast index. However, the variance or the difference between the maximum value and the minimum value of the pixel values in the image may be extracted. The contrast index based on the maximum and minimum pixel values in the image has a smaller amount of computation than the standard deviation and variance, but the effect is relatively small.

As an asymmetry index θ _asymmetric , an average of absolute values (first powers) of differences between the face image I (x, y) and its inverted image is extracted by the following equation (Equation 22).

The value exceeding the range is limited so that the reliability index θ _asymmetric thus extracted falls within the 4-bit range of [0, 1, 2,..., 15]. In the above description, the absolute value (first power) is used as the asymmetric index, but the square of the difference or the like may be used. Further, the same effect can be obtained by using a value such as a sum instead of the average. Further, if the maximum value of the difference is detected and used as an asymmetric index, the amount of calculation can be reduced.

The reliability index extraction unit 122 extracts the feature vector v, the reliability index θ _contrast, and the reliability index θ _asymmetric from the face image, and outputs them as face metadata. As described above, the face metadata generation procedure can be executed by a computer using a computer program.

(2) Face Similarity Calculation Next, the operation of the face similarity calculation unit 14 will be described. In the face similarity calculation unit 14, the distribution estimation unit 141 uses the reliability indices θ _{contrast, 1} and θ _{constrast, 2} and θ _{asymmetric, 1} and θ _{asymmetric, 2} of the two face metadata _, and parameter information relating to the posterior distribution. The distance calculation unit 142 calculates the similarity d between the facial features using the two feature vectors v ₁ and v ₂ of the two face metadata and the parameter information regarding the posterior distribution.

  Here, the case where the face similarity is calculated by (Equation 8) or (Equation 18) will be described.

In this embodiment, the reliability indexes θ ₁ and θ ₂ represented by (Equation 8) or (Equation 18) are vectors, and each element is θ ₁ = (θ _{contrast, 1} , θ _{asymmetric, 1} ). ^T , θ ₂ = (θ _{contrast, 2} , θ _{asymmetric, 2} ) ^T. Since the contrast index and the asymmetry index are each expressed by 4 bits, there are 256 possible states of θ _i . One state among 256 states can be specified by the given reliability index θ _i .

As will be described later, the distribution estimation unit 141 calculates the variance value σε _{, k} (θ) or the variance value σ _{W, k} (θ) of the difference vector with respect to the reliability index θ (256 states) obtained in advance. The variance value σ _{B, k} (θ) is stored in the table, the reliability indices θ ₁ and θ ₂ are used to refer to the values of the respective variance value tables, and the obtained variance values are used as the posterior distribution information. As shown in FIG. In the case of face identification, the values of the variance values σε _{, k} (θ ₁ ) and variance values σε _{, k} (θ ₂ ) required in (Equation 8) may be output to the distance calculation unit 142, and face identification is performed. In this case, the variance value σ _{W, k} (θ) and the variance value σ _{B, k} (θ) necessary for (Equation 18) are output.

  The distance calculation unit 142 calculates an adaptive Mahalanobis distance or an adaptive discrimination distance according to (Equation 8) or (Equation 18), and outputs it as the similarity d.

The variance values in the above-described variance value table are calculated using face image data samples prepared in advance. From the feature vector set [v _i ] and its reliability index [θ _i ] in the face image sample, each variance value can be calculated as follows.

Here, “(i, j) belongs to class W” means that data i and data j are data obtained from the same class (same person) (within the class), and “( “i, j) belongs to class B” means that data i and data j are data obtained from different classes (different persons) (between classes). Nε (θ), N _W (θ), and N _B (θ) are the number of combinations of data belonging to each class. In this way, when calculating the variance value for each bin of θ, if the number of data is too small, the data of neighboring bins are merged to ensure the number of samples (this is K-nearest neighbor method in distribution estimation (similar to merging bin by Bishop literature, pp. 53).

Here, in the case of (Equation 18), as in (Equation 4), σ _{W, k} ([θ _i ]) ² = σ _{W, k} (θ ₁ ) ² + σ _{W, k} (θ ₂ ) ² , Note that σ _{B, k} ([θ _i ]) ² = σ _{B, k} (θ ₁ ) ² + σ _{B, k} (θ ₂ ) ² .

Since the error variance σε _{, k} (θ) ² and the intraclass variance σ _{W, k} (θ) ² are the same, the distances of both (Equation 8) and (Equation 18) are calculated by the face image matching system. In this case, these distributed value tables may be shared.

In addition, since the error distribution and the interclass distribution often have a strong correlation, the reliability index can be obtained even if the interclass variance σ _{B, k} (θ) ² is used instead of the error variance σε _{, k} (θ) ^2. The accuracy is improved compared to the case of not using (but the accuracy is better if error variance is used).

Thus, by calculating the similarity between the face metadata using the posterior distribution information via the reliability index θ _contrast and the reliability index θ _asymmetric , it is possible to perform face recognition with high accuracy. As described above, the computer can execute the face similarity calculation procedure using a computer program.

  Here, the degree of similarity is calculated using (Equation 8) and (Equation 18), but it is also possible to approximate the calculation by various calculation methods such as the following to increase the speed.

By not calculating the second term on the right side of each of the above formulas (the part of ln), it is possible to achieve further high-speed computation.

Also, when calculating the similarity according to (Equation 6) or (Equation 16), basically, from the face image data sample prepared in advance, the covariance matrix Σε of the difference vector error necessary for each calculation is used. Calculate the covariance matrix Σ _W (θ) within the class of (θ) and difference vector, and the covariance matrix Σ _B (θ) between classes, and prepare them as covariance tables. The covariance table may be referred to. In this method, the amount of calculation increases because the distance is calculated using the covariance matrix, but the accuracy of similarity calculation can be improved when there are sufficient learning samples.

By assuming a mixed normal distribution for the posterior distribution of (Equation 3), the interclass distribution of (Equation 14), and the interclass distribution, and estimating the distribution function, respectively, (Equation 10) and (Equation 20) An adaptive mixture Mahalanobis distance or an adaptive mixture discrimination distance may be calculated. In this case, the covariance matrix Σ _s (θ ₁ , j), P (j) representing the mixed normal distribution is obtained from the face image data sample in the same manner as calculating the posterior distribution information using the variance and the covariance matrix. The parameters for specifying the mixture distribution, such as, may be obtained and stored as a table. This estimation may be performed using a general likelihood estimation method such as a maximum likelihood estimation method or an EM algorithm.

  In the above description, a case where a single face image is registered and retrieval is performed using a single face image has been described. However, a plurality of images are registered for one face, and a single face image is registered. When using the search, for example, the following may be performed.

The search-side feature vector is v _que , the registration-side feature vector is v _{reg, k,} and the similarity d _multi (v _que , [v _{reg, 1} , v _{reg, 2} , ... , v _{reg, n} ]), the similarity may be calculated based on the calculation formulas shown in the following equations (Equations 27 and 28).

あるいは、

Or

  Similarly, in the case of registration of a plurality of images per face and search by a plurality of images, the similarity for one face data is calculated by calculating the similarity by calculating the average or the minimum value of the similarity of each combination. can do. This means that the matching system of the present invention can be applied to face recognition in a moving image by imitating the moving image as a plurality of images.

  Also, in the above explanation, the explanation was centered on the identification of the same person's face and the identification of the face, but for example, considering the male face as one category and the female face as one category, information on each distribution is given. It can also be applied to facial expression recognition for recognizing facial expressions in consideration of gender identification that identifies male and female faces, or considering facial expression categories such as laughed faces, angry faces, and sad faces. Age can be determined by setting categories such as teens, 20s, 30s and 40s, and obtaining error distribution, intraclass distribution and interclass distribution for each category. The present invention can be applied to face recognition.

It is a block diagram which shows the structure of the face image matching system by one Embodiment of this invention.

Explanation of symbols

11: Face image input part
12: Face metadata generator
13: Face metadata storage
14: Face similarity calculator
15: Face image database
16: Control unit
17: Display
121: Facial feature extraction unit
122: Reliability index extraction unit
141: Distribution estimation part
142: Distance calculator

Claims (22)

In the face similarity calculation device for calculating the similarity of the face image based on the face feature amount extracted from the face image and the reliability index representing the reliability of the face recognition result using the face feature amount,
The statistics of the distribution related to the comparison amount between the facial feature quantities are stored in advance for each reliability index value, and the statistics corresponding to the reliability index value is used as a parameter of the posterior distribution of the comparison quantity. Distribution estimation means to output;
Distance calculating means for outputting, as the similarity, a distance between face feature amounts calculated based on a posterior distribution of the comparison amount using the comparison amount and the parameter;
A face similarity calculation device characterized by comprising: The distribution estimating means stores in advance a statistic of the distribution of the difference vector between facial feature quantities for each reliability index value, and the statistic corresponding to the reliability index value is stored in the difference vector. Output as a parameter of the posterior distribution,
The distance calculation means outputs, as the similarity, the distance between facial feature amounts calculated using a distance function derived from a log likelihood of the posterior distribution of the difference vector, using the difference vector and the parameter. ,
The face similarity calculation device according to claim 1.   The distance calculation means calculates an adaptive Mahalanobis distance as a similarity degree derived from a logarithmic likelihood of a normal distribution in the reliability index, assuming that a posterior distribution of a difference vector between facial feature quantities is a normal distribution. The face similarity calculation device according to claim 2. The distribution estimation means estimates a variance σ _{s, k} ([θ _i ]) ² of each element k of the difference vector s with respect to the reliability index [θ _i ],
The distance calculation means calculates an adaptive Mahalanobis distance using the variance σ _{s, k} ([θ _i ]) ² of each element k.
The face similarity calculation device according to claim 3. The distribution estimation means includes a variance value table that stores in advance the variance σ _{s, k} ([θ _i ]) ² of each element k of the difference vector s with respect to the reliability index [θ _i ], and includes the reliability index [θ _i 5. The face similarity calculation device according to claim 4, wherein a variance value required for calculating an adaptive Mahalanobis distance is output by referring to the variance value table.   The distance calculation means calculates the adaptive mixed Mahalanobis distance derived from the logarithmic likelihood of the mixed distribution in the reliability index as the similarity, assuming that the posterior distribution of the difference vector between the facial feature quantities is a mixed distribution. The face similarity calculation apparatus according to claim 2, wherein: In the face similarity calculation method for calculating the similarity of the face image based on the face feature amount extracted from the face image and the reliability index representing the reliability of the face recognition result using the face feature amount,
The statistics of the distribution related to the comparison amount between the facial feature quantities are stored in advance for each reliability index value, and the statistics corresponding to the reliability index value is used as a parameter of the posterior distribution of the comparison quantity. An output distribution estimation step;
A distance calculating step of outputting, as the similarity, a distance between face feature amounts calculated based on a posterior distribution of the comparison amount using the comparison amount and the parameter;
A face similarity calculation method characterized by comprising: In the distribution estimation step, a statistic of a distribution of difference vectors between facial feature quantities is stored in advance for each reliability index value, and the statistic corresponding to the reliability index value is stored in the difference vector. Output as a parameter of the posterior distribution,
The distance calculating step outputs the distance between the facial feature quantities calculated using a distance function derived from a log likelihood of the posterior distribution of the difference vector using the difference vector and the parameter as the similarity. ,
The face similarity calculation method according to claim 7.   9. The adaptive Mahalanobis distance derived from the logarithmic likelihood of the normal distribution in the reliability index is calculated as the similarity, assuming that the posterior distribution of the difference vector between the facial feature quantities is a normal distribution. The face similarity calculation method described. Estimate the variance σ _{s, k} ([θ _i ]) ² of each element k of the difference vector s with respect to the reliability index [θ _i ],
Calculating an adaptive Mahalanobis distance using the variance σ _{s, k} ([θ _i ]) ² of each element k,
The face similarity calculation method according to claim 9. Variance sigma _{s, k} for each element k of the difference vector s for the confidence index _{[θ i] ([θ i} ]) 2 stored in advance in the dispersion value table, the variance value table by the reliability index [theta _i] 11. The face similarity calculation method according to claim 10, wherein a variance value required for calculating the adaptive Mahalanobis distance is generated by referring to the face similarity.   The adaptive mixed Mahalanobis distance derived from the logarithmic likelihood of the mixed distribution in the reliability index is calculated as the similarity, assuming that the posterior distribution of the difference vector between the facial feature quantities is a mixed distribution. The face similarity calculation method described in 1. The distribution estimation means includes a first statistic of a difference vector distribution (intraclass distribution) between facial feature quantities in a class that should be regarded as a match, and a facial feature quantity between classes that should be regarded as a mismatch. The second statistic of the difference vector distribution (interclass distribution) is stored in advance for each reliability index value, and the first statistic and the second statistic corresponding to the reliability index value are stored. The statistics are output as the parameters of the intraclass distribution and the interclass distribution, respectively.
The distance calculation means uses the difference vector and the parameter to calculate the distance between the facial feature quantities calculated using a distance function derived from a logarithmic likelihood of a ratio between the intraclass distribution and the interclass distribution. Output as similarity,
The face similarity calculation device according to claim 1. The adaptive discriminant distance derived from the log likelihood of the ratio of the respective distributions in the reliability index is calculated as the similarity, assuming that the intraclass distribution and the interclass distribution are normal distributions, respectively. Item 14. The face similarity calculation device according to Item 13. By estimating the intra-class variance σ _{W, k} ([θ _i ]) ² and the inter-class variance σ _{B, k} ([θ _i ]) ² of each element k of the difference vector s with respect to the reliability index [θ _i ] The face similarity calculation device according to claim 14, wherein the adaptive discrimination distance is calculated as a similarity. For estimation of intra-class variance σ _{W, k} ([θ _i ]) ² and inter-class variance σ _{B, k} ([θ _i ]) ² of each element k of difference vector s with respect to reliability index [θ _i ] A first variance value table that pre-stores intra-class variance σ _{W, k} ([θ _i ]) ^2, and a second variance value table that pre-stores inter-class variance σ _{B, k} ([θ _i ]) ² With
16. The face similarity according to claim 15, wherein a variance value required in an adaptive discrimination distance is estimated by referring to each of the first and second variance value tables according to the reliability index [θ _i ]. Calculation device. Assuming that the intraclass distribution and the interclass distribution are each a mixture distribution, an adaptive mixture discrimination distance derived from a logarithmic likelihood of a ratio of each mixture distribution in the reliability index is calculated as a similarity. The face similarity calculation device according to claim 13. The distribution estimation step includes a first statistic of a distribution of a difference vector between facial feature quantities (class distribution) in a class that should be regarded as a match, and a facial feature quantity between classes to be regarded as a mismatch. The second statistic of the difference vector distribution (interclass distribution) is stored in advance for each reliability index value, and the first statistic and the second statistic corresponding to the reliability index value are stored. The statistics are output as the parameters of the intraclass distribution and the interclass distribution, respectively.
The distance calculating step uses the difference vector and the parameter to calculate the distance between the face feature values calculated using a distance function derived from a logarithmic likelihood of a ratio of the intraclass distribution and the interclass distribution. Output as similarity,
The face similarity calculation method according to claim 7. The adaptive discriminant distance derived from the log likelihood of the ratio of the respective distributions in the reliability index is calculated as the similarity, assuming that the intraclass distribution and the interclass distribution are normal distributions, respectively. Item 19. The face similarity calculation method according to Item 18. By estimating the intra-class variance σ _{W, k} ([θ _i ]) ² and the inter-class variance σ _{B, k} ([θ _i ]) ² of each element k of the difference vector s with respect to the reliability index [θ _i ] 20. The face similarity calculation method according to claim 19, wherein the adaptive discrimination distance is calculated as the similarity. For estimation of intra-class variance σ _{W, k} ([θ _i ]) ² and inter-class variance σ _{B, k} ([θ _i ]) ² of each element k of difference vector s with respect to reliability index [θ _i ] A first variance value table that pre-stores intra-class variance σ _{W, k} ([θ _i ]) ^2, and a second variance value table that pre-stores inter-class variance σ _{B, k} ([θ _i ]) ² With
21. The face similarity according to claim 20, wherein a variance value required in the adaptive discrimination distance is estimated by referring to the first and second variance value tables respectively by the reliability index [θ _i ]. Calculation method. Assuming that the intraclass distribution and the interclass distribution are each a mixture distribution, an adaptive mixture discrimination distance derived from a logarithmic likelihood of a ratio of each mixture distribution in the reliability index is calculated as a similarity. The face similarity calculation method according to claim 18.

Images