JP3811474B2 - Face part position detection method and face part position detection apparatus - Google Patents

Description

  The present invention relates to a face part position detection method and a face part position detection apparatus.

As a method for detecting a face region (eye, nose, mouth, etc.) by detecting a face region from the obtained image, for example, there is a method for combining four-way surface features and color information (see Non-Patent Document 1). .
In addition, it is conceivable to use a technique using pattern matching as disclosed in Patent Document 1 for face part detection. In pattern matching, face part detection is performed by collating a face part pattern registered in advance with an acquired image.
Hitoshi Hongo, Kazuhiko Yamamoto, “Facial region and facial part extraction by estimating skin color in moving region”, Journal of the Institute of Image Information and Television Engineers, The Institute of Image Information and Television Engineers, December 1998, Vol. 52, No. 12, p. 86-93 JP-A-7-141506

However, with these methods, when some facial parts are concealed, the position of the facial parts that are not concealed cannot be detected with high accuracy.
An object of the present invention is to accurately detect the position of a face part that is not concealed even when some face parts are partially or completely concealed.

According to the first aspect of the present invention, the template matching of the four-way surface feature is performed on the face area in the captured image, the initial similarity in the template matching of the four-way surface feature, the specific face part, and the specific face. A facial part position detection method for obtaining a peripheral matching probability representing a relationship with a peripheral facial part of the part, and detecting a position of the specific facial part by a relaxation matching method using the initial similarity and the peripheral matching probability In order to derive the peripheral matching probability , a neighborhood correspondence probability that is the maximum value obtained by multiplying the correspondence probability at each position of the face region by a Gaussian distribution for each specific face part is obtained and multiplied by the neighborhood correspondence probability. When the peripheral matching probability of each facial part is obtained by calculation, the peripheral matching probability is calculated by eliminating at least one of the neighborhood correspondence probabilities that are not the maximum value.

According to a second aspect of the present invention, image matching means for capturing a face, and template matching of four-directional surface features are performed on a face area in an image captured by the imaging means, and the template matching of the four-directional surface features is performed. A template matching means for calculating an initial similarity in the image, a peripheral matching probability representing a relationship between the specific facial part and a peripheral facial part of the specific facial part, and obtaining the initial similarity and the peripheral matching probability. by relaxation matching method using configure the facial parts position detection device and a relaxation aligning means for detecting the position of the facial parts, the relaxation aligning means for guiding said peripheral adaptation probability, face each particular face parts The neighborhood correspondence probability, which is the maximum value of the correspondence probability at each position in the region multiplied by the Gaussian distribution, is calculated and multiplied by the neighborhood correspondence probability. When obtaining the engagement probability, characterized by eliminating at least one of the neighboring corresponding probability not maximum value that was assumed to determine the peripheral adaptation probability.

  In the present invention, even when some facial parts are concealed, there is an excellent effect that the position of the facial parts that are not concealed can be detected with high accuracy.

DESCRIPTION OF EMBODIMENTS Hereinafter, an embodiment embodying the present invention will be described with reference to FIGS.
As shown in FIG. 1A, the face part position detection device 10 includes a video camera 11 as an imaging unit and a computer 12 including a display device 121. The video camera 11 is a CCD camera. Individual frames (image data) photographed by the video camera 11 are input to the computer 12 as color images. M is a person in the shooting range of the video camera 11. F is the face of person M. The computer 12 causes the display device 121 to display an image captured by the video camera 11.

FIG. 1B is a functional block diagram of the computer 12.
The image acquisition unit 13 acquires an image from the video camera 11. An image G in FIG. 1A shows an example of an image acquired by the image acquisition unit 13. In the image G, f (1) represents a right eye that is a facial part, and f (2) represents a left eye that is a facial part. f (3) represents a mouth which is a facial part, and f (4) represents a nose which is a facial part.

  The face area detection unit 14 detects a face area from the image G (see FIG. 1A) acquired by the image acquisition unit 13. The detection of the face area is performed using, for example, a technique disclosed in Non-Patent Document 1 (a technique for detecting a face area by combining color information and motion information). Data necessary for this method (for example, face pattern data for pattern matching, reference skin color data, etc.) is stored in the storage means 16.

  The template matching unit 15 performs template matching of the four-direction surface features. The four-direction plane feature is a feature amount created by dividing into four directions of a horizontal direction, a right-up direction, a vertical direction, and a right-down direction depending on the edge direction in the image. In other words, the four-direction plane feature is a vector field in four directions (horizontal direction, right-up direction, vertical direction, right-down direction) at each pixel based on the gradient of the gray value in the image, and the gray value is determined in each direction. This is a feature amount that is divided into another surface and subjected to a Gaussian filter.

  The storage unit 16 stores templates T1, T2, T3, and T4 of four-direction surface features. T1 is a template related to the right eye, T2 is a template related to the left eye, T3 is a template related to the mouth, and T4 is a template related to the nose. Hereinafter, the templates T1, T2, T3, and T4 are represented as Ti (i = 1, 2, 3, 4). i represents a face part. The template matching unit 15 performs template matching on the detected face region using the stored template Ti of the four-directional surface features. Then, the template matching unit 15 calculates the initial similarity of the four-direction plane feature based on this template matching.

The relaxation matching means 17 detects the position of the facial part using the relaxation matching method.
FIG. 3 is a flowchart showing a face part position detection program executed by the computer 12. Hereinafter, face detection by the face part position detection apparatus 10 will be described with reference to this flowchart.

  In step S <b> 1, first, the image acquisition unit 13 determines whether or not to acquire an image from the video camera 11. That is, in the present embodiment, the image acquisition from the video camera 11 is performed at predetermined time intervals t (for example, t = 0.1 seconds), and the image acquisition unit 13 performs image acquisition. It is determined whether it is time. If it is determined that it is time to acquire an image (YES in step S1), the image acquisition unit 13 acquires an image from the video camera 11 (step S2). On the other hand, when it is determined that the image acquisition unit 13 does not acquire the image (NO in step S1), step S1 is repeated.

After acquiring the image from the video camera 11, the face area detecting means 14 searches for the face area in the acquired image (step S3), and determines whether there is a face area in the image (step S4). ). If it is determined that there is no face area (NO in step S4), the computer 12 returns to step S1. When it is determined that there is a face area (YES in step S4), the template matching unit 15 scans the face area with the template Ti and performs template matching of the four-directional surface features on the face area. An initial similarity Γik ⁰ is calculated (step S5). The images in FIGS. 2A and 2B represent the face area f detected from the image G by the face area detecting means 14.

In the present embodiment, the resolution of the four-way plane feature is reduced to 1/8 of the input resolution, and the resolution of the four-way plane feature of each facial part i is 8 × 4. The initial similarity Γik ⁰ is expressed as in Expression (1).

式（１）におけるｎは、顔領域におけるテンプレートＴｉによって切り取られた数（本実施形態では１２８通りの数）を表し、Ｉｋ（ｎ）は、テンプレートＴｉによって顔領域における位置（Ｘｋ，Ｙｋ）（Ｘ，Ｙは、２次元座標を表す）で切り出した４方向面特徴を表す。位置（Ｘｋ，Ｙｋ）は、テンプレートＴｉの中心の座標である。

In Expression (1), n represents the number cut by the template Ti in the face area (128 numbers in this embodiment), and Ik (n) represents the position (Xk, Yk) ( X and Y represent four-direction plane features cut out in two-dimensional coordinates). The position (Xk, Yk) is the coordinates of the center of the template Ti.

The template matching unit 15 normalizes the plurality of initial similarities Γik ⁰ obtained for each template Ti by dividing them by their maximum values. The normalized initial similarity Pik ⁰ (hereinafter referred to as initial correspondence probability) is expressed by the following equation (2).

ステップＳ５の処理後、弛緩整合手段１７は、弛緩整合法を用いて顔部品の位置を決定する（ステップＳ６）。ステップＳ６では以下のような処理が行われる。まず、弛緩整合手段１７は、式（３）に基づいて近傍対応確率Ｑｊｅ^０を算出する。

After the processing of step S5, the relaxation matching means 17 determines the position of the facial part using the relaxation matching method (step S6). In step S6, the following processing is performed. First, the relaxation matching means 17 calculates the neighborhood correspondence probability Qje ⁰ based on the equation (3).

ｊ（＝１，２，３，４であってｊ≠ｉ）は、顔部品ｉに対する周辺の顔部品を表す。σは、ガウス分布の標準偏差である。ｅは、顔部品ｊに対する顔領域内での相対位置（Ｘｅ，Ｙｅ）を表す。Ｘｅは、式（４）で表され、Ｙｅは式（５）で表される。

j (= 1, 2, 3, 4 and j ≠ i) represents a peripheral face part with respect to the face part i. σ is the standard deviation of the Gaussian distribution. e represents a relative position (Xe, Ye) within the face region with respect to the face part j. Xe is represented by Formula (4), and Ye is represented by Formula (5).

式（３）におけるｅ′は、顔領域の全体を表し、ｅ′の位置を（Ｘｅ′，Ｙｅ′）とすると、ｅ，ｅ′間の距離Ｄｅｅ′は、式（６）で表される。

In equation (3), e ′ represents the entire face region, and when the position of e ′ is (Xe ′, Ye ′), the distance Dee ′ between e and e ′ is represented by equation (6). .

式（３）は、顔領域での位置ｅにおける対応確率Ｐｊｅ^０にガウス分布を掛けて最大値を算出する計算式である。式（３）において、相対位置（Ｘｅ，Ｙｅ）が顔領域外にある場合には、Ｑｊｅ^０＝０としている。

Expression (3) is a calculation expression for calculating the maximum value by multiplying the corresponding probability Pje ⁰ at the position e in the face region by the Gaussian distribution. In Expression (3), when the relative position (Xe, Ye) is outside the face area, Qje ⁰ = 0.

Next, the relaxation matching means 17 calculates the peripheral matching probability qik ⁰ based on the equation (7).

周辺適合確率ｑｉｋ^０は、顔部品ｉに対する全ての周辺の顔部品ｊのうちの選ばれた顔部品ｊに関して、近傍対応確率Ｑｊｅ^０を掛け合わせて得られるものである。本実施形態では、弛緩整合手段１７は、（７）式に用いる近傍対応確率Ｑｊｅ^０として、値が最大の近傍対応確率と、次に大きな値の近傍対応確率とを選択する。例えば、顔部品ｉ＝１（右目）に関してＱ２ｅ^０が最大値であってＱ３ｅ^０が次に大きな値であった場合、式（７）は、次式（８）となる。

The peripheral matching probability qik ⁰ is obtained by multiplying the selected face part j among all the peripheral face parts j for the face part i by multiplying by the neighborhood correspondence probability Qje ⁰ . In the present embodiment, the relaxation matching means 17 selects the neighborhood correspondence probability having the maximum value and the neighborhood correspondence probability having the next largest value as the neighborhood correspondence probability Qje ⁰ used in the equation (7). For example, when Q2e ⁰ is the maximum value and Q3e ⁰ is the next largest value for the face part i = 1 (right eye), Expression (7) becomes the following Expression (8).

次に、弛緩整合手段１７は、式（９）に基づいて、対応度Γｉｋ^ｔを算出する。ｔは、１，２，３・・・を表す。

Next, the relaxation matching means 17 calculates the correspondence Γik ^t based on the equation (9). t represents 1, 2, 3,.

周辺適合確率ｑｉｋ^ｔは、式（１０）で表される。

The peripheral matching probability qik ^t is expressed by Expression (10).

弛緩整合手段１７は、式（１０）を用いた演算においても、式（１０）に用いる近傍対応確率Ｑｊｅ^ｔとして、値が最大の近傍対応確率と、次に大きな値の近傍対応確率とを選択する。例えば、顔部品ｉ＝１（右目）に関してＱ２ｅ^ｔが最大値であってＱ３ｅ^ｔが次に大きな値であった場合、式（１０）は、次式（１１）となる。

Relaxation matching means 17, also in the operation using the equation (10), selected as a neighboring corresponding probability Qje ^t used in equation (10), and the maximum neighboring corresponding probability value, and a neighborhood corresponding probability of the next largest value To do. For example, if Q2E ^t regard face part i = 1 (right eye) were then large value Q3e ^t a maximum value, equation (10) becomes the following equation (11).

そして、弛緩整合手段１７は、顔部品ｉにおける対応度Γｉｋ^ｔをそれらのうちの最大値で割って正規化された対応確率Ｐｉｋ^ｔを算出する。対応確率Ｐｉｋ^ｔは、式（１２）で表される。

Then, the relaxation matching unit 17 calculates the normalized correspondence probability Pik ^t by dividing the correspondence degree Γik ^t of the facial part i by the maximum value of them. The correspondence probability Pik ^t is expressed by Expression (12).

弛緩整合手段１７は、式（９），（１０），（１２）を用いた演算を所定回数ｔ（本実施形態では４回）繰り返す。

The relaxation matching unit 17 repeats the calculation using the equations (9), (10), and (12) a predetermined number of times t (four times in the present embodiment).

After the processing in step S6, the relaxation matching unit 17 stores k, which is Pik ⁴ = 1 in the last obtained correspondence probability Pik ⁴ , in the storage unit 16 as the position of the face part i (step S7).

In this embodiment, the following effects can be obtained.
(1-1) Images g (01), g (02), g (03), and g (04) in FIG. 2C represent face part images obtained based on the calculated initial correspondence probability Pik0. . Images g (11), g (12), g (13), and g (14) represent face part images obtained based on the calculated corresponding probability Pik ¹ , and images g (21) and g (22). , g (23), g ( 24) represents a face part image obtained based on the calculated corresponding probability Pik ^2. Images g (31), g (32), g (33), and g (34) represent face part images obtained based on the calculated corresponding probability Pik ³ , and images g (41) and g (42). , g (43), g ( 44) represents a face part image obtained based on the corresponding probability Pik ⁴ calculated. The images g (41), g (42), g (43), and g (44) represent the position of the face part i determined by setting k at which Pik ⁴ = 1 as the position of the face part i. FIG. 2 (c) shows the changing process of the correspondence probability Pik ^t. The degree of correspondence at the correct position of the face part i is emphasized, and the degree of correspondence at a different position of the face part i is suppressed. You can see how it goes.

In the image G in FIG. 1A, the left eye that is a facial part is concealed. Frames W1, W3, and W4 in FIG. 2A schematically represent the face part positions detected by the face part position detection apparatus 10. In other words, the frame W1, W3, W4 in FIG. 2 (a), as a neighboring corresponding probability Qje ^t used in equation (10), the value is computed by selecting the next larger neighboring corresponding probability and the maximum in the vicinity of the corresponding probability It is a schematic representation of the face part position detected in this case. Frame H1, H3 in FIG. 2 (b), H4 schematically facial parts position detected when operation by selecting all neighboring corresponding probability Qje ^t as a neighboring corresponding probability Qje ^t used in equation (10) It is shown in As shown in FIG. 2B, the position of the face part that is not concealed becomes inaccurate because the peripheral matching probability qik ^t is lowered by concealing other face parts.

In the present embodiment, as a neighboring corresponding probability Qje ^t used in equation (10), for example, the value selects the next larger neighboring corresponding probability and the maximum in the vicinity of the corresponding probabilities. In other words, since the expression is a value as near corresponding probability Qje ^t used in (10) to eliminate the smallest near the corresponding probability, even if is concealed part of the face parts as shown in FIG. 2 (a), The position of the face part that is not concealed can be detected with high accuracy.

(1-2) Detecting the position of a face part with high accuracy even when some face parts are partially hidden or the face is inclined or sideways and the shape of the face part changes. Can do.
In the present invention, the following embodiments are also possible.

(1) as a neighboring corresponding probability Qje ^t used in equation (10), the value may be selected only up to the vicinity corresponding probability. The neighborhood correspondence probabilities to be excluded are a plurality of neighborhood correspondence probabilities arranged successively in order of increasing values.

(2) The ear may be a facial part.
(3) Eyebrows may be used as facial parts.
(4) The edge of the eye or the edge of the mouth may be a facial part. That is, a part of the eyes and a part of the mouth can be regarded as face parts.

The technical idea that can be grasped from the embodiment described above will be described below.
[1] The face part position detecting method according to claim 1, wherein the excluded neighborhood correspondence probability includes at least a neighborhood correspondence probability of a minimum value.

  [2] The face part position according to any one of [1] and [1], wherein the excluded neighborhood correspondence probabilities are a plurality of neighborhood correspondence probabilities arranged successively in order of increasing value. Detection method.

[3] The position of the face part according to any one of [1], [1], and [2], wherein the excluded neighborhood correspondence probabilities are all except the neighborhood correspondence probability having the maximum value. Detection method.
[4] After detecting the face area from the captured image, the template matching of the four-direction surface feature is performed on the detected face area, and any one of [1] and [2] The method for detecting the position of a facial part as described in the item.

  [5] The image processing apparatus includes a face area detection unit that detects a face area from the captured image, and the template matching unit performs template matching of four-direction surface features on the face area detected by the face area detection unit. Item 3. The face part position detection apparatus according to Item 2.

FIG. 3A is a block diagram illustrating a face detection device according to an embodiment. (B) is a functional block diagram of a computer. (A), (b) is an image figure showing a face area. (C) is explanatory drawing which shows the change of corresponding | compatible probability Pik ^t . The flowchart showing a face component position detection program.

Explanation of symbols

10: Facial component position detection device. 11: Video camera (imaging means). 12: Computer (face area detection means, template matching means, relaxation matching means). 14. Face area detecting means. 15: Template matching means. 17: Relaxation matching means. G ... Image. f: Face area. f (1): Facial part (right eye). f (2): Face part (left eye). f (3): Facial part (mouth). f (4): Face part (nose). Γik ⁰ ... Initial similarity. qik ^t ... peripheral matching probability. Qje ^t ... Neighborhood correspondence probability.

Claims (2)

The template matching of the four-way plane feature is performed on the face area in the captured image, the initial similarity in the template matching of the four-way plane feature, the specific facial part, and the peripheral facial parts around the specific facial part, In a face part position detection method for detecting a position of the specific face part by a relaxation matching method using the initial similarity and the peripheral match probability,
In order to derive the peripheral matching probability , by calculating the neighborhood correspondence probability which is the maximum value of the value obtained by multiplying the correspondence probability at each position of the face area by a Gaussian distribution for each specific face part, and multiplying the neighborhood correspondence probability by A face part position detection method for obtaining the peripheral matching probability by eliminating at least one of the neighborhood correspondence probabilities which is not the maximum value when determining the peripheral matching probability of each facial part . Imaging means for imaging the face;
Template matching means for performing template matching of the four-way plane feature on the face area in the image captured by the imaging means, and calculating initial similarity in the template matching of the four-way plane feature;
A peripheral matching probability representing the relationship between a specific facial part and the peripheral facial parts of the specific facial part is obtained, and the position of the facial part is detected by the relaxation matching method using the initial similarity and the peripheral matching probability. And a relaxing alignment means for
The relaxation matching means obtains a neighborhood correspondence probability that is a maximum value of a value obtained by multiplying the correspondence probability at each position of the face region by a Gaussian distribution for each specific facial part in order to derive the peripheral matching probability, and A face part position detecting apparatus that obtains the peripheral fit probability by eliminating at least one of the neighborhood correspondence probabilities that are not the maximum value when obtaining the peripheral fit probability of each face part by an operation of multiplying the probabilities .

Images