JP3319408B2 - Facial feature extraction device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for extracting facial features, and more particularly, to an accurate eye position candidate even when it is not possible to clearly distinguish between a white eye and a black eye as in an infrared image. The present invention relates to a facial feature extraction device capable of extracting a region that becomes

[0002]

2. Description of the Related Art Japanese Unexamined Patent Publication (Kokai) No. 3-533 discloses a method for detecting an iris having a radius R.
A technique for detecting dark circular regions of pixels is described.
According to this technique, the horizontal and vertical directions intersecting a circular area having a radius of R pixels are used.
A rectangular area in the direction is set, and the difference between the sum of the luminance values of the part inside the circular area of the rectangular area and the sum of the luminance values of the part outside the circular area is calculated. This difference has a maximum value when the circular area having the radius R pixels conforms to the size of the iris (pupil). Therefore, the difference is compared with a predetermined threshold to determine the position of the iris and the eye. It is determined whether or not is opened and closed.

However, according to the above-mentioned conventional technique, when it is attempted to detect the position of the iris, if the white and black eyes cannot be clearly distinguished due to individual differences, the position of the iris cannot be detected. There is a problem that can not be. Further, as shown in FIG. 3, in the case of an infrared image captured by irradiating infrared light, it is often difficult to clearly distinguish between the white eye and the black eye, and thus it is sometimes difficult to detect the position of the iris. Many. Further, in the related art, when calculating the iris, regions in two vertical and horizontal directions are used, and therefore, it is difficult to match with a circular boundary having a different shape.

Further, Japanese Patent Application Laid-Open No. Hei 7-93561, and a paper entitled "Extraction of Facial Features by Combination of Shape Extraction and Pattern Matching" (Transactions of the Institute of Electronics, Information and Communication Engineers, D-II Vol.
J80-D-II No.3 pp. 2170-2177
(August 1997) describes a technique for detecting the position of an eye using a separation filter. With this technology,
As shown in FIG. 1A, the radius R1 and the radius R2 (> R
Using the window 8 adapted to the shape (circle) of the iris by the concentric circle of 1), the area 1 in the circle with the radius R1 and the area 1 in the circle with the radius R1 are removed from the area in the circle with the radius R2. The degree of separation from the area 2 is calculated, and an area having a high degree of separation is extracted. The degree of separation takes into account the variance of each of the area 1 and the area 2 and the variance of the entire window. And the radius R1
Since the variance value is maximized when the region 1 of conforms to the size of the iris (pupil), the edge and the contour of the eye can be extracted.

FIG. 1B uses a window 6 composed of two large and small squares shown in FIG. 1C and arranged concentrically and having the same function as a concentric window 8 having a radius R1 and a radius R2. 3 shows a simulation result of detecting the position of the eye with a separation filter. FIG.
(D) is a diagram showing the relationship between the magnitude of the degree of dispersion (large, medium, 0) and the magnitude of the luminance in the area 1 and the area 2.

In the prior art, since the window has a circular area 1 conforming to the shape of the iris, as can be understood from FIG. 1B, when the eyes of the original image are open, That is, when the iris can be detected, the position of the eye can be detected.

However, in the above-mentioned conventional technique, since the position of the eye is detected using a window having a circular area adapted to the shape of the iris, the iris is detected as in the case where the eye is thin. If this is not possible, there is a problem that the position of the eye cannot be detected regardless of the visible light image and the infrared image.

That is, in the prior art, when the same threshold value as that in the case where the eyes in FIG. 1 are open is set and the position of the eyes of the original image in which the eyes are narrow as shown in FIG. An output result of the degree of dispersion filter as shown in FIG. 2B is obtained, and as shown in FIG. 2C, candidate points of the eye position disappear, so that the eye position cannot be detected. On the other hand, if the threshold value is set to a low value, as shown in FIG. 2D, many candidate points appear in addition to the eye position candidates, so that the eye position cannot be detected.

[0009] The causes of narrowing of the eyes include individual differences, eyes closed, and the face not facing the photographing apparatus.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and relates to the shape of the eyes of an original image (the eyes are thin, the eyes are closed, the eyes are open, etc.). It is another object of the present invention to provide a facial feature extracting device capable of extracting facial features such as eyes.

[0011]

In order to achieve the above-mentioned object, the present invention provides an image processing system comprising at least three rectangular areas which are set so as to be adjacent to each other in the vertical direction of a face on an input face image. A brightness average value calculation means for calculating a brightness average value in each rectangular area in the window; a calculation means for calculating a difference between brightness average values of adjacent rectangular areas and a sum of the brightness average value differences; Extracting means for comparing the difference between the average value and the sum of the differences between the luminance average values with a threshold value and extracting a pixel larger than the threshold value.

In the present invention, the luminance average calculating means
An average luminance value in each rectangular area in a window including at least three rectangular areas set so as to be adjacent and continuous in the vertical direction of the face on the input face image is calculated. Since the eye portion on the face image has a difference in luminance along the vertical direction of the face, the calculation means calculates the difference between the average luminance values of the adjacent rectangular areas and the sum of the differences between the average luminance values. The extraction means is
The difference between the average brightness values and the sum of the differences between the average brightness values are compared with a threshold value, and pixels larger than the threshold value are extracted.

The extracted pixels are pixels in which the difference between the average brightness values and the sum of the differences between the average brightness values are large in adjacent rectangular regions, so that a candidate region representing the feature of the eye is extracted. The position of the eye can be detected by performing a binarization process by comparing the values of the pixels in the pixel with a threshold value. Further, using the detected position of the eye as a reference, other characteristic portions of the face such as eyebrows can be detected.

When performing the binarization process, a histogram representing the frequency distribution of the sum of the average luminance values is created, and the luminance average when the integrated value integrated from the higher frequency becomes a predetermined ratio with respect to all frequencies is generated. The sum of the values can be used as the threshold.

According to the present invention, regardless of the opening and closing of the eyes, and even when the white and black eyes cannot be distinguished from each other as is often the case with an infrared image, the eyes are narrow. However, it is possible to detect the position of the eye, which is one of the features of the face. Also, since a rectangular window is used,
The position of the eye can be detected even when the eye is photographed thinly due to differences in the shape of the eye due to individual differences, eyes closed or the face not facing front.

In the present invention, if the number of rectangular areas is increased to four or more by dividing the rectangular area located at the top of the window into two equal parts, etc., the eyes and eyebrows, which are features of the face, are obtained. And can be detected simultaneously.

[0017]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment in which the present invention is applied to the case of extracting an eye which is one of the features of a face will be described below with reference to the drawings.

In the first embodiment, as shown in FIG.
A camera 10 for photographing a face, a computer for image processing,
And a display device 28 which is connected to a computer and which is configured by a CRT for displaying an extracted binary image or the like.

The computer for image processing is a camera 10
Analog-to-digital (A / D) converter 12 for digitally converting an image signal inputted from the CPU, a CPU 14, and a ROM 1 storing a program of a face image extraction processing routine.
6, RAM 18, image memory 2 for storing image data
0, an average value register 22 for storing the average luminance value, a histogram memory 24 for storing a histogram representing the frequency distribution of the total average luminance value, an output port 26 connected to the display device 28, a value for the total luminance average value, It is composed of an image memory 32 to which 0 is written, and a bus 30 such as a control bus or a data bus connecting these.

Next, a face image extraction processing routine according to the first embodiment will be described with reference to FIG.

The image signal input from the camera 10 is A
The image data is converted into digital image data by the / D converter 12 and stored in the image memory 20. In the present embodiment, as shown in FIG. 6, a rectangular area 1, a rectangular area 2, and a rectangular area 2 are set from the upper side to the lower side so as to be adjacent and continuous in the vertical direction of the face on the face image of the image data. And rectangular area 3
A rectangular window 40 composed of three rectangular areas is used. In this window 40, the rectangular area 1 and the rectangular area 3 have the same shape and the same area, and the width of the rectangular area 2 is the same as that of the rectangular area 1 and is larger than the area of the rectangular area 1. Have.

First, the face image data is read from the image memory 20, and a window 40 is set on the input face image so that three rectangular regions are continuous in the vertical direction of the face image.

In step 100 of FIG. 5, the luminance integrated values are calculated by integrating the luminance values of all the pixels in each rectangular area. In step 102, the luminance average value P1, P2, P3 in each rectangular area is divided by dividing the integrated luminance value in each rectangular area by the total number of pixels in each rectangular area.
(Refer to FIG. 7) is calculated and stored in the average brightness value register 22 according to the position of the window.

In the next step 104, the differences (P1-P2) and (P3-P2) between the average luminance values in the adjacent rectangular areas.
Is compared with a threshold value K1 to determine whether or not the following condition is satisfied.

(P1-P2> K1) and (P3-P2> K1) When it is determined in step 104 that the above condition is satisfied, that is, since the difference between the average luminance values in adjacent rectangular regions is large, If it is determined that the area is a candidate area, in step 106, the sum of the differences between the average brightness values (P1-P2) + (P3-P2) is stored in an image memory 32 different from the image memory in which the face image is stored. Write the value of).
The coordinates to be written are written to the coordinates of the center point of the rectangular area 2. At the same time, in step 108, the total of the histogram (P1-P2) + (P3-
The frequency for P2) is increased by one.

If it is determined in step 104 that the above condition is not satisfied, that is, if the difference between the average luminance values of adjacent rectangular areas is small, it is estimated that the area is a face skin area, and in step 110, 0 is written in an image memory 32 different from the image memory in which the face image is stored. The coordinates to be written are written to the coordinates of the center point of the rectangular area 2. At the same time, in step 112, the frequency at which the sum of the histogram memory 24 is 0 is increased by 1.

In the next step 114, it is determined whether or not the processing of steps 100 to 112 has been completed for the entire face image. If the processing has not been completed, the window is shifted by one pixel and the process returns to step 100. , And the above processing is repeated. Note that the direction in which the window is shifted may be any of the horizontal direction, the vertical direction, and the oblique direction.

The above processing is performed on the entire face image, which is the input image, while shifting the rectangular window by one pixel, so that the luminance value is converted to the sum of the differences between the luminance average values and the histogram. Is obtained.

If it is determined in step 114 that the processing has been completed for the entire face image, then in step 116, the sum of the differences between the average luminance values (P1-P2) + based on the histogram stored in the histogram memory.
A threshold value K2 for (P3-P2) is calculated. That is, the frequencies are integrated from the higher frequencies in the histogram, and the value of the sum (P1-P2) + (P3-P2) when the integrated value reaches a predetermined ratio with respect to all the frequencies is set to the threshold value K.
Set as 2. The predetermined ratio for the total frequency is set in advance by simulation or the like.

In step 118, the sum (P1-P2) of the differences between the average brightness values for each pixel in the image memory in which the sum (P1-P2) + (P3-P2) of the difference in the average brightness value has been written in step 106. ) + (P3-P2) is compared with the threshold value K2 to determine whether or not the following condition is satisfied.

(P1-P2) + (P3-P2)> K2 For a pixel satisfying the above condition, the value at the time of output in step 120 is set to 1, and for a pixel not satisfying the above condition, step 122 is performed. The value at the time of output is set to 0 and binarization is performed. The above-described binarization process is continued until it is determined in step 124 that the process has been completed for the entire image. Then, by performing processing on the entire image, a binary image is obtained.

FIG. 8 shows the processing result when the above processing is performed. FIG. 8A shows a processing result when the original image has the eyes open, and FIG. 8B shows a processing result when the original image has the eyes closed.
The eyes, which are the features of the face, are satisfactorily extracted from the binarized image, and the positions of the eyes can be specified from the binarized image. Note that the center image in FIGS. 8A and 8B is a processed image when the processes of steps 100 to 112 are performed on the entire image.

FIG. 9B shows a processing result when a binarized image is obtained when the face is not facing the front as shown in FIG. 9A, and the eyes are well extracted. FIG. 9C shows the processing result when the eyes are extracted when the face is not facing the front, and the eyes are well extracted.

Next, a second embodiment of the present invention will be described. In the first embodiment, when processing is performed on an image of a face in which eyes and eyebrows are approaching as shown in FIG. 10, when the eyebrow enters the rectangular area 1, the luminance average value of the rectangular area 1 May become small, and the output result of the binarized image may eventually become small. Therefore, in the second embodiment, the rectangular area 1 of the window 40 shown in FIG.
As shown in FIG. 1, two upper and lower rectangular areas 1A having the same area,
A window 4 consisting of four rectangular areas divided into two 1B
When the eyebrow falls within the rectangular area, the calculation is performed excluding the area where the eyebrow enters, thereby preventing the output result of the binarized image from being reduced.

Next, a face image extraction processing routine according to the second embodiment will be described with reference to FIG. In FIG. 12, the same parts as those in the processing routine of FIG.

The face image data is read from the image memory 20, and a window 44 is set on the input face image so that four rectangular areas are continuous in the vertical direction of the face image.

After calculating the integrated luminance values of all the pixels in each rectangular area in step 100 of FIG.
02, each of the integrated luminance values is divided by the total number of pixels in each rectangular area to obtain an average luminance value P in each rectangular area.
1A, P1B, P2, and P3 are calculated and stored in the average luminance value register 22. The rectangular areas 1A and 1B are rectangular areas 1
Is divided into two, so that (P1A) is different from the average luminance value P1 of the rectangular area 1 described in the first embodiment.
+ P1B) / 2 = P1.

In the next step 103, the rectangular areas 1A,
The difference (P1B-P1A) between the average luminance values within 1B is compared with a threshold value K3.

In step 103, (P1B-P1A) <K
3 is satisfied, that is, when the difference in the average luminance value between the adjacent rectangular regions is small, it is estimated that the region is the skin region of the face, and reflected in the calculation result as described below.
If (P1B-P1A) <K3 is not satisfied, it is assumed that the eyebrow has entered the rectangular area 1A, and the calculation is performed by excluding the data of the rectangular area 1A having the eyebrows as described below.

That is, if the determination in step 103 is affirmative, it is determined that the eyebrows are not within the rectangular area 1A, and step 1 described in FIG. 6 of the first embodiment is performed.
The same processing as in step 04 to step 108 is performed.

On the other hand, if the determination in step 103 is denied, it is determined that the eyebrow has entered the rectangular area 1A, and the average brightness value P1A in the rectangular area 1A is excluded so as not to be used in the calculation. Each of the differences (P1B-P2) and (P3-P2) between the average brightness values in the regions 1B, 2, and 3 is compared with the threshold value K4 to determine whether or not the following condition is satisfied. The threshold value K4 can be set to the same value as the threshold value K1 in step 104.

(P1B−P2> K4) and (P3−P2> K4) When it is determined in step 105 that the above condition is satisfied, that is, since the difference between the average luminance values of adjacent rectangular regions is large, If it is determined that the area is a candidate area, in step 107, the sum of the differences between the average brightness values is transferred to an image memory different from the image memory in which the face image is stored (P1B-P2) + (P3-P2). Write the value of The coordinates to be written are written to the coordinates of the center point of the rectangular area 2. At the same time, in step 109, the total sum of histograms (P1B-P2) + (P3-
The frequency for P2) is increased by one.

If it is determined in step 105 that the above condition is not satisfied, 0 is written to an image memory different from the image memory in which the face image is stored, as in the first embodiment. At the same time, in step 112, the frequency whose sum is 0 is increased by 1.

The above processing is repeated until it is determined in step 114 that the processing has been completed for the entire face image. If it is determined in step 114 that the processing has been completed, the histogram created in step 116 is generated. , The threshold value K2 for the sum of the differences between the average brightness values is calculated in the same manner as described above. In step 118, (P1-P2) + (P3-P2) in step 106 is
In step 107, (P1B-P2) + (P3-P
For each pixel in the image memory in which 2) is written, the value of each pixel (P1-P2) + (P3-P2) or (P1
(B−P2) + (P3−P2) is compared with the threshold value K2 to determine whether or not the following condition is satisfied. Note that, as described above, P1 = (P1A + P1B) / 2.

(P1-P2) + (P3-P2)> K2 or (P1B-P2) + (P3-P2)> K2 For the pixel satisfying the above condition, the value output at step 120 is set to 1 For pixels that do not satisfy the above condition, the value when output in step 122 is set to 0, and binarization is performed to obtain a binarized image as described in the first embodiment.

In each of the above embodiments, an example has been described in which the vertical size of the rectangular area 2 is substantially twice the vertical size of the rectangular area 1. However, the rectangular size may be varied according to individual differences (eye thickness). The vertical size of the area 2 may be changed.

In this case, a plurality of vertical sizes of the rectangular area 2 are set, and the output is calculated for each vertical size of each rectangular area 2 at the coordinates of the eye extracted first, and the highest output value is obtained. The vertical size of the calculated rectangular area 2 can be selected.

As described above, by selecting the optimum window size at the time of the first extraction, it is possible to stably extract the position of the eye corresponding to individual differences, particularly, the thickness of the eye.

In each of the above embodiments, the example of detecting the position of the eyes has been described. However, by changing the size of the window and applying the same method as described above, the face other than the eyes can be identified. The eyebrows, mouth, and the like, which are the characteristics of, can be detected.
Then, by using the positional relationship of these feature points for verification, highly reliable facial features can be extracted.

[0050]

As described above, according to the present invention,
Since a rectangular window is set taking into account only the vertical boundary, it is possible to accurately extract regions that are candidates for facial features such as eyes even in the case of an infrared image in which it is not possible to clearly distinguish between white and black eyes. Is obtained.

[Brief description of the drawings]

FIG. 1A is a diagram showing a conventional window,
(B) is a diagram showing a simulation result of eye position detection according to the related art, (C) is a diagram showing a window used in the simulation, and (D) is a diagram showing the relationship between the degree of separation and the luminance of each region. FIG.

FIG. 2 is a diagram for explaining a problem of the related art;
(A) is an original image, (B) is an output result of a separation filter,
(C) is a binarized image when the threshold value is the same as in FIG. 1,
(D) is a binarized image when the threshold value is reduced.

FIG. 3 is a diagram showing an infrared image.

FIG. 4 is a block diagram of the present embodiment.

FIG. 5 is a flowchart showing a face image extraction processing routine according to the first embodiment.

FIG. 6 illustrates a window configuration according to the first embodiment.

FIG. 7 is a diagram illustrating an average luminance value according to the first embodiment.

8A is a diagram illustrating a processing result according to the first embodiment when the eyes are open, and FIG. 8B is a diagram illustrating a processing result according to the first embodiment when the eyes are closed; FIG.

FIG. 9A is a diagram showing a processing target image, and FIG.
7A is a diagram illustrating a processing result obtained by processing the processing target image of (A) according to the first embodiment, and FIG. 7C is a diagram illustrating a processing result when an eye position is extracted.

FIG. 10 is a diagram showing an image of a face where eyes and eyebrows are approaching.

FIG. 11A is a diagram illustrating a window configuration according to the second embodiment, and FIG. 11B is a diagram illustrating a luminance average value according to the second embodiment;

FIG. 12 is a flowchart showing a face image extraction processing routine according to the second embodiment.

[Explanation of symbols]

 Reference Signs List 10 camera 12 analog-digital converter 20 image memory 22 average value register 24 histogram memory

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int. Cl. ⁷ , DB name) G06T 1/00 330 G06T 7/00 G06T 7/00 510

Claims (1)

(57) [Claims] 1. A luminance for calculating an average luminance value in each rectangular area in a window composed of at least three rectangular areas which are set adjacent to each other in the vertical direction of a face on an input face image. Average value calculating means; calculating means for calculating the difference between the luminance average values of adjacent rectangular areas and the sum of the differences between the luminance average values; and a threshold value for calculating the difference between the luminance average values and the sum of the differences between the luminance average values. An extracting means for extracting a pixel larger than a threshold value by comparing the value with a value;