JP4831344B2 - Eye position detection method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically detect positions of eyes from a face image highly reliably by a convenient method. <P>SOLUTION: A plurality of gray-scale images having their lightness sequentially varied are formed from a face image, and block areas of pixels are detected which sequentially appear in the faded-out face area as the gray-scale images fade in from a fade-out state of a high lightness to a low lightness, and block areas appearing as pairs out of the detected block areas of pixels are selected as position candidates of eyes, and the positions of eyes are specified on the basis of frequencies in appearance of respective position candidates of eyes in all the gray-scale images. <P>COPYRIGHT: (C)2007,JPO&amp;INPIT

Description

  The present invention relates to a method for automatically detecting an eye position from a face image.

  The position of the eye in the face image can be obtained by overlaying a hairstyle model image on the subject's face image to form a hairstyle simulation image, or by overlaying a makeup image such as lips or eyebrows on the subject's face image. In this case, the image is detected for adjusting the size of the image or aligning the image. Detection of the eye position is also performed in face image blindfolding processing, face image frame processing, creation of an image for personal authentication, and the like.

  Conventionally, as a method for detecting an eye position from a face image, a method of detecting a face by extracting a skin color region and further detecting an eye by pattern matching or the like has been used (Non-Patent Document 1, Patent Document 1). ).

2005 5th LSI IP Design Award Winning Paper “Face candidate point detection method that enables fast and reliable face detection” (Nikkei Business Publications) JP 2004-94917 A

  However, the extraction of the skin color area is affected by the lighting environment. Therefore, reliability becomes a problem when an unspecified number of face images taken in various lighting environments are targeted. Further, the pattern matching method has a problem that the amount of calculation becomes enormous.

  In contrast, an object of the present invention is to enable automatic detection of a position of an eye from a face image with a simple method with high reliability.

  When the present inventors create a plurality of grayscale images with successively changing brightness from the face image and observe the images that are faded in sequentially from the faded-out images on the high brightness side, If the appearing pixels belong to the pupil region, in this case, the pixels in the pupil region appear in pairs, and if the frequency of appearance over all grayscale images of the pixel regions that appear in pairs is accumulated, It was found that the eye position can be specified based on this, and that the eye position specifying method can be applied not only to still images but also to moving images.

That is, the present invention creates a plurality of grayscale images with lightness successively changing from a face image,
As the grayscale image fades in from a state that fades out at high brightness to a low brightness, it detects a cluster area of pixels that gradually appears in the faded-out face area,
Select the detected pixel cluster area that appeared as a pair as the eye position candidate,
Provided is an eye position detection method for specifying an eye position based on the frequency of appearance of all eye position candidates over all grayscale images. In this case, the face image is a frame of a still image or a moving image. I will provide a.

Further, the present invention provides a face image acquisition means, a function of creating a plurality of gray scale images whose brightness has been changed sequentially from the face image, and the gray scale image is faded in from a state where the gray scale image has faded out to a low brightness. Accordingly, a function for detecting a pixel cluster area that gradually appears in a faded-out face area, a function for selecting a pair of detected pixel cluster areas that appear as a pair, and a position candidate for each eye The eye position detection system includes a calculation unit having a function of specifying the position of the eye based on the appearance frequency over all the gray scale images, and in this case, the face image acquisition unit includes a still image or An aspect is provided in which a moving image is a camera capable of shooting.

  According to the method or system of the present invention, a plurality of grayscale images whose brightness changes sequentially from the face image are created, and the grayscale image fades in from a state where the grayscale image fades out to a low brightness. Since the pixel cluster area that gradually appears in the faded face area is detected as an eye position candidate, and the eye position is specified based on the appearance frequency over all grayscale images of the eye position candidate, the lighting environment, the skin color of the subject, Regardless of the color of the subject's eyes, the orientation of the face in the face image, etc., the eye position can be detected with high reliability and position accuracy, and pattern matching is not required. The position can be detected.

  Therefore, it is possible to form a hairstyle simulation image that fits various hairstyle images to an arbitrary face image using the eye position as a reference, or a partial facial image after makeup using an eye position as a reference It is possible to form a natural simulation image when a simulation image is formed by fitting a plurality of pieces of image information using the eye position as a reference, such as forming a makeup simulation image to be fitted to It becomes. In addition, the method or system of the present invention can also be suitably used when performing image processing such as mosaicing and filling with reference to the position of the eye, such as blindfolding processing for personal information protection. Furthermore, since these image processes can be performed at high speed, these image processes can be performed not only on still images but also on moving images.

  Hereinafter, the present invention will be described in detail with reference to the drawings.

  FIG. 2 is a block diagram of an eye position detection system for carrying out this method, and FIG. 1A is a flowchart showing an embodiment in which the eye position detection method of the present invention is performed on a still image. These are the flowcharts of one Example which performs the detection method of the eye position of this invention with a moving image.

  The system 10 includes an imaging apparatus 1 and a personal computer main body 2, and a display 3, an image scanner 4, a printer 5, and the like are connected to the personal computer main body 2.

  The imaging device 1 is provided as means for acquiring a test subject's face image, and a commercially available still image or moving image camera such as a digital still camera, digital video camera, or web camera can be used. When a moving image is to be detected as an eye position, a video capture is connected between the imaging device 1 and the personal computer body 2 as necessary.

  The personal computer main body 2 has an image processing function for creating a plurality of gray scale images whose brightness has been changed successively (for example, one in which pixel values are changed by 2 to 3 gradations in a 256 gradation gray scale image layer). ing.

Gray scale image creation methods include (1) a method using the average of the maximum and minimum values of R, G, and B for each pixel, and (2) the average of each value of R, G, and B. (3) There is a method in which each value of R, G, B is multiplied by a predetermined weighting coefficient (NTSC, etc.) and then averaged, etc. In the present invention, In the method, for example, the following equation Y (output luminance) = 0.298912 × R + 0.568811 × G + 0.114478 × B
It is preferable to change the gradation by 2 to 3 gradations using Such a gray scale image processing function can be obtained, for example, by installing in the personal computer main body 2 commercially available image processing software such as Adobe System Photoshop.

  Further, the personal computer main body 2 has a function of changing the contrast in accordance with the change in brightness (for example, a function of decreasing the contrast as the brightness is increased and increasing the contrast as the brightness is decreased), and the grayscale image hierarchy. A function that assigns list numbers in order of low or high brightness, and in the grayscale image hierarchy, pixels that gradually appear in the faded-out face area as the brightness fades from low fade to high brightness A function for detecting the cluster area of the detected pixel, a function for selecting the detected pixel cluster area as a pair of eye positions, and the frequency of appearance of each eye position candidate over the grayscale image of all layers. It has a function of counting and specifying the position of the eye based on the value.

  Further, the personal computer main body 2 has a function for adjusting the size of the face image, a function for setting a detection target region of the eye position in the face image, a blurring function for the face image, and a vertical and horizontal size of the cluster area of pixels to be detected. A function for preliminarily setting the eye position within a predetermined range, a function for storing a positional relationship between eye position candidates, a width of the eye position candidate, and the like as eye position conditions when specifying a specific eye position from the eye position candidates The eye position candidate satisfying the eye position condition is selected and specified as the eye position.

  In addition, when a moving image is an eye position detection target, the personal computer main body 2 has a function of selecting a frame to be transferred from the imaging apparatus 1 at a predetermined frame rate as necessary. When specified, it has a function of storing the position and setting the vicinity of the position as a detection target area of the eye position in the next frame.

  In the eye position detection method in the still image using this system 10, as shown in FIG. 1A, first, a face image as an eye position detection target is acquired in the personal computer main body 2. The face image may be obtained, for example, by taking a face image of the subject with the imaging device 1 and importing the face image into the personal computer main body 2 or reading the face photograph of the subject with the image scanner 4, such as the Internet. You may acquire using a communication line.

  Next, as necessary, the number of pixels of the face image is adjusted to a size suitable for the eye position detection process. More specifically, if the number of pixels in the face image is too large, an excessive burden is placed on the detection process, and if it is too small, the detection accuracy of the eye position decreases, so the image size is reduced to about 480 × 360 to 320 × 240. change.

  Further, if the face image is sharp, many fine edges appear in subsequent image processing, and noise for detecting eye position candidates is generated. Therefore, the face image is subjected to blurring processing as necessary. In this case, it is preferable that the degree of blurring is weak to medium. For example, for each pixel, the weighting blur filter of 1 to 5 is used for the pixel of interest and the surrounding 25 pixels. Performs blurring to output the average value. This blurring process may be performed a plurality of times as necessary.

  Furthermore, in the eye position detection method of the present invention, since color information is not necessary for the face image, if the acquired image is a color image, it is converted to a grayscale image in order to reduce subsequent processing amount. To do.

  Next, in the acquired face image, it is preferable to set a detection target region for the eye position in advance. For example, as shown in FIG. 3, a rectangular frame 21 is set as the eye position detection target region for the grayscale face image 20. As a method for setting the rectangular frame 21, regarding the face image 20, assuming that 1/8 of the length of the vertical side is A, the area 3 </ b> A and 4 </ b> A is the center O of the area, and the face image 20 The center coordinates of the face image 20 may be set at a position of 3.5 A from the upper side of the face image 20 on the center line of the width of the face image 20. Thus, the method of setting the rectangular frame 21 as the eye position detection target region can be applied to any image in which the upper body is shown.

  Next, as shown in FIG. 4, a plurality of grayscale images whose brightness has been sequentially changed are formed. More specifically, for example, in a grayscale image having 256 gradations, a fade-in image having about 30 to 100 layers in which the brightness is gradually decreased is created from an image having a pixel value of 0 that is completely faded out. The eye part always appears in the 30 to 100 layer image. In this case, it is preferable to lower the contrast as the brightness is increased and to increase the contrast as the brightness is decreased, because the eye part appears more clearly in the fade-in image.

  As can be seen from FIG. 4, when the brightness of a grayscale image is increased and the faded-in image layer is formed by successively reducing the brightness from an image that has been completely faded out, normally, the cluster of pixels is usually first in the pupil. Is detected, and then a block of pixels is detected in the nose and mouth. Further, since the cluster of pixels in the pupil region appears as a pair of left and right, the group of pixels of the pair of left and right is selected. Depending on the hierarchy, pixel cluster areas appear in pairs in areas such as the mouth, eyebrows, and forehead in addition to the pupils. At this stage, these pixel cluster areas are also selected as eye position candidates.

  In the detection of the cluster area of pixels, it is preferable to reverse the gradation of the grayscale image whose brightness has been sequentially changed as shown in FIG. 5 in order to facilitate image processing.

  In addition, as a condition for the cluster of pixels to be detected as eye position candidates, when 1/8 of the length of the vertical side of the grayscale face image 20 is A as described above, the size of the pixel cluster area is as follows. Is set within the range of 0.4A to 0.7A, preferably 0.5A to 0.7A, 0.5A to 1.0A, preferably 0.7 to 1.0A. Usually, the size of the cluster area of the pixel at the eye position has a size within this range. By setting such a size condition, the size of the subsequent area required for detection of the eye position is set. The amount of processing can be reduced.

  FIG. 6 shows a block area of pixels appearing in a face area in a fade-in image of a certain hierarchy as a rectangle, and a pair of these areas connected by a straight line.

It is preferable that the following (1) to (4) are set in advance as conditions for selecting a candidate eye position from among the clustered pixel regions.
(1) The lateral distance of the cluster area of a pair of left and right pixels is within a certain range.
(2) The vertical distance between the left and right pixel cluster areas is within a certain range.
(3) Of the paired left and right pixel cluster regions, the left pixel cluster region is in the left half region of the rectangular frame 21 of the face image.
(4) Of the paired left and right pixel cluster regions, the right pixel cluster region is in the right half region of the rectangular frame 21 of the face image.

  Next, as shown in FIG. 7, a pair of eye position candidates appearing for each fade-in image is accumulated over all layers of the grayscale image, the frequency of appearance is counted, and ranking is performed in descending order of the count number. Create a list. In this count, the eye position candidate that appears first when the brightness is gradually reduced from the faded-out face image and continues to appear until the final stage is normally the maximum count (ranking first). Therefore, in principle, the position of the eye position candidate with the maximum count is specified as the eye position.

However, if the following conditions (a) to (c) are all satisfied, or if the condition (d) is satisfied, the position of the eye position candidate with the second largest count is set as the eye position. As specified.
(a) When the second position candidate for the ranking is higher than the first position candidate for the ranking
(b) When the distance between the center of the eye position candidate of the 2nd ranked eye is longer than the distance between the centers of the position candidate of the 1st ranked eye
(c) When the left and right pupil regions of the second ranking eye position candidate are both outside the positions corresponding to the left and right pupils of the first ranking eye candidate
(d) The vertical distance between the first position candidate of the ranking and the second position candidate of the ranking is about the distance between the eyes and the eyebrows, and the first position candidate of the ranking is ranked first. When the position is above the second position candidate

  Of these, (a) to (c) are intended to prevent erroneous determination of the position of the mouth as the position of the eye, while the first ranking in the ranking may rarely be the mouth area. d) is for preventing erroneous determination that the eyebrows around the eyes are the positions of the eyes.

  As described above, as shown in FIG. 8, in the face image (original image before image processing), the eye position (more precisely, the position of the pupil) can be accurately detected.

  On the other hand, in the eye position detection method in the moving image using this system 10, as shown in FIG. 1B, first, the frame of the moving image transferred from the imaging device 1 is acquired as a face image, and the eye in this frame is acquired. The position of is detected. The eye position detection method itself is the same as the eye position detection in the still image, the formation of a hierarchy of fade-in images in which the brightness is sequentially changed in the grayscale image, the detection of the cluster of pixels appearing in the fade-in image, Selection of eye position candidates (a clustered area of pixels appearing in a pair on the left and right), accumulation of eye position candidates across all hierarchies of the grayscale image and counting their appearance frequency, eye position identification by maximum count, eye A specific correction is performed based on the position candidate position and the distance between the eyes and the center of the eyes.

  However, in the initial frame transferred from the imaging device 1, the detection target area of the eye position is preferably not limited to a specific area but the entire image.

  Next, the suitability of the specified eye position is determined. As this determination method, for example, when a 100-level fade-in image is created between pixel values 0 to 255, it is determined that the eye is not visible when the maximum count number is 5 or less. When determining the suitability of the eye position by this method, the number of layers of the fade-in image and the maximum count number for judging the suitability of the eye may be determined to be proportional. Therefore, when the fade-in image created between pixel values 0 and 255 is 200 layers, it is determined that the maximum count number is 10 or less, which is not noticeable.

  When it is determined that the eye position is properly detected by determining whether the eye position is appropriate, the eye position in this frame is registered.

  Then, a frame (hereinafter referred to as “frame N”) transferred from the imaging device 1 is acquired, and the eye position is detected in the same manner as described above, using only the registered eye position periphery as the eye position detection target area. To do. If it is determined that the position of the eye has been properly detected in this frame N, the position is registered. In the next frame N + 1, the vicinity of the most recently registered eye position is set as the detection target area. By limiting the detection target area of the eye position in this way, the processing speed can be increased.

  Here, when the eye position is properly detected, the setting around the eye position as the eye position detection target area in the next frame is, for example, a rectangle centered on the middle point between the eyes. The lateral direction is 1.5 to 3 times, preferably 1.8 to 2.2 times the distance between the eyes and the center of the eye, and the longitudinal direction is 0.5 to 2 times, preferably the distance between the eyes and the center of the eyes. Is in the range of 0.8 to 1.2 times.

  On the other hand, if it is determined that the specified eye position is inappropriate in the initial frame, the eye position is detected in the next frame without further detecting the eye position in this frame. .

  If it is determined that the eye position detected in the frame N is inappropriate, the position of the eye is detected using the entire image of the frame N as a detection target region. If it is determined that the position is properly detected, this is registered, and in the next frame N + 1, the position of the eye is detected using the vicinity of the position of the most recently registered eye as the detection target area. On the other hand, if the position of the eyes is not properly detected in the frame N even if the entire image is the detection target area, the next frame N + is detected in this frame N without further detecting the position of the eyes. Proceed to detect eye position at 1.

  Note that the reason why the eye position is not properly detected is that (i) the eye is not present in the image, (ii) the eye is present but the eye is closed, and (iii) the eye position. However, in any case, if the eye position is not properly detected in one frame, the eye position is detected in the next frame. To do.

  Further, when the detection of the eye position cannot catch up with the transfer of the frame due to the size of the image and the frame rate, the eye position is detected every predetermined number of frames.

  For the purpose of detecting the contents of the image and the eye position in the frame where the eye position is not properly detected or in the frame where the eye position is not detected to detect the eye position every predetermined number of frames. Accordingly, the eye position may be specified by complementing the eye position in the frame in which the eye position is detected, and the frame in which the eye position is not detected is not displayed on the display. Also good.

  The eye position detection method according to the present invention is an accurate and simple calculation regardless of changes in the skin color, pupil color, face orientation, and illumination environment of a face image subject to eye position detection. In this way, the eye position can be detected at a high processing speed. Therefore, the position of the eye can be detected from the face image regardless of whether it is a white person or a black person in both a still image and a moving image.

  The eye position thus detected is the formation of a hairstyle simulation image for fitting various hairstyle images to an arbitrary face image using the eye position as a reference, and the face part after makeup using the eye position as a reference Used in various scenes where eye position detection is required, such as forming makeup simulation images that fit images to arbitrary face images, face image blindfolding, face image frame processing, and personal authentication image formation can do.

  The present invention is useful in various scenes where detection of the eye position is required as described above.

It is a flowchart of the detection method of the eye position in a still image. It is a flowchart of the detection method of the position of the eye in a moving image. It is a block diagram of an eye position detection system. It is explanatory drawing of the detection target area | region of the position of eyes. It is the gray scale image which changed lightness sequentially. It is a grayscale inverted image of a gray scale image in which the brightness is sequentially changed. It is explanatory drawing of the cluster area | region of a pixel in a certain fade-in image. It is explanatory drawing of the state which accumulated the position candidate of eyes over all the hierarchy of a gray scale image. It is the original image which displayed the position of the detected eye.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image pick-up device 2 Personal computer body 3 Display 4 Image scanner 5 Printer 10 Eye position detection system 20 Gray scale face image 21 Rectangular frame as eye position detection target area

Claims (8)

Create a multi-level grayscale image in which the brightness faded out at a high brightness from the gray scale image faded out at a high brightness from the blurred face image,
In the grayscale image of the plurality of layers, a cluster area of pixels gradually appearing in the face area is detected as the brightness gradually changes from a grayscale image faded out at a high brightness to a grayscale image having a low brightness.
Select the detected pixel cluster area that appeared as a pair as the eye position candidate,
The frequency of appearance of all eye position candidates over all grayscale images is accumulated, counted, and ranked, and the position of the eye position candidate with the maximum count is set as the eye position, but the following (a) to ( d) Condition
(a) When the second position candidate for the ranking is higher than the first position candidate for the ranking
(b) When the distance between the center of the eye position candidate of the 2nd ranked eye is longer than the distance between the centers of the position candidate of the 1st ranked eye
(c) When the left and right pupil regions of the second ranking eye position candidate are both outside the positions corresponding to the left and right pupils of the first ranking eye candidate
(d) The vertical distance between the first position candidate of the ranking and the second position candidate of the ranking is about the distance between the eyes and the eyebrows, and the first position candidate of the ranking is ranked first. Among the cases above the second position candidate, if all of the conditions (a) to (c) are satisfied, or if the condition (d) is satisfied, the count number is second. A method for detecting an eye position that identifies the positions of eye position candidates that have been many as eye positions.   The eye position detection method according to claim 1, wherein the eye position is specified based on a positional relationship between eye position candidates or a lateral width of the eye position candidates.   The eye position detection method according to claim 1, wherein the gray scale image is inverted to detect a cluster area of pixels.   The eye position detection method according to claim 1, wherein the face image is a frame of a still image or a moving image.   5. The eye position according to claim 4, wherein when an eye position is specified in one frame of a moving image, an area around the eye position specified most recently in a subsequent frame is set as an eye position detection target area. Detection method. An eye position detection system comprising a face image acquisition means and a calculation means,
The computing means has a blurring function of a face image, a function of creating a grayscale image of a plurality of layers by gradually reducing the brightness faded out at a high brightness from a grayscale image faded out at a high brightness from the face image, the plurality A function that detects the cluster area of pixels that gradually appear in the face area as the brightness gradually changes from a gray scale image that fades out at a high brightness level to a low-scale gray scale image in the gray scale image of the hierarchy, and the detected pixel cluster area A function to select the eye position candidate that appears as a pair, and the frequency of appearance of each eye position candidate over the grayscale image of all layers is accumulated, counted, and ranked to identify the eye position With the ability to
The function of specifying the eye position is the position of the eye position candidate with the maximum count as the eye position, provided that the following conditions (a) to (d)
(a) When the second position candidate for the ranking is higher than the first position candidate for the ranking
(b) When the distance between the center of the eye position candidate of the 2nd ranked eye is longer than the distance between the centers of the position candidate of the 1st ranked eye
(c) When the left and right pupil regions of the second ranking eye position candidate are both outside the positions corresponding to the left and right pupils of the first ranking eye candidate
(d) The vertical distance between the first position candidate of the ranking and the second position candidate of the ranking is about the distance between the eyes and the eyebrows, and the first position candidate of the ranking is ranked first. Among the cases above the second position candidate, if all of the conditions (a) to (c) are satisfied, or if the condition (d) is satisfied, the count number is second. An eye position detection system that identifies the position of a large number of eye position candidates as the eye position.   The eye position detection system according to claim 6, wherein the calculation unit has a function of specifying an eye position based on a positional relationship between eye position candidates or a lateral width of the eye position candidate.   The eye position detection system according to claim 6 or 7, wherein the face image acquisition means is a camera capable of capturing a still image or a moving image.