JP2007323104A - Method, device and program for discriminating state of eye - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To increase the speed of face detection in a picture while keeping an erroneous detection rate low. <P>SOLUTION: The face picture FP is detected from an entire picture P, and an existence region ER in which eyes should exist is extracted from the face picture FP. Then, it is determined whether or not the opened eyes may exist in the extracted existence region ER. When it is determined the existence, it is determined that the eyes are opened, and when it is determined no existence, it is determined that the eyes are closed. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an eye state determination method, apparatus, and program for determining whether eyes of a face image are in an open state or a closed state.

  Currently, the photographer is advised or closed via the monitor of the digital camera so that the eye state can be detected from the image acquired by the digital camera and the image can be redone if the eye is closed. It has been proposed to synthesize an eye part with an open eye. At this time, as a method of detecting a face from an image or a method of detecting eyes from a face, a detection method using a boosting algorithm is used in addition to a detection method using a neural network (Patent Documents 1 and 2). reference).

This boosting algorithm is a learning method of a two-class classifier that forms a strong classifier by combining a plurality of weak classifiers (weak classifiers). The object is selected.
US Patent Application Publication No. 2002/102024 US Patent Application Publication No. 2005/13479

  However, the boosting algorithms as shown in Patent Documents 1 and 2 determine whether or not the object is an object, and it is assumed that the object is the object A or the object B. Not in. Specifically, when the boosting algorithm is used to determine whether the eyes are open or closed, a classification is made as to whether the images are open eyes or other images. In other words, images other than open eyes are classified into the same category of `` not open eyes '' even if they are of different types, such as closed eyes and wrinkles on the corners of the eyes. There is a problem that becomes difficult. Similarly, when it is determined whether or not the eyes are closed, there is a problem that it is difficult to determine the eyes that are open.

  Furthermore, the present invention is not limited to the boosting algorithm described above, and closed eyes have a substantially linear shape and have a problem that detection is difficult because the shape is similar to the wrinkles of the corners of the eyes.

  Accordingly, an object of the present invention is to provide an eye state determination method, apparatus, and program capable of accurately determining whether an eye is open or closed.

  According to the eye state determination method of the present invention, a face image included in the entire image is detected, an existing region where the eye should exist is extracted from the detected contour of the face image, and an open eye exists in the extracted existing region Whether the eye is open when an open eye is detected from within the existing area, and when the eye is not detected, it is determined that the eye is closed It is characterized by this.

  An eye state determination apparatus according to the present invention includes a face detection unit that detects a face image included in an entire image, a region extraction unit that extracts an existing region where an eye should exist from the face image detected by the face detection unit, Eye detection means for detecting whether or not there is an open eye in the existing area extracted by the area extraction means, and a state in which the eye is open when the open eye is detected in the existing area by the eye detection means And a state determining unit that determines that the eyes are closed when the eyes are not detected.

  According to the eye state determination program of the present invention, a face image included in the entire image is detected by a computer, an existing region where the eye should be present is extracted from the detected outline of the face image, and the eye is opened in the extracted existing region. Detects whether or not an eye exists, determines that the eye is open when an open eye is detected from within the existing area, and determines that the eye is closed when no eye is detected It is made to function like this.

  Here, as long as the region extracting unit extracts an existing region where eyes are expected to exist from the face image, regardless of the method, for example, the eyes are arranged based on the contour, size, etc. of the extracted face image. It is also possible to extract an existence region having a high probability of being present.

  The eye detection means may be any method as long as it detects whether or not there are open eyes. For example, the eye detection means may be one that detects an open eye using pattern matching, or includes a discriminator that performs final discrimination from a plurality of discrimination results by a plurality of weak discriminators. The discriminator may be one that has been learned by using an image of an open eye as a correct sample image and using an image of a closed eye as an incorrect sample image. At this time, it is preferable that the discriminator has been learned using an eye peripheral image in addition to the closed eye image as an incorrect sample image.

  According to the eye state determination method, apparatus, and program of the present invention, a face image included in the entire image is detected, an existing region where the eye should be present is extracted from the detected face image, and the eye is opened in the extracted existing region. It is determined whether or not there is an eye, and when an open eye is detected from within the existing area, it is determined that the eye is open, and when it is not detected, the eye is closed Focusing on the fact that the position of the eye in the face is known, and that the open eye has more structural features than the closed eye, it is easier to discriminate, and it is easier to discriminate within the eye presence area. If only eyes are detected, closed eyes are detected at the same time. As a result, it is possible to accurately determine whether the eyes are open or closed.

  In particular, the eye detection means includes a discriminator that performs final discrimination from a plurality of discrimination results obtained by a plurality of weak discriminators, and the discriminator uses an eye image that is opened as a correct sample image. If it is learned using a closed eye image as the correct sample image, it is possible to accurately determine the open eye by the discriminator, so the closed eye is also accurately determined, As a result, the eye state can be determined with high accuracy.

  Embodiments of an eye state determination device according to the present invention will be described below in detail with reference to the drawings. The configuration of the eye state determination apparatus 1 as shown in FIG. 1 is realized by executing an eye state determination program read into the auxiliary storage device on a computer (for example, a personal computer). Further, the eye state determination program is stored in an information storage medium such as a CD-ROM, or distributed via a network such as the Internet and installed in a computer.

  The eye state determination apparatus 1 includes a face detection unit 10 that detects a face image FP included in an entire image, a region extraction unit 15 that extracts an existing region where an eye should exist from the face image FP, and an open in the existing region. Eye detection means 20 for detecting whether or not there is an eye, and when the open eye is detected from within the region, it is determined that the eye is open, and when the eye is not detected, the eye is closed And a state discriminating means 30 for discriminating.

  The face detection means 10 has a function of discriminating a face from an entire image acquired by a digital camera or the like, for example, and detecting the discriminated face as a face image FP. The face detection unit 10 is a face in the partial image generation unit 11 that generates the partial image PP by scanning the sub window W over the entire image P and the plurality of partial images PP generated by the partial image generation unit 11. And a face discriminator 12 for detecting a partial image.

  Note that the entire image P input to the partial image generating unit 11 is preprocessed by the preprocessing unit 10a. As shown in FIGS. 2A to 2D, the preprocessing unit 10a has a function of generating a plurality of whole images P2, P3, and P4 having different resolutions by converting the whole image P into multiple resolutions as shown in FIGS. Have. Further, the preprocessing unit 10a normalizes the plurality of generated whole images P by suppressing the variation in contrast in the local region and aligning the contrast at a predetermined level in the whole region of the whole image P (hereinafter referred to as local normalization). It has a function to apply).

  As shown in FIG. 2A, the partial image generating unit 11 scans a sub window W having a set number of pixels (for example, 32 pixels × 32 pixels) in the entire image P, and an area surrounded by the sub window W Are cut out to generate a partial image PP having a set number of pixels. As shown in FIGS. 2B to 2D, the partial image generating unit 11 also generates a partial image PP when the sub window W is scanned on the generated low resolution image. Yes. As described above, by generating the partial image PP from the low-resolution image, even if the face or the face does not fit in the sub-window W in the entire image P, the partial image PP fits in the sub-window W on the low-resolution image. It is possible to perform detection reliably.

The face discriminator 12 of FIG. 1 has a function of performing binary discrimination of whether the partial image PP is a face or a non-face, and is learned by an Adaboost algorithm, and has a plurality of weaknesses. Discriminators CF _{1 to} CF _M (M: number of weak discriminators) are included. Each of the weak classifiers CF ₁ ~CF _M respectively extracts the feature x from the partial images PP, partial images PP by using the feature x is a function of performing determination of whether or not a face. The face classifier 22 is configured to perform the final determination of whether or not a face with a weak classifiers CF ₁ ~CF _M definitive determination result.

Specifically, each of the weak classifiers _CF 1 _{~CF M} extracts coordinates set P1a in the partial image PP as shown in FIG. 3, P1b, the luminance value or the like in P1c. Furthermore, the coordinate values P2a and P2b set in the low resolution image PP2 of the partial image PP, the luminance values at the set coordinate positions P3a and P3b in the low resolution image PP3, and the like are extracted. Thereafter, two of the seven coordinates P1a to P3b described above are combined as a pair, and the difference of the combined luminance is defined as a feature amount x. Are those each weak classifier _CF different feature amount for each 1 ~CF _M is used, for example, the weak classifiers CF ₁ The coordinate P1a, used as a feature quantity difference of brightness in P1c, weak classifier CF ₂ The coordinate The luminance difference between P2a and P2b is used as a feature amount.

Note that although the case where each of the weak classifiers CF ₁ ~CF _M extracts characteristic amounts x, respectively, in advance extracts a feature x described above for a plurality of partial images PP, each of the weak classifiers CF it may be input to the _{1 ~CF M.} Furthermore, although the case where the luminance value is used is illustrated, information such as contrast and edge may be used.

Each weak discriminator CF _{1 to} CF _M has a histogram as shown in FIG. 4, and outputs scores f ₁ (x) to f _M (x) corresponding to the value of the feature quantity x based on this histogram. To do. Further, each of the weak classifiers _CF 1 _{~CF M} have confidence values β ₁ ~β _M indicating the discrimination performance. Each of the weak classifiers _CF 1 _{~CF M} so as to calculate the determination score β _m · _{f m} (x) with a score _{f 1 (x) ~f M (} x) and reliability β ₁ ~β _M It has become. Then, it is determined whether or not the determination score β _m · f _m (x) itself of each weak discriminator CF _m is equal to or greater than a set threshold value Sref, and when it is equal to or greater than the set threshold value, it is determined that the face is a face. (Β _m · f _m (x) ≧ Sref).

Further, each of the weak classifiers _CF 1 _~CF face classifier 12 _M has a cascade structure, each of the weak classifiers _CF 1 _{~CF M} of all it is determined to be a face part image PP only a face image It outputs as FP. That is, only the partial image PP determined to be a face by the weak classifier CF _m is determined by the downstream weak classifier CF _{m + 1,} and the partial image PP determined to be a non-face by the weak classifier CF _m is downstream. No discrimination is performed by the weak discriminator CF _{m + 1} on the side. As a result, the amount of the partial image PP to be discriminated in the downstream weak discriminator can be reduced, so that the discrimination operation can be speeded up. Details of the discriminator having a cascade structure are disclosed in Shihong LAO et al., “High-speed omnidirectional face detection”, Image Recognition and Understanding Symposium (MIRU 2004), July 2004.

Incidentally, each of the weak classifiers _CF 1 _{~CF M} a determination score _S 1 to S _M output from each rather than determining whether a individual determination score threshold Sref above, the weak classifier CF _m in the case of performing the determination, the weak classifiers CF sum ^{_{Σ r = 1 m β r ·}} f r of determination score on the upstream side of the weak classifiers _{CF 1 ~CF m-1 m} is determined score threshold S1ref more may be performed discriminated by whether ^{_{(Σ r = 1 m β r}} · f r (x) ≧ S1ref). Thereby, since the determination which considered the determination score by an upstream weak discriminator can be performed, the determination precision can be improved.

  The area extracting means 15 in FIG. 1 extracts an existing area ER where the eyes should be present in the face image FP extracted by the face detecting means 10, and is generally a face outline (size of the face image FP). The region including the position where the eye exists is extracted as the existing region ER. Specifically, as shown in FIG. 5, when the face is headed up and the chin is down, a region that is a predetermined amount L1ref above the lower end of the face image FP, and a predetermined amount L10ref away from the left and right ends Is to identify. The predetermined amounts L1ref and L10ref are statistically determined in advance as a distance from the contour of the face of a general person to the eye position.

  The eye detection unit 20 in FIG. 1 detects whether or not an open eye exists in the existence region ER, and generates a plurality of peripheral images SCP from the region specified by the region extraction unit 15. The generating unit 21 includes an eye discriminator 22 that discriminates the eye states of the plurality of peripheral images SCP generated by the peripheral image generating unit 21. As shown in FIG. 6, the peripheral image generation unit 21 scans a window frame having a set number of pixels (for example, 32 pixels × 32 pixels) in the existence area ER, and displays an image surrounded by the window frame W10 as a peripheral image SCP. Is supposed to generate as

The eye discriminator 22 in FIG. 1 is learned by the Adaboost algorithm in the same manner as the face discriminator 12 described above, and has a function of discriminating whether or not the surrounding image SCP is an open eye. Yes. Specifically, the eye classifier 22 has a plurality of weak classifiers CF _{10 to} CF _N (N: the number of weak classifiers). Each of the weak classifiers CF ₁₀ ~CF _N extracts feature x from the peripheral images PP, respectively, the partial images PP by using the feature x is a function of performing determination of whether or not a face. The eyes discriminator 22 is configured to perform whether the final determination peripheral image SCP is eyes opened using weak classifiers CF ₁₀ ~CF _N definitive determination result. Here, since the eye discriminator 22 can discriminate the opened eye, the eye discriminator 22 uses the image of the opened eye as the correct answer sample, and the closed eye image as the incorrect answer sample. Eye peripheral images are used.

  The state discriminating means 30 discriminates the eye state based on the open eye discrimination result in the eye detection means 20, and the eye is open when the open eye is detected from within the existence region ER. It is determined that the state is present. On the other hand, when an open eye is not detected, it is determined that the eye is closed because there is no open eye in the existence region ER where the eye should exist.

  FIG. 7 is a flowchart showing a preferred embodiment of the eye state determination method of the present invention. An example of the eye state determination method will be described with reference to FIGS. First, after pre-processing is performed on the entire image P in the face detection means 10 (see step ST1, FIG. 2), a partial frame PP is generated by scanning the window frame on the entire image P (step ST2). . Thereafter, discrimination by the face discriminator 12 is performed on the partial image PP, and the face image FP is detected (step ST3).

  Next, the presence area ER where the eye should exist is extracted from the face image FP detected by the area extraction means 15 (see step ST4, FIG. 5). The peripheral image SCP is generated by scanning the window frame from the existence area ER extracted by the peripheral image generation means 21 (see step ST5, FIG. 6). Then, the eye discriminator 22 determines whether or not the peripheral image SCP is an open eye (step ST6). Thereafter, in the state determination unit 30, when there are open eyes in the plurality of peripheral images SCP generated from the existence region ER, it is determined that the eyes are open, and there are no open eyes in the plurality of peripheral images SCP. Sometimes it is determined that the eyes are closed (step ST7).

  In this way, by determining the eyes that are open in the existence region ER where the eyes should be present, it is possible to accurately determine the two states of whether the eyes are open or closed. First, when an open eye is compared with a closed eye, the open eye has many characteristic structures including a dark circular structure in a horizontally long bright (white) ellipse. On the other hand, closed eyes are mainly composed of a linear structure, have few characteristic structures, and are similar in structure to the wrinkles of the corners of the eyes that are captured as peripheral images of the eyes. Therefore, when comparing a discriminator trained with the open eye as the correct sample and a discriminator trained with the closed eye as the correct sample, the discriminator trained with the open eye as the correct sample is incorrect. The detection rate can be lowered. Thus, using the fact that the open eye has more structural features than the closed eye, it is possible to accurately determine the open eye.

  On the other hand, the region where the eyes are present in the face is known although there are individual differences, and the presence region ER where the eyes should be present can be detected from the detected face image FP. Using this, an existing area ER where eyes are present is extracted in advance from the detected contour of the face image FP, and an open eye is discriminated in the existing area ER. When there is no open eye in the presence region ER where the eye always exists, it means that the eye in the presence region ER is closed. In this way, by using the boosting algorithm that can only determine whether the eyes are open (whether they are objects or non-objects) by determining the eyes that are open in the existence region ER. It is possible to accurately discriminate between two states of eyes or closed eyes (whether the object is an object A or an object B).

  According to the above-described embodiment, focusing on the fact that the position of the eye in the face is known and that the open eye has more structural features than the closed eye, it is easier to discriminate within the eye presence region. If only easy open eyes are detected, closed eyes are detected at the same time. As a result, it is possible to accurately determine whether the eyes are open or closed.

In particular, the eye detection unit 20, which includes a discriminator 22 for final determination of a plurality of determination results by the plurality of weak classifiers CF ₁₀ ~CF _N, discriminator 22, opens as the correct sample images If an eye image is used and learning is performed using a closed eye image as an incorrect sample image, the discriminator 22 can accurately discriminate open eyes. As a result, the eye state can be determined with high accuracy.

  The embodiment of the present invention is not limited to the above embodiment. For example, in the above embodiment, the case of a front face is illustrated, but the present invention can also be applied to a side face or an oblique face. Further, although the case where the Adaboost algorithm is used for the face discriminator 12 (eye discriminator 22) is illustrated, the face or eye may be discriminated using a known neural network technique or the like. Even in this case, the eyes opened from the existence area ER are detected.

The block diagram which shows preferable embodiment of the state determination apparatus of the eye of this invention Schematic diagram showing how the sub-window is scanned in the partial image generating means of FIG. Schematic diagram showing how feature quantities are extracted from partial images by the weak classifier of FIG. The graph figure which shows an example of the histogram which the weak discriminator of FIG. 1 has The schematic diagram which shows a mode that a presence area is extracted from a face image in the area | region extraction means of FIG. The schematic diagram which shows a mode that a periphery image is produced | generated from an existing area in the periphery image generation means of FIG. The flowchart which shows preferable embodiment of the eye state determination method of this invention

Explanation of symbols

1 eye state discriminating apparatus 10 face detecting means 15 area extracting means 20 eye detecting means 21 peripheral image generating means 21 discriminator 22 eye discriminator 30 state discriminating means ER existence area P whole image FP face image

Claims (4)

Detect a face image from the whole image,
Extracting the presence area where the eyes should be present from the detected face image,
Detecting whether or not there is an open eye in the extracted existence region;
When the open eye is detected from within the existence area, it is determined that the eye is in an open state, and when it is not detected, the eye is determined to be in a closed state. How to determine. Face detection means for detecting a face image from the entire image;
An area extracting means for extracting an existing area where an eye should exist from the face image detected by the face detecting means;
Eye detection means for detecting whether or not an open eye exists in the existence area extracted by the area extraction means;
State determining means for determining that the eye is open when the opened eye is detected from within the existence area, and for determining that the eye is closed when the eye is not detected And an eye state discriminating device. The eye detection means includes a discriminator that performs final discrimination from a plurality of discrimination results by a plurality of weak discriminators, and the discriminator uses an image of an eye that is opened as a correct sample image, 3. The eye state discriminating apparatus according to claim 2, wherein learning is performed using an image of a closed eye as a correct sample image. On the computer,
Detect the face image included in the whole image,
Extracting the presence area where the eyes should be present from the detected face image,
Detecting whether or not there is an open eye in the extracted existence region;
An eye for causing the eyes to be determined to be in an open state when the open eye is detected from within the existence area, and to be determined to be in a closed state when the eye is not detected State determination program.

Images