JP2008226073A - Eye detection apparatus, eye detection method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an eye detection apparatus, eye detection method and program which prevents an incorrect region from being searched for eye location. <P>SOLUTION: A camera 10 images the face of a subject, and a CPU 24 of a computer 14 obtains the moving image and stores it in an image memory 22. The CPU 24 processes the moving image stored in the memory 22, searches for the nostrils of the subject on the basis of a histogram value and its peak, determines a region for searching the upper eyelid and lower eyelid after the nostrils are found, and extracts edge lines, as a candidate of the combination of the upper eyelid and the lower eyelid of the subject, from the region. The CPU 24 discriminates, among the extracted edge lines, that the edge lines of an upper eyelid candidate whose center moves in the vertical direction and a lower eyelid candidate, whose center does not move are the eyelids of the subject. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an eye detection device, an eye detection method, and a program for detecting eyes of a driver or the like driving a moving body such as a vehicle.

A technique for detecting the position of a person's face and eyes from an image including the person's face is known. For example, in Patent Document 1, pixel density is detected along a vertical pixel row of an input human face image, and a point is extracted based on the density value of the pixel row and a change value of the density value. A dozing state detection device that captures an eyelid region as continuous data of extracted points is disclosed.
Japanese Patent Laid-Open No. 10-275212

  However, since the dozing state detection device of Patent Document 1 examines the position of the eye based on the density value of the pixel and the change value of the density value, the pixel density change in the eyebrows or the nose is erroneously detected as the eye position. There are things to do. In addition, there are misjudgment factors when searching for the eye position search area, such as the eyebrows disappearing depending on the hairstyle, or the presence of factors such as glasses. Because of these misjudgment factors, the dozing state detection device of Patent Document 1 may make a mistake in the search area for the eye position.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an eye detection device, an eye detection method, and a program that do not mistake the search region for the eye position.

In order to solve the above problems, an eye detection device according to the first aspect of the present invention provides:
Face image storage means for storing an image of the face of the subject,
Based on the face image stored by the face image storage means, a current position candidate detection means for obtaining a candidate for a position under the eyes in the face image;
When the face nostril is searched within a predetermined nostril search area on the face image stored in the face image storage means, and the nostril can be detected, the position under the eyes based on the detected nostril position Current position selection means for selecting a candidate located under the eyes of the face image from among the candidates,
Eye detection means for detecting an image corresponding to the eyes of the subject in an eye search area determined based on the current position selected by the current position selection means;
It is comprised from these.

The current position candidate detection means includes:
Horizontal sobel means for processing a face image stored in the face image storage means with a horizontal Sobel filter to generate a horizontal Sobel image;
Means for generating a histogram of gradation values of pixels on each horizontal line of the horizontal Sobel image generated by the horizontal Sobel means;
Current position candidate output means for obtaining a candidate for a horizontal line passing under the eyes based on a histogram of gradation values and outputting the position of the horizontal line as a candidate for the current position;
It is good also as comprising.

The current position candidate output means includes:
Based on the histogram of gradation values, a predetermined number of high histogram peak values are extracted, and the position where the extracted histogram is obtained is the current position candidate.
This may be a feature.

The current position selection means sets the nostril search region based on one of the candidates for the position under the eyes detected by the current position candidate detection means.
This may be a feature.

A face center line detecting means for detecting a center line of the face image stored in the face image storing means;
The current position selection means includes
Search means for searching for nostril candidates of the face in the nostril search region;
Nostril position determining means for determining the position of the nostril candidate of the face detected by the search means;
Based on the position of each remark candidate determined by the nostril position determining means and the center line, the nostrils that are substantially line-symmetric with respect to the center line of the face among the nostril candidates searched by the search means Means for detecting candidate pairs and distinguishing them from nostrils;
May be further provided.

When the nostril can be detected, the current position selection means is positioned above the nostril and outputs the current position candidate closest to the nostril as a position below the eye.
This may be a feature.

In order to solve the above problem, an eye detection method according to a second aspect of the present invention includes:
Process the stored face image to obtain a candidate for a horizontal line located under the eye in the face image,
Search the nostril of the face within a predetermined nostril search area on the face image,
When the nostril can be detected, the position of the detected nostril is selected as a reference, and the candidate located below the eyes of the face image is selected from the candidates for the horizontal line,
Set an eye search area on the face image based on the selected horizontal line,
In the set eye search area, an image corresponding to the eye of the subject is detected.
It is characterized by that.

In order to solve the above problem, a program according to the third aspect of the present invention is:
Computer
Face image storage means for storing an image of the face of the subject,
Based on the face image stored by the face image storage means, a current position candidate detection means for obtaining a candidate for a position under the eyes in the face image;
When the face nostril is searched within a predetermined nostril search area on the face image stored in the face image storage means, and the nostril can be detected, the position under the eyes based on the detected nostril position Current position selection means for selecting a position located under the eyes of the face image from among the candidates of
Eye detection means for detecting an image corresponding to the eyes of the subject within an eye search area determined based on the current position selected by the current position selection means;
To function as.

  According to the present invention, it is possible to provide an eye detection device, an eye detection method, and a program that do not mistake the search region for the eye position.

  Hereinafter, an eye detection device 50 according to an embodiment of the present invention will be described with reference to the drawings.

First, the configuration of the eye detection device 50 according to the present embodiment will be described with reference to FIGS.
As shown in FIG. 1, the eye detection device 50 includes a camera 10 that captures a driver's face and generates a face image, an illumination light source 12 that illuminates the driver's face, and a computer 14 that extracts feature points from the driver's face image. And a display device 16 connected to the computer 14.

  The camera 10 is composed of a CCD (Charge Coupled Devices) camera, for example, and acquires a gradation image of the driver's face. The face image generated by the camera 10 includes not only the driver's face but also its background.

  The display device 16 includes an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube), and displays a face image captured by the camera 10.

The computer 14 processes the face image acquired by the camera 10 and detects wrinkles.
As shown in FIG. 2, the computer 14 includes an A / D converter 21, an image memory 22, a ROM (Read Only Memory) 23, a CPU (Central Processing Unit) 24, a RAM (Random Access Memory) 25, and a display control device 26. A light source control device 27, a setting memory 28, and an operation device 29.

  An A / D (analog / digital) converter 21 converts an analog image signal photographed by the camera 10 into a digital signal.

  The image memory 22 stores image data generated by the camera 10 and digitized by the A / D converter 21.

  The ROM 23 stores a program for controlling the operation of the CPU. Further, the ROM 23 stores various fixed data for executing image processing to be described later.

  The CPU 24 controls the entire computer 14. Further, the CPU 24 executes a program stored in the ROM 23 to process the face moving image acquired by the camera 10 and detect an eye or eyelid in the face moving image.

  The RAM 25 functions as a work area for the CPU 24.

  The display control device 26 converts video data or the like into a data format that can be output by the display device 16 under the control of the CPU 24, and outputs the data to the display device 16.

  The light source control device 27 controls turning on / off of the illumination light source 12.

  The setting memory 28 stores information (hereinafter referred to as setting information) relating to the setting of processing in which the CPU 24 processes the face moving image to detect eyes or eyelids.

  The operation device 29 receives operation information from the user and sends an operation signal corresponding to the operation to the CPU 24.

Next, an example of fixed data stored in the ROM 23 will be described with reference to FIG.
The ROM 23 stores operators of vertical edge detection and horizontal edge detection Sobel filters as shown in FIGS. The vertical edge detection Sobel filter and the horizontal edge detection Sobel filter are used to emphasize the vertical gray level difference and horizontal gray level difference as shown in FIGS. An operator.

  Hereinafter, referring to FIG. 4 to FIG. 19, the eye detection device 50 according to the present embodiment having the above configuration extracts an upper and lower eyelid as a facial feature point from the driver's face image acquired by the camera 10. explain.

First, the outline | summary of the operation | movement which the eye detection apparatus 50 of this embodiment extracts an upper and lower eyelid is demonstrated.
In the present embodiment, edge lines that are candidates for upper and lower eyelids are extracted as facial feature points from a face moving image of a driver driving a vehicle.

Here, before extracting the edge line, the face image is processed so that the change rate of the gradation of the face image is extracted by the Sobel filter shown in FIGS.
In the following description, the face image that has been subjected to the process of enhancing the horizontal edge by the horizontal edge detection Sobel filter in FIG. 3A is referred to as a “horizontal Sobel image” and is used for vertical edge detection in FIG. A face image that has been subjected to processing for enhancing a vertical edge by a Sobel filter is referred to as a “vertical Sobel image”. Further, the horizontal and vertical sobel images are collectively referred to as a “sobel image”.

  Next, the face center line, which is a vertical line passing through the center of the face, is obtained based on the face image, and the face orientation is the nose based on the coordinates of the face center line and the coordinates of the left and right edges of the face. It is determined whether or not both holes are detectable. If the orientation of the face is not such that both nostrils of the nose can be detected, the process returns to the preprocessing and the processing is repeated.

  Next, based on the horizontal Sobel image, a line (current candidate line) that is a candidate for a line that passes directly under the eye (瞼) (a position immediately under the eye (瞼)) is extracted, and a nostril is located near the current candidate line. Whether or not can be detected. If the nostril can be detected, the current candidate line immediately above the nostril (the closest candidate line located above the nostril to the nostril) is extracted as a line (current line) that passes directly under the eye (瞼). To do.

Next, edge lines are extracted from the region including the vicinity of the current line of the horizontal Sobel image, and among the edge lines that are candidates for the upper and lower eyelids, the edge lines that satisfy a predetermined condition are formed into a set of upper and lower eyelids. Extract as a candidate.
瞼 Opening time series that correlates the heel opening indicating the distance between the center of gravity of the upper heel and the lower heel, and the time when the heel opening was obtained, for each edge line that is a candidate for the upper and lower heels Information is generated, and edge lines whose heel opening time-series information matches a predetermined condition are extracted as correct upper and lower eyelids.

Hereinafter, the operation in which the eye detection device 50 of the present embodiment extracts the upper and lower eyelids will be described in detail.
When the power is turned on, the CPU 24 in the computer 14 repeatedly executes the wrinkle detection process shown in FIG.
That is, the CPU 24 performs pre-processing (step S01), face position / current candidate determination process (step S02), face orientation determination process (step S03), current candidate search process (step S04), nostril detection process (step S05), The wrinkle detection process including the upper / lower wrinkle determination process (step S06) is repeatedly executed.

  The preprocessing (step S01) and face position determination processing (step S02) shown in FIG. 4 are processes for obtaining the position (coordinates) of the face in the left and right direction and the up and down direction of the driver.

The face orientation determination process (step S03) is a process for determining the left and right orientations of the driver's face.
The current candidate search process (step S04) is a process of searching for a line (current candidate line) that is a candidate for a line that passes directly under the driver's eye (eyelid).
The nostril detection process (step S05) is a process for detecting an image corresponding to the driver's nostril near the current candidate line.

  The upper / lower eyelid determination process (step S06) sets an upper face area A indicating a search range for searching for upper and lower eyelids based on the position of the driver's nostril, and corresponds to the upper eyelid from the Sobel image in the upper face area A. Edge line candidates and edge line candidates corresponding to the lower eyelid are extracted, and the eyelid opening is determined from the manner in which the center of gravity of the upper and lower eyelids moves within the edge lines that are candidates for the upper and lower eyelids. This is a process for calculating and determining that the edge line of the upper and lower eyelid candidates satisfying a predetermined condition for the eyelid opening is a genuine eyelid.

  Hereinafter, the operation executed in each step from Step S01 to S06 will be described in detail.

First, the preprocessing (step S01) will be described.
As shown in FIG. 5, the preprocessing (step S01) includes a capture process (step S011), a coordinate conversion process (step S012), a horizontal sobel filter process (step S013), and a vertical sobel filter process (step S014).

  The capture process (step S011) is a process in which the CPU 24 captures a face image for one frame (1/30 second) of the driver photographed by the camera 10 via the A / D converter 21 and stores it in the image memory 22. .

  The coordinate conversion process (step S012) is a process of thinning out pixels to the extent that the CPU 24 can process.

In the horizontal Sobel filter process (step S013), the CPU 24 processes the face image after coordinate conversion using the horizontal edge detection Sobel filter (FIG. 3A) stored in the ROM 23, and the inside of the face image is processed. This is a process of storing in the RAM 25 what has been subjected to the process of emphasizing the horizontal edges.
In the vertical sobel filter processing (step S014), the CPU 24 processes the face image after the coordinate conversion using the vertical edge detection sobel filter (FIG. 3B) stored in the ROM 23, and the inside of the face image is processed. This is a process of storing in the RAM 25 what has been subjected to the process of enhancing the vertical edges.

The processing operation of the Sobel filter will be described more specifically.
As a premise, each pixel constituting the face image is set so that each RGB color has an 8-bit gradation, and any one of 256 gradations from the 0th gradation to the 255th gradation is provided. It shall have. The 0th gradation is black and the 255th gradation is white.

The CPU 24 extracts one pixel from the face image acquired by the camera 10. The extracted pixel is made to correspond to the center of the 3 × 3 matrix in FIG. 3A or 3B, and the gradation of the extracted pixel and each gradation of all the pixels adjacent to the pixel are changed to FIG. The values corresponding to the operators shown in (a) or (b) are respectively multiplied.
The CPU 24 adds up all the values obtained by multiplication and sets the total value to the gradation of the extracted pixel. Hereinafter, the value set as the gradation of each pixel after the operation by the operator in FIG. 3A or FIG. 3B is referred to as “gradation value”.
By repeating these operations in order on the entire face image, a Sobel image indicating the rate of change in gradation of the face image (the horizontal or vertical differentiation of the gradation) is extracted.

  Next, the face end position determination process (step S02) will be described.

  The face both-end position discrimination process (step S02) is a process for identifying lines constituting both ends of the face in the vertical Sobel image obtained by the CPU 24 operating the face image with the vertical edge detection operator of FIG. Any known method can be employed.

  For example, as shown in FIG. 6, the CPU 24 performs gradation value histogram creation processing for detecting both ends of the face based on the vertical Sobel image stored in the RAM 25, and stores the created histogram in the RAM 25 (see FIG. 6). Step S021). Subsequently, the CPU 24 extracts a predetermined number of high histogram peak values, sorts these high peak values (step S022), and extracts end points based on the histogram values (step S023). For example, if the top one or two of the histogram values are extremely large compared to the other, that point is taken as the end point.

Next, the CPU 24 determines whether or not two end points (both ends) have been extracted (step S024). If two are extracted (step S024; Yes), the CPU 24 performs a process of determining the two extracted points as both ends (x coordinates) of the face (step S026). Here, it is assumed that coordinates a and b are obtained as both ends (x coordinates) of the face.
On the other hand, if two end points have not been extracted (step S024; No), the CPU 24 determines the end points by extracting a combination in which the distance between the two points has a plausible interval as a human face width (step S24). S025) Finally, a process for determining both ends (x coordinates) of the face is performed (step S026).

The CPU 24 stores the face end (left and right end) positions (x coordinates) obtained in step S02 in this manner in the RAM 25.
Here, when the face end position (x coordinate) is represented in the driver's face image, it corresponds to the coordinates a and b of the face end position as shown in FIG.

  Next, the processing content of the face orientation determination process (step S03) shown in FIG. 7 will be described. In the face orientation determination process (step S03), the face center line B is obtained from the face image, and the coordinates of the face center line B and the positions of both ends of the face obtained after the face end position determination process (step S02 in FIG. 6) are obtained. Based on this, it is a process for determining whether or not the left and right orientations of the face are such that both nostrils of the nose can be detected.

The method of calculating the center line B of the face (parallel to the y-axis) that is a vertical line is arbitrary, but in this embodiment, the CPU 24 calculates the center (center of gravity) position of the face in the x-axis direction from the following equation. The coordinates are obtained, and the vertical line passing through the center of gravity is calculated as the center line B of the face.
X coordinate of the center of the face = Σ (xi · Gi) / ΣGi
xi: x-coordinate value of the i-th pixel Gi: gradation value of the i-th pixel i: 1 to n (n is the total number of pixels)

  When calculating the center line B of the face (step S031), the CPU 24 determines the face end position (x-coordinate) calculated by the face end detection process (step S02 in FIG. 6) to determine whether or not the driver is facing the front. A) b is read from the RAM 25.

The CPU 24 determines whether or not the driver's face is facing right depending on whether or not the x coordinate of the face center line B is smaller than (a + b) / 2−Dth (step S032). Here, (a + b) / 2 is the center of the left and right end positions, and Dth is a threshold value in the face direction stored in the ROM 23. That is, as shown in FIG. 8, even if the driver's face is slightly shifted from the center of the left and right ends (in this case, the face is slightly facing the right), the degree of the right direction of the face is a problem in detecting both nostrils. If there is no range, the nostril detection process of step S05 is performed.
The face direction threshold Dth may be stored in the RAM 25 or the setting memory 28. Further, the threshold value Dth in the face direction may take an arbitrary value.

When the value of the x coordinate of the face center line B is smaller than (a + b) / 2−Dth (step S032; Yes), the CPU 24 determines that the face is facing right as shown in FIG. Is moved to pre-processing (step S01).
On the other hand, when the value of the x coordinate of the face center line B is not smaller than (a + b) / 2−Dth (step S032; No), the CPU 24 determines whether or not the driver's face is facing leftward. It is determined whether or not the x coordinate of B is greater than (a + b) / 2 + Dth (step S033). In other words, even if the driver's face is facing slightly to the left, the nostril detection process in step S05 is performed if the leftward degree of the face is within the range where there is no problem in detecting both nostrils.

When the value of the x coordinate of the face center line B is larger than (a + b) / 2 + Dth (step S033; Yes), the CPU 24 determines that the driver's face is facing left and preprocesses the process (step S01). ).
When the value of the x coordinate of the face center line B is not larger than (a + b) / 2 + Dth (step S033; No), the CPU 24 shifts the processing to the current candidate detection processing (step S04).

Next, the current candidate detection process (step S04) shown in FIG. 9 will be described.
The current candidate detection process (step S04) is a process in which the CPU 24 selects a histogram value candidate corresponding to the eye (eyelid) based on the histogram value.

  The CPU 24 creates a histogram of gradation values in the horizontal direction based on the horizontal Sobel image stored in the RAM 25 (step S041). Based on the histogram, the vertical coordinate (y coordinate) of a line having a high histogram value and a continuous peak value of the histogram is stored in the RAM 25 (step S042).

  FIG. 10 shows the face image and the peak value of the histogram in the horizontal direction (right triangle line). The peak value corresponds to a triangular vertex, and the peak value is larger as the vertex is on the left side. As shown in FIG. 10, the lines with high histogram values and continuous peak values of the histogram are, for example, lines (FIG. 10 (g)) passing through the upper end (the top of the head) of the face from above, and the hair. A line that passes through the boundary between the skin and the skin (Fig. 10 (f)), a line that passes through the eyebrows (Fig. 10 (e)), a line that passes directly under the eye (瞼) (Fig. 10 (d)), and a line that passes directly under the nostril There are seven lines (FIG. 10 (c)), a line passing directly under the mouth (FIG. 10 (b)), and a line passing through the lower end (chin) of the face (FIG. 10 (a)).

  The CPU 24 detects a predetermined number of these lines, sorts them based on the peak value of the histogram of the detected lines (step S043), and selects a predetermined number of the detected lines as a candidate line immediately below the eye (eye). Are stored in the RAM 25 as candidates for a line passing through (step S044). For example, in the case of FIG. 10, four lines having high histogram values (for example, FIGS. 10C, 10D, 10E, and 10F) are stored as current candidate lines.

Next, the contents of the nostril detection process (step S05) shown in FIG. 11 will be described.
The nostril detection process is a process for detecting a nostril located in the vicinity of the current candidate line stored in step S04 from the Sobel image, and determining and outputting the current line passing directly under the eye (eyelid).
In the nostril detection processing (step S05) in the present embodiment, a pixel region having both vertical and horizontal sizes of 3 to 5 pixels is determined as a nostril candidate. Note that the range of the pixel region may be arbitrarily determined.

  As shown in FIG. 11, the nostril detection process (step S05) includes a nostril search area extraction process (step S051), a nostril search area narrowing process (step S052), and a vertical pixel string extraction process (step S053). And a horizontal pixel row extraction process (step S054), a nostril determination process (step S055), and a current line storage process (step S056).

  In the nostril search region extraction process (step S051), the CPU 24 reads the face image from the image memory 22, and calculates the face end position (x coordinate) a and b calculated in the face end detection process (step S021 in FIG. 6) from the RAM 25. The y-coordinate of the current candidate line detected in the reading and current candidate detection processing (step S04 in FIG. 4) is read from the RAM 25, and a ≦ x ≦ b, ymin ≦ y ≦ ymax (ymin is currently present) among the read face images. The region having the smallest y-coordinate value among the candidate lines and ymax being the largest y-coordinate value among the current candidate lines) is cut out. For example, in the case of FIG. 10, ymin is the value of the y coordinate in FIG.

  Subsequently, the CPU 24 narrows down a region for searching for a nostril (a nostril search region) in step S051 (step S052). A method for narrowing down the nostril search region is arbitrary. In the present embodiment, the nostril search area is set as the area within the bold line shown in FIG. 12 by apportioning the face image area cut out in step S051 at a predetermined ratio.

In the vertical direction pixel row extraction process (step S053), as shown in FIG. 13, the CPU 24 scans the pixels of the binarized image in the nostril search area in the y direction, and extracts pixel rows that are nostril candidates. It is processing.
The pixel (x, y) to be processed is set to the initial value (x0, y0) as a starting point for extracting a nostril candidate image from the nostril search area narrowed down in step S052 (step S0531). Here, x0 is the value of the x coordinate of the lower left vertex of the nostril search area, and y0 is the value of the y coordinate of the lower left vertex of the nostril search area.

  The CPU 24 performs scanning in the y direction while incrementing the y coordinate value of the pixel (x, y) by one, and the gradation value is in the vertical direction from the 0th to 10th gradations (black or almost black). The coordinates of successive pixel rows are stored in the RAM 25 (step S0532). Here, the reason why the pixels of the 0th to 10th gradations are counted is that the gradation values of the nostrils are the pixels of the 0th to 10th gradations. In addition, you may count the pixel of arbitrary gradation values as a nostril.

When the scanning is completed, the CPU 24 is composed of 3 to 5 pixels that are continuous in the vertical direction in the pixel row whose pixel gradation values are the 0th to 10th gradations (the number of the nostrils is 3 to 5 pixels). It is determined whether there is a pixel column (step S 0533). If there is such a pixel column (step S 0533; Yes), the coordinates of the pixel column are stored in the RAM 25 (step S 0534).
As a result, black pixel rows having 3 to 5 in the vertical direction that are candidates for configuring the nostril feature points are extracted.

  If there is no pixel column composed of 3 to 5 pixels continuous in the vertical direction among the pixel columns having the gradation values of the 0th to 10th gradations (step S0533; No), the process proceeds to step S0535. .

  Subsequently, the CPU 24 increments the x coordinate of the pixel (x, y) by 1, and prepares to scan the next line with the y coordinate as an initial value (= y0) (step S0535).

  When the process of step S0545 ends, the CPU 24 determines whether scanning has been completed in all the nostril search areas (step S0536). If the scanning is finished (step S0536; Yes), the CPU 24 then moves the process to a horizontal pixel row extraction process (step S054). Otherwise (step S0536; No), the scanning is not completed. As a result, the process moves to a step S0532.

  The horizontal pixel row extraction process (step S054) is a nostril candidate image from the binarized image in the nostril search area narrowed down in step S053 by the same operation as the vertical pixel row extraction process (step S053). Is a process of detecting a pixel column having gradation values of 0th to 10th gradations having a magnitude of 3 to 5 in the horizontal direction.

  Although the method itself for detecting the pixel rows in the horizontal direction is arbitrary, the method shown in FIG. 14 can be used. The horizontal pixel row extraction process (step S054) in FIG. 14 is basically the same as the vertical pixel row extraction process (step S053) in FIG. 13 except that the pixels in the nostril search region are scanned in the x direction. is there.

  First, the CPU 24 sets the processing target pixel (x, y) to an initial value (x0, y0) as a starting point for extracting a nostril candidate image from the nostril search region narrowed down in step S052. (Step S0541). Here, x0 is the value of the x coordinate of the lower left vertex of the nostril search area, and y0 is the value of the y coordinate of the lower left vertex of the nostril search area.

  The CPU 24 performs scanning in the x direction while incrementing the x coordinate value of the pixel (x, y) by 1, and the gradation value is in the vertical direction from the 0th to 10th gradations (black or almost black). The coordinates of successive pixel rows are stored in the RAM 25 (step S0542). Here, the reason why the pixels of the 0th to 10th gradations are counted is that the gradation values of the nostrils are the pixels of the 0th to 10th gradations. In addition, you may count the pixel of arbitrary gradation values as a nostril.

When the scanning is completed, the CPU 24 is composed of 3 to 5 pixels that are continuous in the horizontal direction in the pixel row whose pixel gradation values are the 0th to 10th gradations (the number of the nostrils is 3 to 5 pixels). It is determined whether or not there is a pixel row (step S0543). If there is such a pixel column (step S0543; Yes), the coordinates of the pixel column are stored in the RAM 25 (step S0544).
As a result, a black pixel row having 3 to 5 in the horizontal direction, which is a candidate constituting the nostril feature point, is extracted.

  If there is no pixel column composed of 3 to 5 pixels continuous in the horizontal direction among the pixel columns having the gradation values of the 0th to 10th gradations (step S0543; No), the process proceeds to step S0545. .

  Subsequently, the CPU 24 increments the y coordinate of the pixel (x, y) by 1, sets the x coordinate to an initial value (= x0), and prepares to scan the next line (step S0545).

  When the process of step S0545 ends, the CPU 24 determines whether the scanning has been completed in all the nostril search areas (step S0546). If the scanning is finished (step S0546; Yes), the CPU 24 then moves the process to the nostril determination processing (step S055), and if not (step S0546; No), the step is assumed that the scanning is not completed. The process moves to S0542.

  As shown in FIGS. 11 and 15, the nostril determination process (step S055) is narrowed down in step S052 after the vertical pixel string extraction process (step S053) and the horizontal pixel string extraction process (step S054) are completed. This is a process for determining whether both nostrils are detected in the correct position in the image in the nostril search region.

  First, the CPU 24 obtains pixel areas of the 0th to 10th gradations in which the continuous numbers in the vertical direction and the horizontal direction are both 3 to 5 (step S0551), and determines whether or not there is the area (step S0552). . If it is determined that the pixel area exists (step S0552; Yes), the pixel area is determined to be a nostril candidate pixel area, and the process proceeds to step S0553.

When the nostril candidate pixel area is determined, the coordinates of the center (center of gravity) position in the x-axis direction and y-axis direction of the nostril candidate pixel area are obtained from the following equation (step S0553).
X-coordinate of nostril center = Σxi · Gi / ΣGi
The y-coordinate of the center of the nostril = Σyi · Gi / ΣGi
xi: x-coordinate value of i-th nostril candidate pixel whose gradation value is 0th to 10th gradation yi: i-th nostril candidate whose gradation value is 0th to 10th gradation Y-coordinate value of pixel Gi: gradation value of candidate pixel of i-th nostril whose gradation value is 0th to 10th gradation i: 1 to n (n is gradation value 0th to 10th) The total number of nostril candidate pixels)

  It is determined whether or not there is a pixel area in which the x-coordinate of the center of gravity is smaller than the x-coordinate of the face center line B among the nostril candidate pixel areas calculated in step S0553 (step S0554). If there is a pixel region that is a nostril candidate whose coordinate x of the center of gravity is smaller than the x coordinate of the face center line B (step S0554; Yes), that pixel row is extracted as a right nostril candidate pixel region (step S0555). ).

The CPU 24 further selects a pixel region that becomes a nostril candidate (candidate for the left nostril) at a position that is substantially line symmetric with respect to the center of gravity position of the candidate pixel region of the right nostril extracted in step S0555 and the face center line B. It is determined whether or not a (pixel region) exists (step S0556).
When the left nostril candidate pixel area exists (step S0556; Yes), the right nostril candidate pixel area extracted in step S0555 and the left nostril candidate pixel area determined in step S0556 are separated from each other. It is determined whether or not it exists (step S0557).

  If the pixel region of the right nostril candidate extracted in step S0555 and the candidate pixel region of the left nostril determined in step S0556 are separated (step S0557; Yes), the pixel that is a candidate for the right nostril The combination of the region and the pixel region that is a candidate for the left nostril is determined as the subject's nostril and stored in the RAM 25 (step S0558).

As described above, the eye detection device 50 according to the embodiment of the present invention can detect both correct nostrils even when the mouth of the subject is included in the nostril search area.
When both nostrils are detected, the CPU 24 moves the process to the current line storage process (step S057).

  When there is no pixel region that satisfies the determination conditions in steps S0552, S0554, S0556, and S0557 (any of steps S0552, S0554, S0556, and S0557; No), the CPU 24 determines that the nostril has not been detected correctly. Return to pre-processing (step S01).

  The CPU 24 reads the current candidate lines detected in the current candidate search process in step S04 in FIG. 4 from the RAM 25, sets the nostril search area, and performs the above-described nostril detection process (FIG. 11, step S05). Thus, the detection of the nostrils can be speeded up by performing the detection of the nostrils only in a predetermined nostril search region rather than the entire face image. Further, as described above, it is possible to determine that the current candidate line located immediately above the nostril is a line (current line) passing directly under the eye (eyelid). In other words, the eyelid detection area can be narrowed down based on the current line determined by the nostril search, and the speed of eyelid detection can be increased.

  As described above, the CPU 24 stores the determined y-coordinate y2 of the current line (in FIG. 10, the y-coordinate corresponding to (d)) in the RAM 25 (step S056).

  Next, the upper and lower eyelid determination processing (step S06) will be described with reference to FIG.

  In the upper / lower eyelid determination process (step S06), the CPU 24 first executes a process of determining an upper face area A that is an area to be searched when extracting the driver's upper / lower eyelids (step S061).

The method for setting the upper face area A is arbitrary.
In the present embodiment, the face end positions (x coordinates) a and b determined in the face end detection process (step S021) detected by the nostril detection process (step S05 in FIG. 11), and the current line storage process (step S055 in FIG. 15). The upper face area A is set on the basis of the y coordinate of the current line determined in (1).
First, the CPU 24 reads, from the RAM 25, the y coordinate y2 of the current line stored in the current line storage process (step S055 in FIG. 15). Then, a range between the y coordinate y2 (y coordinate corresponding to (d) in the case of FIG. 10) and the y coordinate ymax (ymax is the largest y coordinate value of the current candidate lines), and the x coordinate An area included in the range between a and the x coordinate b is set as the upper face area A (step S061).

That is, the pixel (x, y) in the upper face area A set in step S061 has coordinate values in the ranges of a ≦ x ≦ b and y2 ≦ y ≦ ymax as shown in FIG.
This makes it possible to shorten the upper / lower eyelid discrimination processing time by limiting the range when detecting an image showing facial feature points.

  Note that the upper face area A may be changeable by the user via the operation device 29.

The CPU 24 extracts edge lines that are candidates for upper and lower eyelids from the Sobel image in the upper face area A (step S062).
Here, the upper face area A is subjected to the horizontal Sobel filter processing (step S013 in FIG. 5), so that a pixel having a brightness in the y-axis direction is changed from a bright pixel to a dark pixel as shown in FIG. The edge line that moves is represented by a dotted line as a minus edge, and the edge line that moves from a dark pixel to a bright pixel in the y-axis direction is represented by a solid line as a plus edge.

  In step S063, the CPU 24 extracts a combination of a minus edge and a plus edge satisfying the following three formulas from the minus edge line and the plus edge line.

First, the CPU 24 determines whether or not there is a combination of a minus edge and a plus edge that satisfies Expression (1), and extracts a combination of a minus edge and a plus edge that satisfies Expression (1) (Step S063).
| Lp−Lm | <Lth (1)
However, Lp is a value representing the length of the plus edge, Lm is a value representing the length of the minus edge, and Lth is a threshold value. The lengths of the minus edge and the plus edge that satisfy Equation (1) are approximated within the range of the threshold value Lth.

Further, the CPU 24 determines whether or not there is a combination of a minus edge and a plus edge that satisfies Expression (2), and extracts a combination of a minus edge and a plus edge that satisfies Expression (2) (Step S064).
| Cxp−Cxm | <Cxth (2)
Where Cxp is the x coordinate of the centroid of the plus edge, Cxm is the x coordinate of the centroid of the minus edge, and Cxth is a threshold value. The x-coordinates of the centroids of the minus edge and the plus edge that satisfy Equation (2) are approximated within the range of the threshold value Cxth.

Further, the CPU 24 determines whether or not there is a combination of a minus edge and a plus edge that satisfies Expression (3), and extracts a combination of a minus edge and a plus edge that satisfies Expression (3) (Step S065).
Dg <Dth (3)
However, Dg is the distance between the centroid of the minus edge and the centroid of the plus edge, and Dth is a threshold value. The distance between the centroids of the minus edge and the plus edge satisfying Expression (3) is within the threshold value Dth.

That is, a combination of a minus edge and a plus edge satisfying the mathematical expressions (1) to (3) is a combination of a minus edge and a plus edge that are somewhat equal in length, close in position, and have the same x-coordinate of the center of gravity. .
From FIG. 18A, a combination of a minus edge and a plus edge satisfying the mathematical formulas (1) to (3) is extracted as a candidate for the target person's eyelid. At this time, the combination of plus edge PE1 and minus edge ME1 is candidate 1, the combination of plus edge PE2 and minus edge ME2 is candidate 2, the combination of plus edge PE3 and minus edge ME3 is candidate 3, and plus edge PE4 and minus A combination with the edge ME4 is a candidate 4.

  Then, the eye detection device 50 measures the change with respect to the time axis of the distance Dg between the centroid of the minus edge and the centroid of the plus edge for the candidates 1 to 4. The graphs shown in FIGS. 19A to 19D are graphs showing changes in the distance between centers of gravity Dg of the candidates 1 to 4 between the minus edge and the plus edge with respect to the time axis (opening of the eyelid candidate).

In candidate 1, as shown in FIG. 19A, the distance Dg hardly changes with respect to the time axis. Further, in candidate 2, as shown in FIG. 19B, the distance Dg hardly changes with respect to the time axis.
Candidates 3 to 4 have minimum values MV1 to MV6 in the graph with respect to the time axis of the distance Dg, as shown in FIGS. That is, candidates 3 and 4 can be determined to be a blinking eyelid, and the minimum values MV1 to MV6 can be determined to be the timing when the eye is closed in the blinking eyelid.

The CPU 24 determines whether there is a combination of a minus edge and a plus edge whose center of gravity moves in the y-axis direction or the opposite direction (step S066).
When there is no combination of a minus edge and a plus edge whose center of gravity moves in the y-axis direction or the opposite direction (step S066; No), the CPU 24 determines that there is no wrinkle and stores it in the setting memory 28. The setting information is changed, the setting area is changed or enlarged (step S067), and the process returns to step S062.

Among the candidates for wrinkles, as shown in FIG. 18B, when the center of gravity of the plus edge moves in the direction opposite to the y-axis direction, the center of gravity of the minus edge moves in the y-axis direction. If the center of gravity moves in the y-axis direction (step S066; Yes), the CPU 24 determines that the candidate whose center of gravity of the minus edge moves in the direction opposite to the y-axis direction is 瞼 (step S068). That is, the CPU 24 determines that the candidate is a habit of blinking.
Thus, the wrinkle is obtained, and the wrinkle detection process shown in FIG. 4 ends.

  As described above, the eye detection device 50 according to the embodiment of the present invention calculates a current line that is a horizontal line passing directly under the eye (eyelid) based on the histogram of the face image and the nostril search, and the vicinity of the current line瞼 is detected from a predetermined area. Therefore, compared with the conventional driver monitoring system that scans the entire driver's face image and selects the concentration value to detect wrinkles, it is not affected by changes in concentration such as eyebrows or under the nose. Therefore, it is possible to accurately detect wrinkles, and it is possible to eliminate misjudgment and unjudgment.

  Furthermore, the eye detection device 50 according to the embodiment of the present invention discriminates wrinkles based on the opening degree of wrinkle candidates. In addition, the driver's eye (eyelid) search area can be accurately captured, and the position of the eyelid that blinks can be detected from the eye (eyelid) search area, so that more accurate information on the driver can be captured. it can.

  Note that, based on the positions of the upper eyelid and the lower eyelid thus identified, it can be used to identify the driver's gaze direction and determine whether the driver is waking up. For example, the position of the pupil corresponding to the position of the eyelid can be obtained, the direction of the face can be obtained, and the direction of the driver's line of sight can be determined from these, and the determined line of sight can be used for various control of the vehicle it can. In addition, when the distance (Dg) between the upper eyelid and the lower eyelid is smaller than a predetermined reference value for a predetermined time or longer, it is possible to determine that the driver is sleeping and perform predetermined control. Become.

In addition, this invention is not limited to the said embodiment, A various deformation | transformation and application are possible.
In the present embodiment, the computer 14 performs eyelid detection processing on the moving image of the face of the subject photographed by the camera 10, but the moving image of the face of the subject obtained from an external device other than the camera 10 is stored in the image memory 22. The eyelid detection process may be performed on the moving image.

  In the present embodiment, for example, a nostril is searched from a predetermined area on seven current candidate lines, and the current line is determined based on the result, but the number of current candidate lines is at least better. For example, of the current candidate lines, two current candidate lines counted from the lower part of the screen, that is, a line passing through the lower end (jaw) of the face (for example, FIG. 10 (a)) and a line passing directly under the mouth (for example, FIG. 10 (b)) is easy to detect and is considered to have no nostrils. Therefore, when the current candidate line is detected, the y-coordinates of the two current candidate lines existing below may not be stored in the RAM 25. In addition, the sixth and subsequent candidate lines counted from below, that is, a line passing through the upper end (the top of the head) of the face (for example, FIG. 10 (f)) and a line passing through the boundary between the hair and the skin (for example, FIG. 10). (g)) is also considered to have no nostrils. Accordingly, the y-coordinates of these current candidate lines may not be stored in the RAM 25. In other words, only three of the current candidate lines (for example, only FIGS. 10C, 10D, and 10E) may be stored in the RAM 25. In this way, the process of searching for a nostril from a predetermined region on the current candidate line can be omitted, and the search for the nostril and determination of the current line are speeded up.

  The setting of the nostril search area is arbitrary. For example, a predetermined area having a current candidate line as a lower end may be set as a nostril search area to search for a nostril. If no nostril is found in that region, the nostril may be searched from the nostril search region set in the immediately above candidate line. If the nostril can be searched, the current candidate line immediately above the current candidate line may be determined as the current line. In the embodiment described above, the nostril search is performed after the nostril search region is determined by proportional distribution. By doing so, the nostril can be searched regardless of the proportional distribution.

  The eye detection apparatus according to the present invention can be realized using a normal computer system, not a dedicated system. For example, a program for executing the above operation is stored and distributed in a recording medium (flexible disk, CD-ROM, DVD-ROM, etc.) readable by the computer system, and the program is installed in the computer system. Thus, an eye detection device that performs the above-described processing may be configured. Alternatively, the eye detection device may be configured by storing the program in a storage of a server device on a communication network such as the Internet and downloading the program by a normal computer system.

  Further, when the above-described function is realized by sharing between an OS (Operating System) and an application, or by cooperation between the OS and the application, only the application part may be stored in a recording medium or storage.

  It is also possible to superimpose a program on a carrier wave and distribute it via a communication network. For example, the program may be posted on a bulletin board system (Bulletin Board System) on a communication network, and the program may be distributed via the network. Then, the above-described processing may be executed by starting this program and executing it under the control of the OS in the same manner as other application programs.

It is a block diagram which shows the structure of the eye detection apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the computer shown in FIG. It is a figure for demonstrating the various data stored in ROM or RAM. It is a figure for demonstrating the outline | summary of the process which concerns on embodiment of this invention. It is a flowchart for demonstrating the pre-processing of embodiment of this invention. It is a flowchart for demonstrating the face both-ends position discrimination | determination process of embodiment of this invention. It is a flowchart for demonstrating the face orientation discrimination | determination process of embodiment of this invention. It is a figure which shows a face image when a driver is facing the right direction. It is a flowchart for demonstrating the present candidate detection process of embodiment of this invention. In a horizontal Sobel image, it is a figure showing a line with a high histogram value and a continuous peak value of the histogram. It is a flowchart for demonstrating the nostril detection process of embodiment of this invention. It is a figure which shows the image which set the nostril search area | region. It is a flowchart for demonstrating the vertical direction pixel row extraction process of embodiment of this invention. It is a flowchart for demonstrating the horizontal direction pixel row extraction process of embodiment of this invention. It is a flowchart for demonstrating the nostril determination process of embodiment of this invention. It is a flowchart for demonstrating the upper and lower eyelid discrimination | determination process of embodiment of this invention. It is a figure for demonstrating the upper face area | region A of embodiment of this invention. (A) is a figure which shows the combination of the edge line used as the candidate of upper and lower eyelids in the face upper area | region. (B) is a diagram for explaining the operation to be indicated by the edge lines of the upper and lower eyelid candidates corresponding to the correct eyelid when determining the correct eyelid from among the edge lines that are candidates for the upper and lower eyelids in (a). It is. It is a figure which shows a time-dependent change of the distance between the gravity centers of the edge line used as the candidate of the upper and lower eyelids in Fig.18 (a).

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Camera 12 Illumination light source 14 Computer 16 Display apparatus 21 A / D converter (face image storage means)
22 Image memory (face image storage means, face centerline detection means)
23 ROM (Horizontal Sobel means)
24 CPU (Current position candidate detection means, Current position selection means,
Eye detection means, horizontal sobel means,
Means for generating a histogram of gradation values;
Current position candidate output means, face centerline detection means, search means,
Nostril position discriminating means, means for discriminating nostril)
25 RAM (work area of CPU 24)
26 display control device 28 setting memory 29 operation device 50 eye detection device

Claims (8)

Face image storage means for storing an image of the face of the subject,
Based on the face image stored by the face image storage means, a current position candidate detection means for obtaining a candidate for a position under the eyes in the face image;
When the face nostril is searched within a predetermined nostril search area on the face image stored in the face image storage means, and the nostril can be detected, the position under the eyes based on the detected nostril position Current position selection means for selecting a candidate located under the eyes of the face image from among the candidates,
Eye detection means for detecting an image corresponding to the eyes of the subject in an eye search area determined based on the current position selected by the current position selection means;
An eye detection device comprising: The current position candidate detection means includes:
Horizontal sobel means for processing a face image stored in the face image storage means with a horizontal Sobel filter to generate a horizontal Sobel image;
Means for generating a histogram of gradation values of pixels on each horizontal line of the horizontal Sobel image generated by the horizontal Sobel means;
Current position candidate output means for obtaining a candidate for a horizontal line passing under the eyes based on a histogram of gradation values and outputting the position of the horizontal line as a candidate for the current position;
The eye detection device according to claim 1, comprising: The current position candidate output means includes:
Based on the histogram of gradation values, a predetermined number of high histogram peak values are extracted, and the position where the extracted histogram is obtained is the current position candidate.
The eye detection apparatus according to claim 2, wherein: The current position selection means sets the nostril search region based on one of the candidates for the position under the eyes detected by the current position candidate detection means.
The eye detection device according to claim 1. A face center line detecting means for detecting a center line of the face image stored in the face image storing means;
The current position selection means includes
Search means for searching for nostril candidates of the face in the nostril search region;
Nostril position determining means for determining the position of the nostril candidate of the face detected by the search means;
Based on the position of each remark candidate determined by the nostril position determining means and the center line, the nostrils that are substantially line-symmetric with respect to the center line of the face among the nostril candidates searched by the search means Means for detecting candidate pairs and distinguishing them from nostrils;
The eye detection device according to claim 1, further comprising: When the nostril can be detected, the current position selection means is positioned above the nostril and outputs the current position candidate closest to the nostril as a position below the eye.
The eye detection device according to claim 1. Process the stored face image to obtain a candidate for a horizontal line located under the eye in the face image,
Search the nostril of the face within a predetermined nostril search area on the face image,
When the nostril can be detected, the position of the detected nostril is selected as a reference, and the candidate located below the eyes of the face image is selected from the candidates for the horizontal line,
Set an eye search area on the face image based on the selected horizontal line,
In the set eye search area, an image corresponding to the eye of the subject is detected.
An eye detection method characterized by the above. Computer
Face image storage means for storing an image of the face of the subject,
Based on the face image stored by the face image storage means, a current position candidate detection means for obtaining a candidate for a position under the eyes in the face image;
When the face nostril is searched within a predetermined nostril search area on the face image stored in the face image storage means, and the nostril can be detected, the position under the eyes based on the detected nostril position Current position selection means for selecting a position located under the eyes of the face image from among the candidates of
Eye detection means for detecting an image corresponding to the eyes of the subject within an eye search area determined based on the current position selected by the current position selection means;
Program to function as.

Images