JP3143819B2 - Eyelid opening detector - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eyelid opening detecting device, and more particularly, to a driving condition detecting device for a vehicle which detects a driving condition from an opening of a driver's eyes and warns the driver in the case of an abnormal condition. The present invention relates to an eyelid opening detection device.

[0002]

2. Description of the Related Art As a method of detecting a blink of a person by image processing, there are two methods of detecting an original image or an edge image of the original image.
A method of extracting an eye region by binarization and measuring its height, and a method of judging an open / closed eye based on whether or not an iris (so-called black eye) can be detected as a dark region (Japanese Patent Laid-Open No. 3-53)
3), a method of obtaining the area of the eye region from the saturation and the hue difference and the brightness difference between the skin and the eyeball portion in a color image (IEICE Transactions D-II, Vol. J77-D-II, NO
2, pp. 470-472 (1994) "Proposal of blink extraction method using chromaticity and brightness information".

[0003]

However, the method based on the binarization of an image has a problem that an eye region cannot be accurately extracted due to a change in lighting conditions, an individual difference in face structure, or the like. In the method of detecting the iris as a circular area,
There is a problem in that only open and closed eyes can be distinguished, and furthermore, the iris is not always detected as a circle during the opening / closing operation of the eye or while the eye is open, so that it cannot be detected accurately. In addition, the method using a color image requires a sufficient amount of white light source, and there is a problem that the use environment is limited.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problem. When detecting a blink of a person, the eyelid which is hardly affected by individual differences such as lighting conditions and the structure of a target person's face. It is an object to provide an opening detection device.

[0005]

To achieve the above object, the present invention provides a method for inputting a gray-scale image of a region including an eye, and forming a one-dimensional image representing a change in gray-scale of a pixel on a reference line in a blinking direction in the region. 1 that indicates the magnitude of the grayscale change of each pixel on the reference line based on the one-dimensional image.
Edge image calculation means for calculating a two-dimensional edge image; and a first boundary point indicating a boundary between an upper eyelid and an eyeball and a boundary between a lower eyelid and an eyeball based on an extreme value of a change in shading in the one-dimensional edge image. Eyelid detecting means for detecting the second boundary point, and eyelid opening detecting means for detecting the distance between the detected first boundary point and the second boundary point as the eyelid opening degree It was done.

[0006]

In the edge image calculation of the present invention, a gray-scale image of a region including the eyes is input, and a one-dimensional image representing a gray-scale change of a pixel on a reference line in a blinking direction is extracted in this region. Since a gray-scale image is used instead of a color image, it can be used in a wide environment without requiring a white light source with a sufficient amount of light.

The edge image calculating means calculates, for each pixel, the magnitude of the gradation change on the reference line based on the one-dimensional image, and calculates a one-dimensional edge image representing the magnitude of the gradation change for each pixel. The eyelid detecting means is configured to determine a first boundary point indicating a boundary between the upper eyelid and the eyeball and a second boundary point indicating a boundary between the lower eyelid and the eyeball based on the extreme value of the change in shading in the one-dimensional edge image.
The boundary point of is detected. Since the one-dimensional edge image is processed using the extreme value of the change in shading without binarization, the one-dimensional edge image is less likely to be affected by individual differences such as lighting conditions and a target person's face structure.

Then, the eyelid opening detecting means detects the distance between the extracted first boundary point and the second boundary point as the eyelid opening.

[0009]

Embodiments of the present invention will be described below in detail with reference to the drawings. In this embodiment, the eyelid opening detection device of the present invention is applied to a blink detection device.

As shown in FIG. 1, this embodiment includes an image photographing unit 1 composed of a CCD camera or the like for photographing an image including the face of a subject to be examined. The image photographing unit 1 includes an illuminating unit 2 including an infrared strobe or an infrared LDE for illuminating an object to be photographed in synchronization with an operation of a shutter of the image photographing unit 1 and a personal computer 7 for performing image processing.
Is connected. The personal computer 7 is connected to a display device 8 such as a CRT for displaying the number of blinks and the like.

The personal computer 7 has a CP
U, ROM storing the program of the image processing routine shown in the flowchart in FIG. 15, RA storing data, etc.
M and a bus connecting them.
This personal computer 7 will be described with functional blocks divided for each function realizing means determined based on hardware and software. As shown in FIG. 1, the personal computer 7 has a face which is an output image from the image photographing unit 1. An image input unit 3 for inputting an image is provided. The image input unit 3 includes, for example, an A / D converter and an image memory that stores image data of one screen. The image input unit 3 is connected to an eye image extraction unit 4 that extracts an image of a small area including eyes from the face image output from the image input unit 3, that is, an eye image.

The eye image extracting unit 4 processes the eye image extracted by the eye image extracting unit 4 and measures the eyelid opening. It is connected to a blink detection unit 6 that detects the number of blinks and the like from the obtained time-series eyelid opening data.

As shown in FIG. 2, the eye image extracting section 4 searches for an eye area from a face image and specifies an extraction position, and an extraction area specified by the eye area searching section 41. A small area extracting unit 42 for extracting a small area image including an eye as an eye image based on the position.

Further, as shown in FIG. 3, the eyelid opening measuring unit 5 extracts a one-dimensional image representing a change in shading from the input eye image, and based on the one-dimensional image, a pixel for each pixel on a reference line. Image generation unit 51 for generating a one-dimensional edge image representing the magnitude of the change in shading of the image, and edge image generation unit 51
And an edge extremal point search unit 52 for searching for a portion indicating an extremal point of the edge value from the edge image generated in step (1).

The eyelid opening measuring unit 5 determines an initial position for searching for a boundary point between the eyelid and the eyeball based on the luminance distribution of the eye image extracted by the eye image extracting unit 4. A position determining unit 54 is provided. The edge extremum point search unit 52 and the initial position determination unit 54 are connected to a search section determination unit 53 that determines a search section based on the results of the edge extremum point search unit 52 and the initial position determination unit 54.

The search section determining section 53 is connected to a code checking section 55 for checking the sign of the edge value of the edge image generated by the edge image generating section 51 in the search section determined by the search section determining section 53. ing.

Search section determining section 53 and code checking section 55
Based on the test results of the code checking portion 55 and an output of the search interval determining section 53, obtains and determines whether the boundary point is obtained was extracted if boundary point, only such a boundary point determining lever It is connected to a boundary point determination unit 56 that outputs a request to the search section determination unit 53 to perform a search again. The boundary point determining section 56 is connected to an opening degree counting section 57 that counts the boundary point distance determined by the boundary point determining section 56 and outputs the distance as an eyelid opening degree.

Next, the operation of this embodiment constructed as described above will be described with reference to the functional blocks described above and the CPU shown in FIG.
Will be described with reference to an eyelid opening detection routine according to the first embodiment.

First, a face image of a subject is photographed by the image photographing section 1. At this time, in order to reduce the influence of ambient disturbance light, for example, the face of the subject is illuminated by illuminating the illumination unit 2 composed of an infrared strobe in synchronization with the image capturing of the image capturing unit 1. . The illumination unit 2 may be an infrared LED that emits infrared light, in addition to an infrared strobe. Also, if continuous light is emitted from the illumination unit 2, the image capturing unit 1
The synchronization with the device is not required, and the configuration is simplified.

The image photographing unit 1 is, for example, 1/10
In synchronization with the high-speed shutter operation in about 00 seconds,
If the infrared LED emits pulse light, a stable image can be obtained even in an environment where there is much disturbance light such as outdoors. Furthermore, the illumination unit 2 can be omitted in a limited environment such as a room.

The image photographed in this way is taken into the image input unit 3 as a video signal (step 10).
0). The image input unit 3 A / D converts the video signal to generate a two-dimensional digital image (step 10).
2). In the present embodiment, the subsequent processing is performed by digital processing based on this digital image. Therefore, hereinafter, simply referring to an image means a digital image.

In the eye image extracting section 4, the eye area searching section 41 searches the face image for an eye area and instructs the small area extracting section 42 to set an area including the eye as an extracted area.
In step 2, a small area including one eye is extracted as an eye image (steps 104 and 106). FIG. 4 shows the eye image thus extracted. In order to search for this eye area by the eye area search unit 41, for example, an eye area may be searched from a face image by image processing using a template matching method,
Alternatively, the operator may indicate an eye area by pointing an eye on a face image using an eye area indicating means such as a keyboard, a mouse, an electronic pen, or a light pen.

Next, the operation of the eyelid opening measuring section 5 which is a feature of the present invention will be described. When an eye image as shown in FIG. 4 is obtained as an output of the eye image extraction unit 4, the luminance change of the original image on the center line L shown in FIG. 5A is as shown in FIG. 5B. In the edge image generation unit 51, FIG.
A one-dimensional image as described above is extracted and input, and edge processing is performed to calculate a one-dimensional edge image representing the magnitude of shading change for each pixel on the reference line as shown in FIG. 5C (step 108). , 110).

The operator performing the edge processing may be, for example, an operator for obtaining a simple difference as shown in FIG. Further, an operator having a smoothing function as shown in FIG. 6B can reduce the influence of noise. Note that a Sobel operator, a Pelewit operator, or a Roberts operator may be used as an operator for performing edge processing.

Here, the degree of eyelid opening is defined as the distance h between the boundary point A between the upper eyelid and the eyeball and the boundary point B between the lower eyelid and the eyeball on the center line L in FIG. For example, in order to measure the opening, the boundary points A and B may be detected. In general, since the eyeball part is lower in reflectance than the eyelid which is the skin part, it is photographed darkly,
At the preceding boundary point A, the luminance suddenly changes from light to dark from top to bottom, and at the subsequent boundary point B, the luminance suddenly changes from dark to light from top to bottom. At such a point of sudden change in brightness, the value of the edge image has an extreme point, and therefore, a point having an extreme value (peak value) on a one-dimensional edge image as shown in FIG. For example, it becomes possible to measure the degree of eyelid opening.

However, such a sudden change in luminance occurs not only at the boundary between the eyelids and the eyeball, but also at the boundary between the pupil and the iris, and wrinkles and shadows around the eyes. Therefore,
In this embodiment, in order to prevent erroneous detection due to wrinkles and shadows around the eyes, a point having an extreme value is selected on the edge image and the opening of the eyelid is measured by the following method.

First, on the one-dimensional edge image shown in FIG. 5C, the edge extremum point searching unit 52 selects a point Pi (i = 0, 1) having an edge value having a maximum value in the positive direction and an edge value. Is calculated as a point Mi (i = 0, 1) having a minimum value in the negative direction (the absolute value is large) (step 112). Hereinafter, both the maximum value and the minimum value will be referred to as an edge extreme point.

The initial position determination unit 54 determines the darkest point C of the one-dimensional image on the middle dotted line L in FIG. 5B among the edge extreme points Pi and Mi as the initial position of the search. And the combination of the innermost edge extreme points
i and point Mi are selected one by one (step 11).
4). In the case of FIG. 5, the combination of the point P0 and the point M0 corresponds to the edge extremum point which is the initial position of the search.

Next, the search section determination section 53 determines the first search section based on the edge extremum point searched by the edge extremum point search section 52 and the edge extremum point which is the initial position of the search. . That is, a search section is determined such that a positive extreme point is searched upward and a negative extreme point is searched downward so as to search for an outer edge extreme point from the initial position of each search (step). 116). In FIG. 5, since the point P1 and the point M1 are the next edge extreme points, the interval between the point P0 and the point P1 is the next upward search section, and the interval between the point M0 and the point M1 is This is the next downward search section. The sign checking unit 55 determines whether or not the sign of the edge value is inverted in the search section determined in this way, and outputs the determination result to the boundary point determining unit 56. The boundary determination unit 56 outputs a command to the search section determination unit 53 to repeat the search upward and downward, respectively, until no new search section is found or the sign of the edge value is inverted in the search section. (Steps 116, 118, 120). In FIG. 5, the point P
Since the edge value is always positive between 0 and the point P1 and the edge value is always negative between the points M0 and M1, the points P1 and M1 are set as new starting points, and the search is performed. Will be repeated. The search section determination unit 53 similarly determines search sections upward and downward based on the search results of the edge extremum point search unit 52, however, in FIG.
Since there is no new edge extremal point above and below M1, the boundary point determination unit 56 determines that the search is over, and determines the boundary point A between the eyelid and the eyelid between the points P1 and M1. , B (step 122).

Then, in the opening degree counting section 57, the boundary points A,
The distance between B is counted and output as the eyelid opening (step 124).

FIG. 7A shows another example of an eye image in which a shadow due to wrinkles exists below the lower eyelid. In this case as well, a one-dimensional image on the center line L in FIG.
A one-dimensional edge image as shown in FIG. 7C is calculated for the one-dimensional original image of FIG. In this example, as a search result of the edge extreme point search unit 52, the edge extreme point Pi (i = 0, 1,
2), Mi (i = 0, 1, 2) is obtained, so that a search section is determined by the search section determination section 53 according to a command from the boundary point determination section 56, and the sign check section 55 checks the sign of the edge value. Repeat the operation. 7 is the same as FIG. 5 up to the point P1 and the point M1 starting from the point P0 and the point M0. However, when searching from the point M1 to the point M2, , The sign of the edge value is inverted, and a positive region edge extreme point P2 exists. Therefore, the search ends at the point M1, and the distance between the points P1 and M1 is output as the eyelid opening.

FIG. 8A shows another example of an eye image in which there is a shadow due to wrinkles on the upper eyelid. In this case, as in the above case, a one-dimensional image on the center line L in FIG.
A one-dimensional edge image as shown in FIG. 8C is obtained for the one-dimensional original image of FIG. In this example, as the search results of the edge extreme point search unit 52, edge extreme points Pi (i = 0, 1, 2),
Since Mi (i = 0, 1, 2) is obtained, a search section is determined by the search section determination section 53 according to a command from the boundary point determination section 56,
The operation of checking the sign of the edge value in the sign checking unit 55 is repeated. 8 is the same as that of FIG. 5 up to the point P1 and the point M1 starting from the point P0 and the point M0, but the eyelids are searched when the point P1 is searched for the point P2. The sign of the edge value is inverted due to the presence of the dark area due to the upper shadow,
There is a negative edge extreme point M2. Therefore, the search ends at the point P1, and the distance between the points P1 and M1 is output as the eyelid opening.

According to this embodiment, the boundary point between the eyelid and the eyeball can be extracted irrespective of individual differences such as the structure of an individual face by this processing, and accurate measurement of the eyelid opening can be performed. Will be possible.

If the above-described processing is repeatedly performed on the images output from the image pickup unit 1 one after another at predetermined time intervals, the output of the eyelid opening degree measurement unit 5 is time-sequential as shown in FIG. For example, an eyelid opening degree data string is obtained every 1/30 seconds. The blink detection unit 6 calculates and outputs the blink closing time and the number of blinks per unit time from the eyelid opening data sequence.

In the above embodiment, as shown in FIG. 5 (a), image processing was performed on one reference line on the center line L. However, as shown in FIG. The processing is performed at an appropriate distance d on both sides of the center line L, for example, 3
If the measurement is performed for a plurality of reference lines each for a total of seven and the eyelid opening degrees are averaged, measurement that is less affected by noise or the like can be performed. Further, the schematic shape of the eye can be detected from the coordinates of the boundary point.

In the above embodiment, the processing is performed on the center line L of the eye image output from the eye image extracting unit 4.
As shown in FIG. 11, if processing is performed for a plurality of reference lines provided at appropriate intervals d over the entire horizontal width of the small region including the eye, even if the eye region does not exist at the center of the eye image,
Accurate measurement is possible. When such a process is performed, as shown in FIG. 12A, the process for the reference line L1 passing through the non-eye portion, or as shown in FIG. Processing of the reference line L2 passing through the shadow due to wrinkles or the like becomes a problem.

However, in the case shown in FIG. 12A, the one-dimensional edge image does not have a clear edge extreme point as shown in FIG. In general, the extreme point of an edge caused by a shadow is shown in FIG.
As is often the case, as in (b), only edge extreme points that are equal to or greater than a predetermined value can be measured without being affected by them.

If the shadow edge is stronger than expected,
Even if an erroneous detection is made as an eye boundary point, as shown in FIG. 14, the boundary point group is divided into an eye and a shadow part and is divided into two groups. If only the boundary points of the eyes are extracted from the determination of the size of the transformed region, the eyelid opening can be accurately measured.

Further, if the eye boundary point group is calculated as described above, for example, the center position of the eye area can be obtained from the barycenter coordinate by calculating the barycenter coordinate of the eye boundary point group. Therefore, the eye image can be extracted by the small area extracting unit 42 based on the center position of the eye area in the previous frame without finding the position of the eye area by the eye area searching unit 41 in the previous embodiment. In this case, since the position of the eyes moves in the image due to the movement of the subject's head and the like, the eye area does not always exist at the center of the new eye image, but the eyelid opening measurement is performed over the entire width of the small area. Since the processing in the unit 5 is performed, accurate eyelid measurement can be performed. That is, normally, the eye can be tracked and the eyelid opening can be measured without operating the eye area search unit 41.
Since the processing by 1 only needs to be performed once at the time of startup and only when eye tracking has failed due to rapid movement of the subject or the like, higher-speed processing can be performed.

[0040]

As described above, in the present invention, the edge image is processed by using the extreme value of the change in shading without being binarized. The effect that the eyelid opening can be stably measured without being performed is obtained.

Further, since the eyelid opening is directly measured, not only the eye opening / closing but also the change of the eye opening / closing speed and the opening at the time of eye opening can be measured.

Since the edge image is processed one-dimensionally, the amount of calculation can be reduced.

[Brief description of the drawings]

FIG. 1 is a block diagram of the present embodiment.

FIG. 2 is a block diagram illustrating details of an eye image extracting unit in FIG. 1;

FIG. 3 is a block diagram showing details of an eyelid opening measurement unit.

FIG. 4 is a plan view showing an eye image.

5A is a diagram illustrating an original image of an eye image, FIG. 5B is a diagram illustrating a one-dimensional original image obtained from the original image, and FIG. 5C is a diagram illustrating a one-dimensional edge image obtained from the one-dimensional original image. .

6A is a conceptual diagram showing an operator for obtaining a simple difference, and FIG. 6B is a conceptual diagram showing an operator having a smoothing function.

7A is an original image of an eye image having a shadow below a lower eyelid, FIG. 7B is a one-dimensional original image obtained from the original image, and FIG. 7C is obtained from the one-dimensional original image FIG. 3 is a diagram illustrating one-dimensional edge images.

8A is an original image of an eye image having a shadow on the upper eyelid, FIG. 8B is a one-dimensional original image obtained from this original image, and FIG. 8C is obtained from this one-dimensional original image FIG. 3 is a diagram illustrating one-dimensional edge images.

FIG. 9 is a diagram showing time series data of an eyelid opening degree.

FIG. 10 is a conceptual diagram in a case where a process by an eyelid opening measurement unit is performed on a plurality of reference lines.

FIG. 11 is a conceptual diagram in a case where processing is performed over the entire horizontal width of a small area including an eye.

FIG. 12A is a reference line L1 passing through a non-eye portion;
And (b) is a diagram showing a one-dimensional edge image obtained from the original image of (a).

13A is a diagram showing a reference line L2 passing through a non-eye portion, and FIG. 13B is a diagram showing a one-dimensional edge image obtained from the original image of FIG.

FIG. 14 is a diagram showing a state in which a group of boundary points is divided into two groups by being divided into an eye portion and a shadow portion.

FIG. 15 is a flowchart illustrating an image processing routine according to the present embodiment.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Image imaging part 2 Illumination part 3 Image input part 4 Eye image extraction part 5 Eyelid opening measurement part 6 Blink detection part

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-3-533 (JP, A) JP-A-3-202045 (JP, A) JP-A-6-32154 (JP, A) (58) Field (Int.Cl. ⁷ , DB name) A61B 3/113 B60K 28/06

Claims (6)

(57) [Claims] 1. A grayscale image of an area including an eye is input, and a one-dimensional image representing a grayscale change of a pixel on a reference line in a blinking direction is extracted in the area, and based on the one-dimensional image, a one-dimensional image on the reference line is extracted. 1 representing the magnitude of shading change for each pixel
Edge image calculating means for calculating a three-dimensional edge image; and a first boundary point indicating a boundary between an upper eyelid and an eyeball and a boundary between a lower eyelid and an eyeball based on an extreme value of a change in shading in the one-dimensional edge image. Eyelid detecting means for detecting a second boundary point; and eyelid opening detecting means for detecting a distance between the detected first boundary point and the second boundary point as an eyelid opening degree. Opening detector. 2. The image processing apparatus according to claim 1, wherein said edge image calculating means is configured to output the extracted primary image.
In the original image, the values of adjacent pixels on the reference line are used.
The one-dimensional edge image by smoothing each pixel.
2. An eyelid according to claim 1, wherein the image is calculated.
Opening detector. 3. The method according to claim 2, wherein the eyelid detecting means includes the one-dimensional edge.
The outermost edge of the image where the sign of the extreme value
The extreme pixel on the side is defined as a first boundary point and a second boundary point
An eyelid according to claim 1 or claim 2, characterized in that:
Opening detection device. 4. An eyelid detecting means according to claim 1 , wherein said eyelid detecting means includes an area including said eye.
On a plurality of reference lines in the blink direction,
And detecting a second boundary point.
4. The eyelid opening detection device according to claim 3. 5. An eyelid detecting means according to claim 1 , wherein said eyelid detecting means comprises :
From the pixel at the edge extreme point where the value of the image is equal to or greater than a predetermined value, the first
A boundary point and a second boundary point are detected.
The eyelid opening detection device according to claim 4. 6. The eyelid opening detecting means includes:
The statistical processing of the coordinates of the boundary points detected on the line
Rasterize and determine the size of clustered areas
Extract eyelid opening by extracting only eye boundary points
An eyelid opening detection device according to claim 4 or 5, wherein
Place.