JP3116674B2 - Driver status detection device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for detecting a driver's state by detecting whether the driver's eyes are open or closed and detecting a drowsiness.

[0002]

2. Description of the Related Art Conventionally, as an apparatus capable of detecting the state of a vehicle driver, there is an apparatus disclosed in Japanese Patent Application Laid-Open No. 62-247410, for example. This binarizes the captured image data, recognizes the position of the driver's eye as an area based on the continuity of white pixels and black pixels, determines open / closed eyes in the area, and detects the state of the driver. It is configured to be usable for detecting a driver's falling asleep.

[0003]

However, in the above-described vehicle driver state detecting apparatus, the eye area is set only by the continuity of the white pixels and the black pixels in the binary image data. When a person wearing the object is targeted, frames of glasses or the like remain as noise in the binarized data,
There is a problem that the eye area cannot be set accurately.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and has as its object to provide a driver state detecting device capable of further improving the detection performance of a driver's eye region.

[0005]

According to a first aspect of the present invention, there is provided a driver state detecting apparatus for inputting a driver's face image as face image data; A light emitting means for irradiating an invisible light beam on the face, a glasses presence determining means for determining the presence or absence of glasses by image recognition of a reflection point of the light emitting means reflected on the glasses in a grayscale image sent from the image input means, and the presence or absence of the glasses By the judgment means
If it is determined that there are glasses, the glasses worn by the driver
The presence of the eye based on the reflection point of the light emitting means reflected in the lens of
Eye existence area setting means for setting an area, open / closed eye detection means for detecting the driver's open / closed eye in the area, and a driver for judging the state of the driver based on the open / closed eye signal from the open / closed eye detection means State determination means.

[0006] the driver state detecting apparatus of the second invention, before
When the presence / absence of eyeglasses is determined by the eyeglass presence / absence determination means, a second eye existence area setting means for determining an area including the eyes of the face image data is provided, and the presence of the second eyes is provided. Area setting means for binarizing an input image sent from the image input means, and white pixel count for scanning the image in the left-right direction and counting the number of continuous white pixels in the binarization means Counting means, stable part detecting means for detecting a stable part at the end in the width direction of the face image from the amount of change in the number of pixels, continuation of the contour line of the face image from the presence or absence of the stable part and the amount of change in the number of continuous pixels Continuity continuity determination means for determining the nature, and face edge identification means for identifying the edge of the face based on the determination,
An eyeball left / right area determining means for determining a lateral position of an eye present area from an end of the face specified by the face end specifying means, and a vertical lower part in the determined lateral present area of the eye. Setting the eye existence area by black area detection means for detecting a black area from the eye, and eye vertical area determination means for determining the vertical position of the eye existence area based on the detected black area. It is characterized by.

[0007]

[0008]

According to the first aspect of the present invention, a driver's face image is input as face image data by image input means, an invisible light beam is radiated on the driver's face by a light emitting means, The presence or absence of spectacles is determined by recognizing the image of the reflection point of the light-emitting unit reflected on the spectacles in the grayscale image transmitted from the input unit. Then, the eyeglass presence determination means
If it is determined that glasses are present, the eye existence area setting
The step is reflected in the lens of the glasses worn by the driver
An eye existence area is set based on the reflection point of the light means , the driver's open / closed eye is detected by the driver's open / closed eye detector in the area, and a driver's state determination means outputs the driver's open / closed eye signal. The state of the driver is determined based on the signal.

[0009] The second invention by the above configuration, when it is determined that without glasses by the spectacle presence determining means, a region including the eyes of the face image data by the presence area setting means of the second <br/> eye The driver's open / closed eye detection means detects the driver's open / closed eye in the area and outputs an open / closed eye signal, and the driver state determination means determines the driver's state based on the open / closed eye signal. At this time, the second eye existence region setting means binarizes the input image sent from the image input means by the binarization means, and scans the image in the horizontal direction by the binarization means by the white pixel number counting means. Then, the number of continuous white pixels is counted, the stable portion at the end in the width direction of the face image is detected from the change amount of the number of pixels by the stable portion detecting means, and the presence / absence of the stable portion and the number of continuous pixels are detected by the contour continuity determining means. The continuity of the contour line of the face image is determined from the change amount of the face, and the face edge specifying means specifies the face edge based on the continuity determination. Further, the lateral position of the eye existence region is determined from the end of the face specified by the face edge specifying device by the eyeball left / right region determination device, and the lateral existence region of the eye determined by the black region detection device is determined. Then, a black region is detected from below in the vertical direction, and the position of the eye is determined in the vertical direction based on the black region detected by the vertical region determining means of the eyeball, thereby setting the region where the eye is present.

[0010]

[0011]

FIG. 1 is a block diagram showing an embodiment of a vehicle driver state detecting apparatus according to the present invention, which is applied to an automobile.
FIG. 2 is a flowchart showing the operation of the vehicle driver's state detection device based on the configuration of FIG.

In FIG. 1, the state detecting device is an infrared lamp 1,
2 (infrared lamps A and B), image input camera 3, A /
It has a D converter 4, an image memory 5, an eyeglass presence / absence determination circuit 6, an eyeball presence position determination circuit 7, an open / closed eye determination circuit 8, and a dozing determination circuit 9.

In FIG. 1, an image input camera 3 is installed in the instrument panel in front of the driver in order to photograph the driver's face.

When the driver wears spectacles at the position of the image input camera 3, the reflected image of the infrared lamp A itself, which is reflected on the lens of the spectacles, is photographed by the image input camera 3. (Hereinafter referred to as reflection). In this case, the infrared lamp A
Is located on the lens S of the spectacles when the incident light of the image of the infrared lamp A is
Is provided in consideration of the reflection angle / incidence angle so that the light is reflected by the camera and enters the image input camera 3.

On the other hand, the infrared lamp B is provided at a position that does not cause reflection on the spectacles. In this embodiment, an infrared lamp for night illumination is used as the infrared lamp B.

In this embodiment, the input image of the image input camera 3 is 520 pixels in the horizontal (X) direction and 500 pixels in the vertical (Y) direction as shown in FIG. 3 (a window showing an area photographed by the camera 3). The angle of view is adjusted so that the face portion becomes almost full in the vertical direction.

The image memory 5 is connected to the image input camera 3 via the A / D converter image 4, and inputs and stores input image data obtained by the image input camera 3. Note that A
The / D converter 4 converts the input image captured by the image input camera 3 into a digital amount and supplies the digital amount to the image memory 5.

The glasses presence / absence determination circuit 6 is connected to the image memory 5 and determines the presence / absence of a high luminance point of the infrared lamp A reflected on the lens portion of the glasses based on the input image data stored in the image memory 5. .

The eyeball presence position determining circuit 7 is connected to the spectacle presence / absence determination circuit 6, and defines the eyeball presence position region according to the result of the spectacle presence / absence determination circuit 6.

The open / closed eye determination circuit 8 is connected to the eyeball present position defining circuit 7 and processes the image data in the image memory 5 within the area defined by the eyeball present position defining circuit 7 to determine the open / closed eye.

The drowsiness determination circuit 9 is connected to the open / closed eyes determination circuit 8 and determines the presence or absence of a driver's drowsiness from the determination result of the open / closed eyes determination circuit 8.

Next, the operation of the whole device for detecting the state of a driver will be described with reference to the flowchart of FIG.

First, in step S21, the seating of the driver is confirmed by a seat sensor or the like. When the seating of the driver is confirmed, the process proceeds to step S22, in which the timer is started to turn on the infrared lamp A for causing the spectacles to be reflected.

Thereafter, the flow proceeds to step S23, where the spectacle presence / absence determination circuit 6 determines the presence / absence of a high luminance point of the infrared lamp A reflected on the lens portion of the spectacles. If there is a high-luminance point, the process moves to step S28, and the eye-ball-existing-position specifying circuit 7 sets the eye-existing area by the eye-existing-area setting method B (described later).

On the other hand, if there is no high-luminance point, the process proceeds to step S24, and the search for the high-luminance point by the glasses presence / absence determination circuit 6 is repeated for a predetermined time.

In this case, if no high-luminance point exists even after the lapse of a predetermined time, the flow shifts to step S25 to stop the timer and clear the timer counter. Thereafter, the process proceeds to step S26, where the eyeball existence position defining circuit 7
The eye area is set by the setting method A (described later) of the eye existence area.

The processes in steps S26 and S28 and thereafter are performed based on a binarized image obtained by binarizing a grayscale image as a current image with a certain threshold value.

After setting the eye existence area in steps S26 and S28, the flow proceeds to steps S27 and S29 to turn off the infrared lamp A for determining the presence or absence of glasses. Thereafter, step S2
The process proceeds to step S10, and the open / closed eye determination circuit 8 determines the open / closed eye in the eye existence region set in steps S26 and S28. In step S211, the dozing determination circuit 9 makes a dozing determination based on the time-series determination result of the open / closed eyes.

The details of the processing of setting the eye existence area A in step S26 will be described with reference to FIGS. The details of the process of setting the eye existence region B in step S28 are described in FIG.
This will be described with reference to FIG. Details of the processing for determining whether the eye is open or closed in step S210 will be described with reference to FIGS.

FIG. 4 is a flowchart for determining the horizontal position of each of the right and left windows related to the setting A of the eye existence area in step S26 of FIG.

In steps S101 and S102 in this figure, as shown in FIG. 7, the center of the image (X coordinate = 250)
Is set as the search start line, and the number of white continuous pixels WXCL and WXCR are separately counted from the left and right, and the X coordinate XLM and XR at the left and right ends when the number of white continuous pixels is the maximum.
M is stored. This processing will be described later with reference to FIG.

Subsequently, in step S103, it is determined whether or not the sum of the number of continuous pixels on the left and right is greater than 200.
If the number of eyebrows, eyeballs, etc. is 200 or less, step S104
And left stability counter WLCON and right stability counter WRCO
N, left stability flag STFLGL, right stability flag STFL
GR is cleared, and the process returns to step S101 to scan the next line.

If it is determined in step S103 that the sum of the numbers of continuous pixels on the left and right is greater than 200, it is checked in step S105 whether the left end point detection flag OKFGL has been set. If it has been set, the right end search after step S120 has been performed. Move on to processing.

If the left end point detection flag OKFCL is not set (OKFCL = 0), step S10
In steps S6 and S107, if the difference between the number WXCL of continuous white pixels and the number MAEL of continuation in the immediately preceding scan line is less than 10, the right stability counter WCON is counted up.

In steps S108 and S109, the current scanning line is a stable candidate portion having a small difference from the number of continuations of the immediately preceding scanning line. Is stored as X2.

In steps S110 to S112, it is checked whether the left stability counter WLCON has exceeded 10 and the difference between X1 and X2 is less than 30, and if this condition is satisfied, a left stability flag STFGL is set. White continuous pixels W
XCL is newly set as MAEL.

In step S106, the number of continuous white pixels WXC
If the difference between L and the continuation number MAEL in the previous scan is 10 or more, it is determined that there is a possibility that the continuity of the outline has been lost, and the process proceeds to steps S113 to S119.

In this part, it is determined whether the number of continuous pixels has changed greatly due to the eyebrows or the like or the contour line is broken based on the change in the number of continuous pixels and the existence of a stable portion.

First, at step S113, the left stability flag S
The TFLGL is checked, and if a stable portion exists and the STFLGL is set before the number of continuous white pixels greatly changes, it is checked in step S114 whether the number of continuous pixels has increased.

If the number of continuous pixels has increased, it is determined that the contour has been interrupted, and the X coordinate X1 stored in step S108 in step S115 is set as the left end of the face.

If the stable portion does not exist before the number of continuous white pixels greatly changes, or if the number of continuous pixels decreases even though the stable portion exists, the portion where the contour is interrupted changes from the portion where the contour is interrupted to the portion having the contour. , Or scanning of eyebrows, eyes, eyeglasses, and the like.

If the number of continuous white pixels in the current scan is smaller than the previous number of continuous white pixels by 50 or more in step S116, it is determined that the scanning has been shifted from the portion where the outline is interrupted to the portion where the outline is present. And the left end point X of the current scan
Let L be the left end of the face.

When the left end of the face is set in step S115 or step S118 as described above, step S11 is executed.
In step 9, the left end point detection flag OKGLG is set, and the process proceeds to step S120.

If the current continuous number is not greatly reduced from the previous continuous number, it is considered that the eyebrows, eyes, eyeglasses, or a large shaded part, so that X1 and X1 are determined in step S117.
2. Clear WLCON, STFLGL and step S
Proceed to 112.

In steps S120 and S121, a decision is made as to whether or not the right contour of the face has been cut off, and the right end is set when the contour has been cut off, in the same manner as in steps S105 to S109.

The above processing is performed in steps S122 and S12.
As shown in FIG. 3, both the left and right end detection flags of the face are set, or the processing is continued until the preset search range in the Y direction ends.

When both end point detection flags are set, the horizontal width setting ends. When the search range in the Y (vertical) direction is completed, steps S124 to S124 shown in FIG.
It moves to the process of S127.

Here, it is checked whether or not the left and right end point detection flags are set. If not set, the left and right end point detection flags are set when the continuous number of left and right white pixels is the maximum in all the scans stored in step S102. End X coordinate XLM, XR
Let M be the left and right ends of the face.

If the left and right edges of the face can be detected as described above, the horizontal position of the window can be determined according to the following equation.

X-axis center = Xc = XLM + ((XR
M−XLM) / 2) • Left X coordinate of left eye window = X1 = XLM • Right X coordinate of left eye window = X2 = Xc-25 • Left X coordinate of right eye window = XX1 = Xc + 25 • Right eye window Right X coordinate = XX2 = XRM In this way, even if the contour line of the face image loses continuity due to external light radiating from one side of the face, the presence or absence of a stable portion at the width direction end of the face and the number of continuous pixels By determining the continuity of the outline from the change, the left and right edges of the face can be specified.

Further, in the binarized image, the left and right edges of the face are detected by scanning the center of the image, that is, from the approximate center of the face to the left and right, based on the continuation of white pixels. Thus, accurate detection can be performed.
In addition, the left and right edges of the face can be detected at the cheeks and the like without being affected by hairstyles, glasses, and the like. Furthermore, since the left and right ends of the face are independently detected, even if one end of the face cannot be detected due to the influence of external light, the opposite end can be detected.

The process of storing the left and right ends XLM and XRM when the number of continuous white pixels is the maximum in steps S101 and S102 will be described below with reference to the flowchart shown in FIG.

First, in step S201, vertical scanning Y
The coordinates are set to 40. This Y = 40 is obtained by speeding up the process on the assumption that the maximum value of the face does not exist within this range. In step S202, the X coordinate value 250 of the horizontal search scanning start line is converted into a left search X coordinate XL and a right search X coordinate XR.
Set to. If the vehicle driver is within the angle of view of the camera as shown in FIG. 7, the X coordinate value definitely specifies the line existing in the face.

Next, in step S203, it is checked whether or not the right scanning end flag OKR has been set. If it has been set, the left end of the face is searched for after step S211. If ORK is not set, step S20
In step S205, whether the pixel J (XR, Y) is white is determined to be white.
In step S206, the search X coordinate XR is counted up.

In step S204, the pixel J (XR,
If Y) is not white, OKR is set in step S207, and in step S208, the maximum value XRM of the right end point stored so far is compared with the current end point XR, and XR is set.
Is larger (if it is on the right side), step S
At 209, XR is set as a new end point XRM.

Next, in steps S211 to S217, the same processing as described above is performed on the left side. The difference from the search on the right side is that the search X coordinate XL is counted down in step S214, and the smaller (the left side) of the left end point XLM and the current end point XL stored in steps S216 and S217 is used. That is, the left end point XLM is set.

The left and right ends are detected in one scanning line. If it is determined in step S221 that both the scanning end flags OKL and OKR have been set, these flags are cleared (= 0) in step S222, and step S223 is performed. Then, the search line Y is counted up, and this processing is completed.

FIG. 6 shows the detection of the coordinates in the Y direction of the eye existence area, which is performed for each of the right and left eyes.

For the left eye, as shown in FIG. 8, the starting point is 10 dots to the left of the right X coordinate X2 of the left eye window, that is, X2-10 (this is to avoid detection of a black nose). X in the horizontal direction (X direction) from the position
The range is from 2-90, and 0 from the Y coordinate YL at the start of the search.
Is searched in the vertical direction (Y direction) in the range, and the search is performed in the horizontal direction at intervals of 4 dots. The Y coordinate YL is set below the scan line whose left and right ends have been determined.

As for the right eye, the starting point is 10 dots to the right of the left X coordinate XX1 of the right eye window, that is, XX1 + 10, and XX1 + 9 in the horizontal direction (X direction) from this position.
The search is performed vertically upward (Y direction) in the range from the Y coordinate YL to 0, and the search is performed in the horizontal direction at intervals of 4 dots.

In FIG. 6, first, in step S61, the memory variable BY1M of the maximum value (lowest point) of the Y coordinate of the black area.
AX is cleared and the detection range defining counter XC in the X direction is cleared.
HECK is initialized to X2-10, and a search range counter YCHECK in the Y direction is initialized to YL.

Next, in step S62, it is determined whether or not the X-direction search range defining counter XCHECK is equal to or less than X2-90. This determination is for determining whether or not all searches in the X direction have been performed. If the search in all of the X directions has not been completed, the process proceeds to step S63, and if completed, the process proceeds to step S63.
The process moves to 74.

A step S63 decides whether or not all the Y directions on the X axis have been searched. If the search in all the Y directions has not been completed, the process proceeds to step S64, and if completed, the process proceeds to step S73.

In step S64, the maximum Y at which a black area appears
It is intended to detect the coordinates, and it is determined whether or not the pixel whose search has started is a black pixel. If the pixel is not a black pixel, the process proceeds to step S72, the counter BLACK of the black pixel is cleared, and the process proceeds to step S71. If the Y coordinate is counted down by one and the determination in step S63 is obtained and the search in all the Y directions is not completed, the next pixel is determined in step S64.

This process is repeated from the start of the search until a black pixel appears. If it is determined in step S64 that the pixel is a black pixel, the process proceeds to step S65, and the black pixel counter BLACK is counted up.

Next, the process proceeds to step S66 to determine whether or not the black pixel has appeared for the first time. Judgment is counter B
The LACKE is checked, and if the counter BLAKE = 1, it is assumed that the LACK has appeared for the first time.

If it is the first black pixel, step S6
7, the Y coordinate is substituted for SETY, and step S6 is executed.
Move to 8. If the black pixel does not appear for the first time in step S66, the process proceeds to step S68.

In step S68, the black pixel counter BL
It is determined whether the value of ACK exceeds 5 or not. This process takes into account that when the number of continuous black pixels is five or less, it is unlikely that the black pixels are eyes. If the number of continuous black pixels is less than 6, the process proceeds to step S72, where the black pixel counter BLACK is cleared, and step S71 is performed.
, The Y coordinate is counted down by one, and the search in the Y direction on the X axis is continued (that is, step S63).
To S68).

If the number of continuous black pixels is five or more in step S68, the process proceeds to step S69. In step S69, the maximum value BY1MAX of the Y coordinate at which the black pixel has appeared is updated for each X axis, and the Y coordinate SETY at which the black pixel region has appeared at the X axis is the maximum value BY1M of the Y coordinate so far.
It is checked whether or not AX exceeds AX. If SETY exceeds BY1MAX, the process proceeds to step S70 to update BY1MAX, and proceeds to step S73 to repeat the same processing (steps S62 to S69) on the next X axis. SETY is BY1M
If AX is not exceeded, the flow shifts to step S73 without updating BY1MAX.

When it is confirmed in step 62 that all the X axes have been searched, the flow shifts to step S74 to set the maximum value YT and the minimum value YB on the Y coordinate of the eye existence area.

FIGS. 9 and 10 are explanatory diagrams showing a method of determining the eye existence area in the case where the driver is wearing glasses, that is, the eye existence area setting B (see FIG. 2) in step S28. It is.

FIG. 9 shows a case where a high-brightness reflection point of an infrared lamp is present on the lens of the glasses worn by the driver, and this high-brightness reflection point can be freely determined by the arrangement and number of the infrared lamps. Since the setting can be made, it is possible to distinguish even if there is reflection of another high luminance point.
Further, since the image is reflected on the lens portion of the spectacles, the existence region of the eye can be determined based on the high luminance point. FIG. 10 shows an example thereof.
The specularly reflected light from the corneal surface of the eyeball illuminated by the infrared lamp is captured by the camera 3 and observed as a bright spot (corneal reflection image) 18. The luminance reflection point 19 is shown.

As described above, according to this embodiment, it is possible to accurately cut out the region where the eyes are present irrespective of the presence or absence of the driver's glasses, and it is possible to detect drowsiness with higher accuracy.

Further, assuming that the sun is directly hitting the face, the sun visor or the like can be linked with the turning on of the infrared lamp for judging the presence or absence of spectacles to shade the area.

FIG. 11 is a flowchart for explaining an example in which the judgment of the open / closed eye is made based on the coefficient of the number of continuous black pixels of the eyeball portion in the eye existence area.

This processing is for detecting the coordinates in the Y direction of the window, and is performed for each of the left and right eyes. First, the coefficient processing of the number of continuous black pixels will be described with reference to FIG.

In FIG. 12, starting from X2 of the right X coordinate of the window of the left eye that has been automatically searched as a start point, the Y coordinate Y of the search start is set in a range from this position to X1 in the horizontal direction (X direction).
A search is made from Y to YB-50 in the vertical direction (Y direction), and the search is performed at intervals of one dot in the horizontal direction.

For the right eye, the starting point is XX1, which is the left X coordinate of the window of the right eye, and the range extends from this position to XX2 in the horizontal direction (X direction). The Y coordinate of the search start is the window coordinate of the right eye. Search YB'-50 vertically upward (Y direction) from YB 'obtained in the setting process,
This is performed at intervals of one dot in the horizontal direction.

In FIG. 11, first, in step S401, the specified counter XECHECK of the detection range in the X direction is set to X.
2, and a specified counter YECH for the detection range in the Y direction.
ECK is initialized to YB and the number of continuous black pixels is cleared.

Next, in step S402, it is determined whether or not the search range defining counter XECHECK in the X direction is equal to or less than YB-50. This judgment is for judging whether or not all the search has been performed in the X direction. If the search is completed in the X direction, the process ends.

If it is determined in step S402 that all of the images have not been searched in the X direction, the process proceeds to step S403.

Then, in step S403, a flag E for detecting the number of continuous black pixels when a new X-axis search is performed
The BFL, the flag EWFL for detecting the number of continuous white pixels, the black pixel continuous counter BLAKE, and the white pixel continuous counter WHITEEE are cleared, and the routine goes to Step S421.

In step S421, it is determined whether or not the search initial value on the Y axis is a white pixel. If the initial search value is not a white pixel, that is, if it is a black pixel, it is determined that hair or the like is in the window, the search in the Y direction on the X axis is stopped, and the flow advances to step S418 to move to the next X axis. Move on to search. If the search initial value of the Y axis is a white pixel, step S4
Move to 22.

In step S422, it is determined whether or not the search has been made in all of the X-axis and Y-axis directions. If all the searches in the Y-axis direction have been completed, the flow shifts to step S418, and shifts to the next X-axis search.

In the flowchart after step S404, a specific example of the search in the Y-axis direction will be described separately for the scanning lines (a) to (f) in FIG.

(1) In the case of the pattern (a) In step S404, it is determined whether or not the continuous white pixel flag EWFL is set (if EWFL = 1, set). If the flag EWFL is set, step S41
0, but in the pattern (a), the flag EWFL
Is not set, the process moves to step S405, and it is determined whether the corresponding dot is a black pixel or a white pixel. FIG.
As is clear from FIG. 2, in the case of the pattern (a), the process proceeds to step S412 because the pixel is a white pixel.

In step S412, the presence or absence of the setting of the flag EBFL of the continuous black pixel (set if EBFL = 1)
Is determined. In the pattern (a), since the flag EBFL is not set, the flow shifts to step S416 to start counting of the white pixel counter WHITEEE.

Next, in step S417, it is determined whether or not the number of continuous white pixels has exceeded 30.
The process proceeds to step S419 until the value exceeds 0, the Y coordinate is counted down, and the same procedure is repeated from step S422 to count up the number of white pixels.

If the number of white pixels exceeds 30 in step 417, the process proceeds to step 418 even if the search range of the Y coordinate is not satisfied, and the process proceeds to the next X-axis search (step S4).
The reason why the number of continuous white pixels is limited to 30 at 17 is that the black pixels that appear after 30 or more white pixels have a low probability of being eyes and a high probability of eyebrows, hair, etc. This is to prevent counting of the represented black pixels.)

In the search for the X-axis in the case of the pattern (a), no black pixel appears, and therefore, step S
Since the value does not shift to 420, the value of the maximum black pixel number BLACKEMAX used for the judgment of the open / closed eyes remains 0,
The numbers are not updated.

(2) In the case of pattern (b) Returning to step S418, the process proceeds to search for a new X axis, and the flag EBF for detecting the number of continuous black pixels in step S403.
L, a flag EWFL for detecting the number of continuous white pixels, a black pixel continuous counter BLAKE, and a white pixel continuous counter W
After clearing HITEE, the process moves to step S404.

In step S404, since the flag EBFL and the flag EWFL have been cleared in step S403, the steps S405 and S405 are performed in the same manner as in the case of the pattern (a).
S412, S416, S417, S420, S419,
From S404 to S405, the Y coordinate is counted down until a black pixel appears, and a white pixel continuous counter WHITE is counted.
Count up E.

In the case of the pattern (b), a black pixel appears until at least 30 white pixels have been exceeded. Therefore, when a black pixel appears, the flow proceeds to step S406 to start a counter BLAKE of continuous black pixels, and proceeds to step S406.
Move to 408.

Step S408 removes noise of black pixels of 2 dots or less. If black pixels appear continuously of 2 dots or less, the process proceeds to step S41.
2, the process proceeds from S416 and S417 to S420, in which the black pixel continuous counter BLAKE which may have been counted as noise is cleared and the search is continued.

If black pixels of three or more dots continue in step S408, the flow advances to step S409 to set a flag EBFK. Steps S419 and S404 to S404 as long as black pixels continue even after the flag EBFK is set.
409 is repeated to count up the black pixels. In addition. Here, if the continuation of the black pixels is interrupted, the process proceeds to step S412.

In step S412, it is determined whether or not the flag EBFL has been set. In the case of the pattern (b), since EBFL has already been set, step S
At 413, the count of the white pixel continuous counter WHITEE is started.

Next, in step S414, the number of continuous white pixels is checked. If the number of continuations is less than 5, step S
419, S404, S405, S412, S413 to S414 are repeated to count down the Y coordinate and count up the white pixels.

If the number of consecutive white pixels becomes five or more in step S414, the flag EWFL is set in step S415.
Is set. Step S after setting the flag EWFL
The flow shifts to 419, where the Y coordinate is similarly counted down, and the flow advances to step S404 via step S422.

In the determination in step S404, the flag EWF
Since L is set, the process moves to step S410, and the number of continuous black pixels BRACKE and the maximum number of continuous black pixels BRAC counted in step S407 in step S410.
Compare with KEMAX.

As a result of the comparison, the continuous black pixel number BLAKE counted on the X axis is the maximum continuous black pixel number BLA.
If the number is larger than CKEMAX, BLA is used in step S411.
CKE is replaced with BLACKEMAX, and the routine goes to Step S418.

In step S410, the number of continuous black pixels BLACKE is increased to the maximum number of continuous black pixels BLACKMAX.
If less, search for the next X axis as it is (step S
Move to 418).

In the flow of the pattern (b) on the X-axis, the continuity of white pixels from the initial value on the Y-axis is checked, and then the continuation of black pixels corresponding to the eye is determined, and the number of black pixels is counted. . Thereafter, by determining white pixels on the eyes, the vertical size of the eyes is recognized as continuous black pixels.

(3) In the case of the pattern (c) In the pattern (c), when the eye part is binarized, the iris part of the eyeball becomes white (at night, in a tunnel, etc., the image of only infrared light). Is common in some cases).

The basic processing in the case of the pattern (c) is the same as that in the case of the pattern (b). In pattern (c), the number of continuous pixels is confirmed. If white pixels in the iris of the eyeball appear during counting up of black pixels, the black pixels are counted even if black pixels in the eye portion are still continuing. It stops and starts counting up the white pixels in the iris.

Here, in step S414, the number of white pixels is 5
If the black pixel appears again before the number of the black pixels is exceeded, the count-up of the black pixel which has been stopped is restarted because the flag EWFL is not set. Thereafter, when a white pixel on the eye is confirmed, the same processing as in the pattern (b) is performed.

(4) In the case of pattern (d) The case of pattern (d) is exactly the same as the case of pattern (b), but if the value of the number of black pixels BLACKE counted up in step S407 is larger, the eye After confirming the continuous white pixels above the black pixel number B, the maximum continuous black pixel number BLACKEMAX is changed to the black pixel number B in steps S410 and S411.
Replace with LACKE and store the maximum number of continuous black pixels in that window.

(5) In the case of pattern (e) Pattern (e) is a case where continuous black pixels other than eyes, for example, eyebrows and hair, are counted. Since black pixels such as eyebrows and hair are not at least under the eyes, Y
If the search on the axis is started and the count of white pixels is repeated, and the number of continuous white pixels exceeds 30 in step S417 (the black pixels formed thereafter are not considered to be eyes), the X axis Is stopped and the process moves to the next X-axis search.

(6) Pattern (F) Pattern (F) is a case where the continuous black pixels of the hair at the left end of the window are counted. The number of continuous black pixels in the hair at the left end of the window may be very large, but no white pixels appear above the continuous black pixels at the left end of the window, and the flag EWFL is not set. Even if the number of pixels increases, the numerical value of the maximum continuous black pixel number BLACKEMAX is not replaced.

Therefore, when all the X-axis searches are completed, the maximum number of continuous black pixels in the vertical direction (Y-axis direction) of the eye is obtained. These processes can be detected regardless of whether the eyes are open or closed, and the open / closed eyes can be recognized based on the maximum number of continuous black pixels.

While the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications can be made.

[0111]

As described above, according to the first aspect of the present invention, it is determined in advance whether or not the driver is wearing glasses based on the reflection point of the infrared lamp, and even if the driver is wearing glasses. Since the configuration is such that the eye existence region can be determined, it becomes possible to accurately detect the eye existence region when the driver is wearing glasses.

Further , according to the second invention, the infrared lamp
Whether the driver is wearing glasses or not
And the presence of eyes even when the driver is not wearing glasses
It is configured so that the location area can be determined, so the driver can
Precise detection of eye presence when not wearing glasses
It became possible to do.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an embodiment of a vehicle driver state detection device according to the present invention.

FIG. 2 is a flowchart showing the operation of the vehicle driver's state detection device of FIG. 1;

FIG. 3 is an explanatory diagram of a window showing an area photographed by a camera.

FIG. 4 is a flowchart for determining the horizontal position of each of the left and right windows related to the setting A of the existence area.

FIG. 5 is a detailed flowchart of a part of the process shown in FIG. 4;

FIG. 6 is a flowchart of a vertical range setting process of a window.

FIG. 7 is an explanatory diagram of a vertical range setting process of a window.

FIG. 8 is an explanatory diagram of determining a coordinate in the Y-axis direction of an eye existing area.

FIG. 9 is an explanatory diagram showing a method for determining an eye existence area when the driver wears glasses.

FIG. 10 is an explanatory diagram showing a method of determining an eye existence area when the driver wears glasses.

FIG. 11 is a flowchart of an open / closed eye detection process.

FIG. 12 is an explanatory diagram of the flowchart in FIG. 11;

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Infrared lamp A (light emitting means) 2 Infrared lamp B 3 Camera (image input means) 4 A / D converter (binarization means) 5 Image memory 6 Glasses presence / absence determination circuit (glasses presence / absence determination means) 7 Eyeball present position Regulatory circuit (eye existence area setting means) 8 Open / closed eye determination circuit 9 Drowsiness determination circuit

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int. Cl. ⁷ , DB name) G06T 1/00 G06T ⁷ /00-7/60

Claims (2)

(57) [Claims] 1. An image input means for inputting a driver's face image as face image data; a light emitting means for irradiating an invisible light beam on a face of the driver; Eyeglass presence / absence determining means for determining the presence or absence of glasses by image recognition of the reflection point of the light emitting means, and light emission reflected on the glasses of the glasses worn by the driver when the presence / absence of glasses is determined by the glasses presence / absence determination means. An eye presence area setting means for setting an eye existence area based on a reflection point of the means; an open / closed eye detection means for detecting a driver's open / closed eye in the area; an open / closed eye signal from the open / closed eye detection means And a driver status determining means for determining the status of the driver based on the information. Wherein when it is determined that without glasses by the spectacle existence determining means comprises a second existing region setting means of the eye for determining an area including the eye of the face image data, the second eye Existence region setting means for binarizing an input image sent from the image input means, and white pixels for scanning the image in the left-right direction and counting the number of continuous white pixels in the binarization means Number counting means, a stable part detecting means for detecting a stable part at the width direction end of the face image from the change amount of the number of pixels, and a contour of the face image from the presence or absence of the stable part and the change amount of the continuous pixel number Continuity continuity determining means for determining continuity, Face edge specifying means for specifying the edge of the face based on the determination, and Eye existence area from the face edge specified by the face edge specifying means Eyeball left / right area determination to determine lateral position Means, a black area detection means for detecting a black area from below in the vertical direction in the determined horizontal existence area of the eye, and a vertical position of the eye existence area based on the detected black area. The driver's state detection device according to claim 1 , wherein the eye existence region is set by the eyeball vertical region determination means to be determined.