JPH0632154A - Driver's condition detector - Google Patents

Abstract

PURPOSE:To surely detect an eyeball whether it is in open eye condition or close eye condition and shorten the processing time for judgment of an open or close eye by providing a driver's condition judging means for judging the condition of the driver, based on the pattern of an open/close eye detected by an open/close eye detecting means. CONSTITUTION:This detector is provided with an image input means CL1 for inputting the face image of a driver, a binarizing means CL2 for binarizing the face image being inputted from the image input means, an eyeball existence area setting means CL3 for seeking the eyeball existence area within the binarized image, and an eyeball detecting means CL4 for detecting the eyeball, based on the continuous black picture element in longitudinal direction within the eyeball existence area. Furthermore, it is provided with an open/close eye detecting means CL5 for detecting the open/close eye of a driver, based on the number of continuous black picture elements of an eyeball being detected, and a driver's condition judging means CL6 judges the condition of a driver, based on the pattern of the open/close eye being detected by this open/close eye detecting means CL5.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a driver's state detecting device for detecting a dozing state or the like from the state of the driver's eyes opening and closing.

[0002]

2. Description of the Related Art As a conventional vehicle driver state detecting device, there is one disclosed in, for example, Japanese Unexamined Patent Publication No. 3-194661. This is to detect the iris part of the eye in a certain range of the driver's eye using the binarized face image of the driver as a detection region, and open and close the driver from the detection result of the iris part. It is configured to detect the eyes and detect the driver's state, and it can be used to detect the driver's dozing or aside.

[0003]

However, in the above-described conventional vehicle driver state detection device, when determining the open / closed eyes of the driver, a black circular region is detected in the detection region, and this is designated as the iris of the eye. Since the eye open state and the eye closed state are determined based on the size of the area after the determination, the binarized image clearly shows that the circular area is clear as shown in the eye open state of Example A of FIG. When the identification and detection are performed, the determination can be performed accurately, but when it is difficult to identify the black circular area on the binarized image as in Examples B to D in FIG.
This may be erroneously determined as eyebrows or the like.

Further, when detecting the iris portion, the MAX value of the brightness difference is obtained by a radial rectangular filter, and the MAX value is calculated.
Since the coordinate of the point that takes a value is determined as the iris center and the open / closed eye is determined from the area of the black pixels around the iris center, there is a problem that the calculation processing time for the determination also takes a long time. Therefore, in view of such conventional problems, the present invention can accurately detect the open / closed eye situation even when the image state is poor, and can quickly determine the driver's state such as the dozing state. An object of the present invention is to provide a state detection device.

[0005]

In order to solve the above-mentioned problems, according to the present invention, as shown in FIG. 1, an image input means CL1 for inputting a face image of a driver, and an image input means CL1 are inputted. Based on the binarizing means CL2 for binarizing the face image, the eyeball existing area setting means CL3 for defining the eyeball existing area in the binarized image, and the continuous black pixels in the vertical direction within the eyeball existing area. Eyeball detecting means CL4 for detecting an eyeball
And an open / closed eye detection means CL5 for detecting the open / closed eyes of the driver based on the number of the continuous black pixels of the detected eyeballs, and the driver's open / closed eye pattern detected by the open / closed eye detection means. The driver state determination means CL6 for determining the state is provided.

[0006]

Since the eyeball is detected by vertically scanning the eyeball existing region of the binarized image and searching for continuous black pixels, the eyeball is surely detected regardless of whether the eye is open or closed. . Since the open / closed eye is discriminated by the number of continuous black pixels of the eyeball, the state is detected in a short time.

[0007]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the drawings. FIG. 2 is a block diagram showing the configuration of the embodiment of the present invention.
An infrared strobe 1 for irradiating the driver's face and a CCD TV camera 3 as an image input means CL1 for photographing a face portion irradiated by infrared rays of the infrared strobe 1
Is installed on an instrument panel (not shown) of the vehicle. The infrared strobe 1 and the TV camera 3 are connected to a timing command circuit 5, and the timing matching between strobe emission and image input of the TV camera is controlled. That is, when the timing command circuit 5 outputs a strobe light emission command to the infrared strobe 1, the infrared strobe 1 emits light to illuminate the driver's face portion, and at the same time, an image input command is output to the TV camera 3. It is designed to photograph the face area illuminated by infrared rays.

The input image of the TV camera 3 is, as shown in FIG. 3, 520 pixels in the horizontal (X) direction and 500 in the vertical (Y) direction.
It is composed of pixels, and the angle of view is adjusted so that the face part is almost full in the vertical direction. An image memory 9 is connected to the TV camera 3 via an A / D converter 7 which converts a captured input image into a digital amount, and stores input image data. The image memory 9 includes an eyeball presence position defining circuit 11
Are connected, and the existence position area of the eyeball is defined based on the input image data stored in the image memory 9. Further, connected to the eyeball existing position defining circuit 11, the eyeball detection circuit 13 processes the data in the eyeball existing position region of the image data of the image memory 9 to detect the eyeball. In the next opening / closing eye determination circuit 15, it is determined from the eyeball data detected above whether the eye is open or closed. The presence / absence of the driver's dozing state is determined by the dozing determination circuit 17 based on the state of the open / closed eyes. The determination result is output to the alarm circuit 17 or the like for notification.

The main flow of the overall operation in the above configuration will be described with reference to FIG. First, when the ignition switch of the vehicle is turned on, it is confirmed that the driver is on the driver's seat, and the entire system is activated, the timer is started in step S1. Then, in step S2, when the face portion of the driver is photographed by the TV camera 3 in synchronization with the emission of the infrared strobe 1, this image is A / D conversion circuit 7
Is converted into a digital signal and stored in the image memory 9 as input image data for one frame in step S3. Here, the video signal is converted into digital data of 256 gradations of 0 to 255, and the white portion is "255" and the black portion is "0". Steps S2 and S3 form the image input means CL1 of the invention.

The input image data stored in the image memory 9 is fetched by the eyeball existing position defining circuit 11 and, in step S4, a binary value is set at a predetermined threshold so that the lightness and darkness of the face portion can be made clear and the eyeball can be extracted. Then, the binarized image J (X, Y) is obtained. Binary image J (X, Y)
Based on the above, in the next step S5 and step S6, or in step S5 ′ and step S6 ′, the lateral (X direction) width and the vertical direction (Y direction) of the left and right eyeball existence regions shown by the broken lines in FIG. ) Width is determined. Step S4 constitutes the binarizing means CL2 of the invention. Further, steps S5 and S5 'constitute lateral width setting means, and S6 and S6' constitute vertical width setting means, and both constitute eyeball existing area setting means CL3.

When the eyeball existing area is determined by the eyeball existing position defining circuit 11, then in step S7, the eyeball detecting circuit 13 counts the continuous black pixels in the vertical direction within the eyeball existing area, and the eyeball is detected. Is detected and determined. Thereafter, in step 8, a threshold value is set in the open / closed eye determination circuit 15, and the eyeball data detected above is compared with the threshold value to detect the open / closed eye. The count of the timer started in step S1 is used for threshold setting here, as described later.

When the eyeball data of the left eyeball existing area has a predetermined large change with the open / closed eye, the threshold value is determined based on the eyeball data of the left eyeball existing area. After the setting and the open / closed eye determination, the process proceeds to step S9. When the change in the eyeball data in the left eyeball area is small, it is then compared with the eyeball data in the right eyeball area, and the threshold value setting and opening / closing is performed using the data in the eyeball area with the larger change. An eye judgment is made. Then, in step S9, the driver's dozing judgment is made by the dozing judgment circuit 17 based on the pattern of the open / closed eyes detected above. Step S7 constitutes the eyeball detection means CL4 of the invention, steps S1 and S8 constitute the open / closed eye detection means CL5, and step S8 constitutes the driver state determination means CL6.

Details of steps S5 and S5 'are shown in FIG.
This is shown in the flow chart of FIG. First, step S10
At 0, the search start point is initialized and the scanning Y coordinate in the height direction is set to 40. This shortens the processing time on the assumption that the maximum width of the face does not exist within the range of Y = 0 to 40. In step S101, as shown in FIG. 7, the center of the image (X coordinate = 250)
Is set as the search start line, and the number of continuous white pixels WXCL and WXC is separately set on the horizontal scanning line on the Y coordinate.
R is counted, and in step S102, the left and right X coordinates XLM and XRM when the number of continuous white pixels is maximum are stored. Steps S100 to S102 constitute the continuous white pixel number counting means of the invention, and a more detailed description of this processing will be given later with reference to FIGS. 8 and 9.

Subsequently, in step S103, it is checked whether or not the sum of the numbers of continuous white pixels on the left and right is larger than 200. When the continuation of the white data is interrupted by the presence of hair, eyebrows, eyeballs, etc., and the above sum is 200 or less,
In step S104, the left stability counter WLCON, the right stability counter WRCON, and the left stability flag STFLGL.
Then, the right stability flag STFLGR is cleared, the process returns to step S101, and the next line is scanned.

If the sum of the numbers of continuous white pixels on the left and right is greater than 200, the process proceeds to step S105, and it is checked whether or not the left end point detection flag OKFGL is set. Move to the right end search process of. Left end point detection flag OKFGL
Is 0, the process proceeds to step S106, where the difference between WXCL and the number MAEL of consecutive scan lines immediately before is 10
It is checked whether it is less than 10 and if it is less than 10, step 1
At 07, the left stability counter WLCON is incremented. At this time, the current scan line is regarded as a stable candidate part having a small difference from the number of consecutive previous scan lines, and the step 1
At 08 and 109, XL when the number WXCL of continuous white pixels is maximum is stored as X1, and XL when the number WXCL is minimum is stored as X2.

In step S110, the left stability counter W
It is checked whether LCON exceeds 10 and the difference between X1 and X2 is less than 30. If this condition is satisfied, the left stability flag STFGL is set in step 111, and the left continuous white color is detected in step 112. The number of pixels WXCL is newly M
AEL.

In step S106, the number of continuous white pixels WXC
When the difference between L and the number of consecutive MAELs in the previous scan is 10 or more, it is determined that the continuity of the image contour line may have been lost, and the process proceeds to steps S113 to S119. Here, it is determined whether the continuous line number has changed significantly due to the presence of the eyebrows or the like, and whether the contour line is interrupted or not is determined by the change in the continuous pixel number and the presence of the stable portion.

First, in step S113, the left stability flag S
TFLGL is checked, and if there is a stable portion before the number of continuous white pixels changes significantly and STFLGL is set, it is checked in step S114 whether the number of continuous pixels has increased. If the number of consecutive pixels is increasing, it is determined that the contour line is interrupted, the process proceeds to step S115, and the X coordinate X1 stored in step S108 of the previous flow is set as the left edge of the face.

If there is no stable portion before the number of continuous white pixels greatly changes in step 113, or if there is a stable portion and the number of continuous pixels decreases in step 114, the contour line is interrupted. It is conceivable that the scanning has shifted from the portion to the portion having the contour line, or the scanning of the eyebrows, eyes, spectacles, or the like. Therefore, at this time, in step S116, it is checked whether or not the number of continuous white pixels of the current scan is reduced by 50 or more from the number of continuous white pixels of the previous scan.

In the case of a decrease of 50 or more, it is assumed that the portion where the contour line is interrupted is moved to the scanning of the portion where the contour line is present,
Proceeding to step 118, the left end point XL of this scan is set as the left end of the face. When the left edge of the face is thus determined in step S115 or step S118, the left edge detection flag OKFGL is set in the next step S119, and then the process proceeds to step S120. If the number of consecutive white pixels of this time is not significantly smaller than the number of consecutive pixels of the previous time in step 116, it is considered to be an eyebrow, an eye, a spectacle portion, or a portion having a large shadow, and the process proceeds to step S117, where X1, X2 , WLCON, and STFLGL are cleared, the process proceeds to step S112.

In steps S120 to S121, similarly to steps S105 to S119 for the left side of the image, it is determined whether the contour line is broken or the right edge of the face is determined for the right side of the image.

The above processes are steps S122 and S1.
23, the left and right end point detection flags OKFGL,
Both OKFGR are set, or otherwise repeated until the scanning of the preset search range in the vertical direction (Y direction) is completed. When the end point detection flag is set, the lateral width setting of the eyeball existing region is completed with X1 and X2 as the end points on the left side and XX1 and XX2 as the end points on the right side.

When the scanning of the search range in the Y direction is completed, it is checked in steps S124 and S126 whether or not the left and right end point detection flags are set, and if they are not set, in step S12.
5. In S127, the left and right X coordinates XLM, XRM when the number of continuous white pixels in the entire scan stored in the previous step S102 is the maximum are determined as the left and right ends of the face. When the left and right edges of the face are determined, the lateral end points of the eyeball existing area are determined by the following equation. X-axis center = XC = XLM + ((XRM-XLM) /
2) Left-side X coordinate of the left-eye-eye-ball-existing region = X1 = XLM Right-side X-coordinate of the left-eye-eye-ball-existing region = X2 = XC-25 Left-side X-coordinate of the right-eye-eye-ball-existing region = XX1 = XC + 25 Right X coordinate = XX2 = XRM Steps S103 to S112 constitute the stable portion detecting means of the invention, steps S106, S113 and S114 constitute the contour line continuity determining means, and steps S115 to S127 constitute the face edge specifying means. . However, step S12
1 includes a stable part detecting means and a contour line continuity judging means for the right side.

Counting the number of continuous white pixels and the left and right X coordinates XLM and X in steps S101 and S102.
The RM storage is performed according to the flow shown in FIGS. As described above, in step S100, the vertical scanning Y coordinate is 4
If it is set to 0, then in step S202, the X coordinate value 250 of the horizontal search scan start line is set to the left search X coordinate XL and the right search X coordinate XR. This X coordinate value (= 250), as shown in FIG. 7, represents a line that is surely present in the face portion if the vehicle driver is present within the photographing angle of view of the TV camera.

Next, in step S203, it is checked whether or not the right scanning end flag OKR is set. If the right scanning end flag OKR is set, the process proceeds to step S211 and the subsequent search for the left edge of the face. If OKR is not set, step S
At 204, it is checked if pixel J (XR, Y) is white,
When it is white, the right continuous white pixel counter WXCR and the search X coordinate XR are respectively incremented in steps S205 and S206. If the pixel J (XR, Y) is not white in the check in step S204, step S2
After the routine proceeds to 07 and the right side scanning end flag OKR is set, in step S208, the maximum value XRM of the right side end point which has been stored so far is compared with the present end point XR. When XR is larger, that is, on the right side, XR is set as a new right end point XRM in step S209.

Next, in steps S211 to S217, the same processing as steps S203 to S209 is performed on the left side of the face. The difference with the processing on the right side is that
In step S214, the search X coordinate XL is counted down, and in step S217, the left-side storage endpoint XLM and the current endpoint XL, whichever is smaller, that is, the one on the left side is the new left-end point XLM.

If the left and right ends of one scanning line are detected and it is determined in step S221 that both scanning end flags OKL and OKR have been set, step S222 is performed.
After this flag is cleared at, the Y coordinate value of the scanning line is incremented at step S223, and this processing ends.

As described above, even when external light is radiated from one side of the face and the continuity of the contour line of the face image is lost, the face contour line is determined from the presence or absence of the stable portion and the change in the number of continuous pixels. The left and right edges of the face are identified by confirming the continuity of. Also,
In the binarized image, the left and right edges of the face are detected by scanning from the center of the image to the left and right and the white pixels are continuous, so the face can be captured without fail even if the background is not white, and proper detection can be performed. it can.

Steps S6 and S6 'of the main flow
The process of setting the vertical width of the eyeball existence region in 2 is divided into two processes, a search process for two black regions and a process for searching for the presence or absence of glasses. In the search processing of two black regions, as to the left eye, as shown in FIG. 10, in order to avoid detection of the black portion of the nostril, from the right X coordinate X2 of the left eyeball existing region to the left side of 10 dots, that is, X.
Starting from 2-10, the Y coordinate YL of the search start in the range from this position to X2-90 in the lateral direction (X direction)
From 0 to 0 is scanned vertically upward, and this is executed at intervals of 4 dots in the horizontal direction. The Y coordinate YL is set below the scan line that determines the left and right edges of the face. Also for the right eye, similarly, 10 dots to the right of the left X coordinate XX1 of the right eyeball existence area, that is, XX1 +10 is the starting point, and the range from this position to the lateral direction (X direction) XX1 +90 is from the Y coordinate YL. Scanning is performed vertically between 0, and this is executed at intervals of 4 dots in the horizontal direction.

The detailed flow of the process of setting the vertical width of the eyeball existing region is shown in FIGS. 11, 12 and 13. First, in step S301, the memory variable BY1M of the maximum value of the Y coordinate indicating the highest point of each of the first and second black areas.
The values of AX and BY2MAX are cleared to 0, and the X-direction detection range defining counter XCHECK is set to X2-10,
Also, the Y-direction search range defining counter YCHECK is set to YL.
Is initialized to.

Then, in step S302, it is checked whether or not the search range defining counter XCHECK in the X direction is X2 -90 or less. This is to determine whether or not all searches have been performed in the X direction. While the search in all X directions has not been completed, the process proceeds to step S303, and a flag FL1 indicating that the first black area is detected, a continuous black pixel counter BLACK, a continuous white pixel counter WHITE, 1
A flag WHITEFL indicating that the interval between the second black area and the second black area is 10 dots or more and the maximum value storage buffers BY1 and BY2 of the first black area and the second black area are cleared.

Then, in step S304, it is checked whether or not the search pixel is black. If black, the continuous white pixel counter WHITE is cleared in step S305, and the continuous black pixel counter BLACK is counted up in step S306. Then, in step S307,
It is checked whether or not the count value of the continuous black pixel counter BLACK is 1. Here, it is determined whether or not the black pixel has been detected for the first time. When the count value is 1, in step S308, the search range defining counter YCH in the Y direction
The current Y coordinate counted by ECK is the lowest point Y of the black area.
It is stored in SETY as a coordinate candidate. That is, the Y coordinate indicated by "1" in FIG. 10 is stored.

Next, in step S309, it is checked whether the count value of the continuous black pixel counter BLACK is 2 or more. When the count value is 2 or more, the routine proceeds to step S310, where it is checked whether or not the first black area detection flag FL1 is set. If the flag FL1 is not set, the process proceeds to step S311 and 1
The value of SETY is substituted and stored in the maximum value storage buffer BY1 of the th black area, and the flag FL1 is set. Then, in step S328, the Y coordinate YC is counted down, and the process moves to the search for the pixel immediately above on the scanning line.

If the flag FL1 is set in step S310, the flow advances to step S312 to check whether the flag WHITEFL is set or not, and check the interval between the first black area and the second black area. Is 10
It is checked whether there are more dots. Then, when the flag WHITEFL is set, it means that the second black area has been detected. Therefore, in step S313, SET is set in the maximum value storage buffer BY2 of the second black area.
The value of Y is substituted and stored. That is, the Y coordinate indicated by "2" in FIG. 10 is stored.

In step S312, the flag WHITEFL is set.
If is not set, the gap between the first black area and the second black area is narrow and the difference between the two is not clear,
In step S314, it is checked whether the number of consecutive black pixels exceeds 50 dots. Continuous black pixel counter B
When the ACK count value exceeds 50, it is determined that the hair is detected, and the buffer BY2 is cleared in step S315. If the number of dots does not exceed 50 dots, the process advances to step S328 to count down the Y-coordinate YC and search for a pixel one level above.

If the search pixel is white in step S304, the process advances to step S316 to clear the continuous black pixel counter BLACK, and then to step S304.
At 317, it is checked whether the first black area detection flag FL1 is set. Then, the flag FL1
When is not set, no black area has been detected yet, and therefore, the process moves to step S328 to search for the pixel one level above. If the flag FL1 is set, the process proceeds to step S318, and the continuous white pixel counter WHITE is incremented.

In the next step S319, the number of white pixels is 10
It is checked whether or not dots are consecutive, and if 10 dots or more are consecutive, it is determined that the area between the eye and the eyebrow or the area between the eyeglass frame and the eye is detected, and the flag WHITEFL is set in step S320. Further, in the next step S321, it is checked whether or not the white pixels are continuous for 80 dots or more. If the white pixels are continuous for 80 dots or more, it is determined that the eyebrows are not detected and the forehead is detected, and step S32 is performed.
At 2, the maximum value storage buffer BY2 of the second black area is cleared.

When the white pixels are not continuous for 80 dots or more, or when the white pixels are not continuous for 10 dots or more in step S319, the process proceeds to step S328, and the Y coordinate YC is counted down to move up one line on the scanning line. Move to search for pixel.

On the other hand, when the values of the buffers BY1 and BY2 are obtained as the candidate points of the first and second black areas, respectively, in steps S323 and S324, the value of the buffer BY1 and the first value stored so far are stored. The maximum value BY1MAX indicating the lowest point of the black area is compared, and the larger value is stored as BY1MAX. Then, in steps S325 and S326, the value of the buffer BY2 and the maximum value BY2M of the second black area stored so far are stored.
AX is compared and the larger value is stored as BY2MAX. For example, in FIG. 10, the Y coordinate of "1" at the center is stored as BY1MAX, and the Y coordinate of "2" on the right side is stored as BY2MAX. In this way, the lowest point BY1 of the first black area
MAX and the lowest point BY2MAX of the second black area are determined. Steps S301 to S326 form the black area detecting means of the invention.

Next, the process of searching for the presence / absence of glasses will be described. Here, in the above black area search, at least 1
When the scan line in which the th black area is detected occurs, a scan line extending in the horizontal direction is set for the area covering the nose portion in the center of the screen each time the scan line, that is, the vertical search is completed. Then, the presence or absence of the black area representing the glasses is searched.

First, in step S329, it is checked whether or not a value is stored in the maximum value storage buffer BY2 of the second black area, and the Y coordinate BYH for spectacles detection is obtained according to the situation. That is, when the maximum value storage buffer BY2 of the second black area has no value and only the maximum value storage buffer BY1 of the first black area has a value, BYH = BY1 + 10 is set in step 330. Maximum value storage buffer BY2 of the second black area
If there is a value in step S331, BYH =
(BY1 + BY2) / 2. In this equation, BYH is the midpoint between BY1 and BY2, but it is arbitrary as long as it is a point located between BY1 and BY2.

Thereafter, in step S332, a black pixel counter BLACKX for counting the number of black pixels.
Is cleared, and initial values XC = X2 and YC = BYH are set to the pixel coordinates in steps S333 and S334. Next, in step S335, it is checked whether the pixel J (XC, YC) is black. In the case of black, after the black pixel counter BLACKX is counted up in step 336,
If it is not black, the process proceeds to step 337 as it is.
In step 337, the pixel coordinates are counted up from XC = X2 in the left eyeball existing area. In step S338, as shown in FIG. 14, it is checked whether or not the pixel J has been searched until it exceeds XX1 in the horizontal direction (X direction).
The pixel search on the horizontal scan line is repeated until XC exceeds XX1.

When the search for one scanning line is completed, in step S339, the black pixel counter BLACK.
It is checked whether the value of X is less than 3, and if the value of the black pixel counter BLACKX is less than 3, it is determined that the frame of the central portion of the glasses has been detected, and step S34
At 0, the glassesless counter MEGOFF is incremented.

When this ends with the end of the search for all the vertical scanning lines within the X-axis direction search range, in step S341, the glasses-free counter ME.
It is checked whether the value of GOFF exceeds 5. If the value of the glasses-free counter MEGOFF is greater than 5, it is determined that the glasses are not worn, and step S342 is performed.
Proceed to. In step S342, the upper and lower Y coordinates YT and YB defining the vertical width of the eyeball existing region are YT = BY1MAX-40 YB = BY1MAX + 10 with reference to the lowest point BY1MAX of the Y coordinate of the first detected black region. Is set to.

If the value of the glasses-free counter MEGOFF is smaller than 5, it is determined that glasses are worn, and the process proceeds to step S343. Here, based on the lowest point BY2MAX of the Y coordinate of the second detected black region,
The Y coordinate of the eyeball existing region is set to YT = BY2MAX-40 YB = BY2MAX + 10.

The above processing is similarly performed when setting the vertical width of the right eyeball existing region. At that time, in the search process for the presence / absence of glasses, the initial value of the search start is set to XC = XX1, and in step 337, the count down from XC = XX1 is performed, and the pixel search is repeated in the opposite direction until XC falls below X2.

As described above, in the vertical width setting, a range having two black areas is searched for, and it is only necessary to perform vertical scanning every four dots horizontally 20 times. Settings are made in a short time. Furthermore, since the search is focused on the lower part of the eyebrow that is not affected by hair or the like and has little image change, the eyeball existing region can be accurately specified even when the user wears a hat.

Next, the details of the eyeball detection in step S7 in the eyeball existing area thus set will be described. Here, for the eyeball existing area on the left side, X is set in the lateral direction (X direction) from the right side X coordinate X2 of the eyeball existing area.
Within the range of 1 up to YB-50 in the vertical direction (Y direction) from the Y coordinate YB at the start of the search, this is repeated for each dot in the horizontal direction. As for the right eyeball existence area, the left X coordinates XX1 to XX2 of the eyeball existence area are also included.
The range is up to and the same search is performed. The process of this flow is shown in FIGS.

First, in step S401, the prescribed counter XECHECK of the search range in the X direction is set to X2, and Y is set to Y.
The direction search range defining counter YECHECK is initialized to YB, and the maximum continuous black pixel number BLACKEMAX is cleared. Next, in step S402, it is checked whether or not the X range search range defining counter XECHECK is smaller than X1. This is to determine whether or not the search has been completed for all X directions. While the search in all X directions is not completed, the process proceeds to step S403,
A flag EBFL indicating that a continuous black pixel has been detected before a new X-axis search, a flag EWFL indicating that a continuous white pixel has been detected, and a continuous black pixel counter BLACK.
E, the continuous white pixel counter WHITE is cleared.

In the next step S421, it is checked whether the search initial pixel on the Y-axis is white. If the pixel is black, it is assumed that hair or the like is inside the eyeball existing area, the Y-direction search on that line is stopped, and the process proceeds to the next X-axis search via step S418. When the search initial pixel is white, the routine proceeds to step 404, where it is checked whether the continuous white pixel flag is set. If the flag EWFL is not set in the initial setting, the process proceeds to step 412 through a check step S405 of whether the corresponding dot is a black pixel or a white pixel, and it is checked whether or not the flag EBFL of continuous black pixels is set. .

When the flag EBFL is not set, the continuous white pixel counter WHIT is determined in step S416.
EE is counted up. Then, until the number of continuous white pixels exceeds 30 by this count value, the continuous black pixel counter BLACK is cleared every time while scanning is continued to the dots in the upper Y direction in steps S417, S420, and S419, and then step 418 is performed. The process moves to the search for the next line in the X-axis direction, that is, the horizontally adjacent line.

Moving to a new line, step S405
If the dot is a black pixel, the process advances to step S406 to clear the continuous white pixel counter WHITE,
Further, in step S407, the continuous black pixel counter BLA
CKE is counted up. After this, step S4
In S08 and S409, if the count value of BLACK exceeds 2, the flag EBFL is set and the pixel moves up one pixel. This is intended to remove noise from black pixels of 2 dots or less.

When the flag EBFL is set in step S412, the continuous white pixel counter WHITEE is incremented in step S413. After that, in steps S414 and S415, the WHITE
When the count value of E becomes 5 or more, the flag EWFL
Is set, and if it is less than 5, the process proceeds to step S419 without being set.

If the flag EWFL is set in step 404, the process proceeds to step S410, and the count value of the continuous black pixel counter BLACKE is compared with the maximum continuous black pixel number BLACKEMAX. Where X
BLACK counted on the current line of the axis is B
When it is larger than LACKEMAX, step S411
Then the value of BLACKE is put into BLACKEMAX,
If BLACK is less than BLACKEMAX, BLACKEMAX is left as it is and the process moves to the next horizontal search.

Next, particularly step S in the above flow.
The operation after 404 will be described for the case of scanning lines a to f in the eyeball existing region illustrated in FIG. (1) In the case of the scanning line “a” Since the scanning line “a” is the first scanning line in the eyeball existing region, the flag EWFL of the continuous white pixels is not set, and thus steps S404 to S405.
Then, it is checked whether the corresponding dot is a black pixel or a white pixel. Since the pixel is white here, the process proceeds to step S412. Furthermore, the flag EBF of continuous black pixels
Since L is not set either, the flow advances from step S412 to step S416 to set the continuous white pixel counter WHI.
TEE is started.

Next, the process returns from step S417 to step S419 to step 404 until the number of continuous white pixels exceeds 30, and the Y coordinate is repeatedly counted down. When the number of continuous white pixels exceeds 30, even if the Y coordinate is not scanned up to the upper edge of the eyeball existing region, the process proceeds from step 417 to step S418 to search for the next line on the X axis. In step S417, black pixels that appear after a considerable number of consecutive white pixels are unlikely to be eyes, and may be eyebrows or hair. Therefore, 30 pixels are set as the upper limit so as not to be counted. In addition,
The maximum number of continuous black pixels is BLA in the scan line search for "a".
This value is never updated while CKEMAX = 0 remains unchanged.

(2) Case of scanning line "B" When the scanning line "B" is searched for as a new X-axis line, the flags EBFL and EWFL and the counters BLACKE and WHITEE are cleared in step S403. It proceeds to step S404. Therefore, as in the case of the scanning line "a", steps S405 and S4 are performed.
12. After step S416, the process proceeds to step S417, and the search for the scanning line is repeated while counting up WHITE.

On this scanning line, the number of continuous white pixels is 30.
A black pixel appears before exceeding, and correspondingly, the process moves from step 405 to step S406, where the continuous white pixel counter WHITEE is cleared. In the next step S407, the count-up of the continuous black pixel counter BLACKE is started, and this count-up is continued as long as the black pixels are continuous. During this period, step S4
In S08 and S409, when three or more black pixels are consecutive, the flag EBFL is set because it is not noise.

When the continuity of black pixels is interrupted, the process moves from step S405 to step S412. Since the flag EBFL is set here, step S41
3, the continuous white pixel counter WHITEE is started, and while counting down the Y coordinate in S419, that is, above the scanning line, while the white pixels continue, WHITEEE
The count-up is repeated. Then, when the number of consecutive white pixels becomes 5 or more, steps S414 and S41.
At 5, the flag EWFL is set.

As described above, after the start of the scanning line in the vertical direction (Y direction) in the eyeball presence region, after the continuation of white pixels,
A series of black pixels corresponding to the eye area was observed, and then
By confirming a predetermined number of continuous white pixels on the eye, the number of continuous black pixels is obtained as the size of the eye in the vertical direction. After the flag EWFL is set, when the process moves to the next Y coordinate through step S419, this time the process proceeds from step S404 to step S410. In step S410, the number of consecutive black pixels of BLACK counted in step S407 and the maximum number of consecutive black pixels BLACKEMAX are compared.

If continuous black pixels appear for the first time on the scanning line "B", since BLACKEMAX = 0, the process advances to step S411, and the count value of BLACKE is put into BLACKEMAX.

(3) In the case of the scanning line "C" The scanning line "C" is a phenomenon that is often seen when binarizing an image when the ambient light is only infrared rays of the strobe, such as at night or in a tunnel, and is an iris. It is a line that passes through the white part. The basic flow is the same as in the case of the scanning line "b", but here, continuous black pixels are confirmed, and during the count-up of the continuous black pixel counter BLACKE, the black pixels in the eye part still continue. The count of BLACKE is interrupted by the white pixels in the iris,
From 405, the process proceeds to S412 and S413.

Here, the continuous white pixel counter WHITEE starts counting up for the white pixels in the iris portion, but since the number of white pixels in the iris portion is changed to black pixels before the number of white pixels exceeds 5, the flag EWFL is set. Without steps S419 to S405 to S40
Returning to 7, the counting up of the interrupted continuous black pixels is restarted. After that, after the white pixel is confirmed on the eye and the white pixel is confirmed, the same processing as that for the scanning line "b" is performed.

(4) In the case of scanning line "d" The flow is exactly the same as in the case of scanning line "b", but the number of consecutive black pixels increases and the count value of BLACKE in step S407 is large. As a result, after the continuous white pixels on the eyes are confirmed, steps S410 and S411 are performed.
In, the value of BLACK is the maximum number of consecutive black pixels BL
It is stored as the value of ACKEMAX.

(5) In the case of scanning line "e" The scanning line "e" is a line that does not pass through the eye part and cuts the black part such as eyebrows and hair. Black pixels such as eyebrows and hair are not at least under the eyes, and when the search on the scanning line is started, white pixels are repeatedly detected. As a result, since the number of continuous white pixels exceeds 30 in step S417, the search on this scan line is stopped and the process moves to the search for the next horizontally adjacent scan line.

(6) In the case of the scanning line "F": This scanning line passes through the hair portion at the left end of the eyeball existing region. Here, the number of continuous black pixels is very large, and continuous white pixels do not appear above the continuous black pixels. Therefore, after the flag EBFL is set, steps S405 to S405 are performed.
Since there is no branch to 412, the flag EWFL is not set either. As a result, even if the number of continuous black pixels due to hair increases, the process does not proceed from step S404 to S410, and the maximum number of continuous black pixels BLACKEMA
The value of X is not replaced. Therefore, continuous black pixels on the scanning line are not erroneously recognized as an eyeball.

As described above, when the search for all the scanning lines is completed over the entire width of the eyeball existing region, the maximum number of black pixels BLACKEMAX becomes the vertical direction of the eyeball (Y
Direction). B in the above flow
LACKEMAX is detected regardless of whether the eyes are open or closed.

Next, FIG. 18 shows a flow of drowsiness detection using the eyeball data detected by the above processing. This corresponds to steps S8 and S9 of the main flow.
As described above, after the timer is started in step S1, the maximum continuous black pixel number BLACKEMAX is obtained as eyeball data for each frame of the image in step S7, particularly step S411 among them. In response to this, in step S501, the maximum value MAX and the minimum value M of BLACKEMAX are obtained each time new frame data is obtained.
IN is updated and stored. After confirming that 5 minutes have elapsed after the timer was started in step S502,
At 503, the threshold value of the open / closed eye is set based on the MAX value and the MIN value of the maximum number of continuous black pixels. Note that the above-mentioned time of 5 minutes is set as a time for surely capturing an image in an eye-closed state by blinking in consideration of the timing of capturing a face image. And by comparison with the above threshold,
Whether the eye is open or closed is detected in step S504, and the dozing state is determined in step S505 based on the pattern of the open / closed eye.

The open / closed eye detection process is repeatedly executed during the driving of the vehicle. In step S505, it is determined that the subject is in a dozing state when the closed eyes are continuously detected for a predetermined time or more. That is, as shown in FIG. 19, a face image of one frame is photographed and input at intervals indicated by black dots on the time axis, and as a result of eye open / closed detection for each frame, the eye closed state continues for a predetermined number, for example, 3 frames. When the above continues, it is determined that the driver is in a dozing state.

According to the present embodiment, the eyeball is detected by scanning the inside of the eyeball region in the vertical direction and searching for continuous black pixels. Therefore, the eyeball can be reliably detected regardless of whether the eyeball is open or closed. Is detected. Then, when setting the eyeball presence area, since the end points of the lateral width of the face are determined separately for the lateral width, even if a shadow is formed in the infrared strobe irradiation image depending on the driver's face orientation, either the left or right The region where the eyeball is present is reliably set and its influence is eliminated. Also, when setting the vertical width, the range having two black areas is searched, and the vertical scanning is performed at intervals of several dots in the horizontal direction.
The setting of the eyeball existing area can be made in a short time with a slight scan. Furthermore, since the search is narrowed down to the lower part of the eyebrow that is not affected by hair and other factors and has little image change, the eyeball presence area can be accurately identified even when the person is wearing a hat, and the accuracy of drowsiness determination can be determined. Is high. Moreover, as described above, since the right and left sides of the face are independently searched, even when the edge on one side of the face is not detected due to the influence of outside light, the edge on the opposite side is detected. Therefore, one of the left and right eyeball existence regions is surely set.

[0071]

As described above, according to the present invention, the eyeball detecting means for detecting the eyeball by scanning the inside of the eyeball existing region of the binarized image in the vertical direction to search for continuous black pixels is provided. Since the detection means discriminates the open / closed eyes of the driver based on the number of consecutive black pixels of the detected eyeball, it has an effect of surely detecting the eyeball regardless of whether the eyeball is open or closed. Moreover, the processing time for discriminating the open / closed eye is significantly shortened as compared with the case where the iris circular portion is obtained and the area is discriminated.

Further, in particular, the lateral position of the eyeball existing region is laterally scanned by the lateral width setting means having a continuous white pixel number counting means, a stable part detecting means, a contour line continuity determining means and a face edge specifying means. With the lateral end of, the eye part can be surely included. Then, when the vertical width is set by the vertical width setting means based on the two black areas detected by the black area detecting means, the black area detection can be performed with a slight scan, so that the setting of the eyeball existing area is short. Done Furthermore, since the hair part and the like can be excluded in the process, there is an advantage that the eyeball existing region can be accurately specified. Further, if the open / closed eye detection means sets the threshold value based on the maximum value and the minimum value in the plurality of images in the maximum number of vertical continuous black pixels in the eyeball existing region, the situation of the environmental light ray and the eyes of each driver Even if there are variations in size,
There is an effect that the open / closed eye can always be easily identified.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of the present invention.

FIG. 2 is a block diagram showing a configuration of an example.

FIG. 3 is an explanatory diagram showing a coordinate system of an input image.

FIG. 4 is a flowchart showing a main flow of the overall operation of the embodiment.

FIG. 5 is a flowchart showing a flow of setting a width of an eyeball existing region.

FIG. 6 is a flowchart showing a flow of setting a lateral width of an eyeball existing region.

FIG. 7 is an explanatory diagram of image scanning in horizontal width setting.

FIG. 8 is a flowchart showing the flow of counting the number of continuous white pixels when setting the width.

FIG. 9 is a flowchart showing the flow of counting the number of continuous white pixels when setting the width.

FIG. 10 is an explanatory diagram of image scanning in setting the vertical width of the eyeball existing region.

FIG. 11 is a flowchart showing a flow of width setting.

FIG. 12 is a flowchart showing a flow of width setting.

FIG. 13 is a flowchart showing a flow of width setting.

FIG. 14 is an explanatory diagram of image scanning in the presence / absence search of glasses.

FIG. 15 is a flowchart showing a flow of eyeball detection.

FIG. 16 is a flowchart showing a flow of eyeball detection.

FIG. 17 is an explanatory diagram of scanning lines in an eyeball existing area.

FIG. 18 is a flowchart showing the processing of opening / closing eye detection.

FIG. 19 is an explanatory diagram of a doze determination.

FIG. 20 is a diagram showing an example of an input image.

[Explanation of symbols]

 CL1 image input means CL2 binarization means CL3 eyeball area setting means CL4 eyeball detection means CL5 open / closed eye detection means CL6 driver state determination means 1 infrared strobe 3 TV camera 5 timing command circuit 7 A / D converter 9 image memory 11 eyeball existence position regulation circuit 13 eyeball detection circuit 15 open / closed eye determination circuit 17 dozing determination circuit 19 alarm circuit

Claims (4)

[Claims] 1. An image input means for inputting a face image of a driver, a binarizing means for binarizing the face image input from the image input means, and an eyeball existence region in the binarized image. The eyeball existing area setting means for defining, eyeball detecting means for detecting the eyeball based on the continuous black pixels in the vertical direction in the eyeball existing area, and the driver based on the number of the continuous black pixels of the detected eyeball. A driver comprising: an open / closed eye detecting means for detecting the open / closed eye; and a driver state determining means for determining the driver's state based on the pattern of the open / closed eye detected by the open / closed eye detecting means. State detection device. 2. The eyeball existing area setting means comprises a width setting means and a height setting means, and the width setting means scans the image in the horizontal direction to count the number of continuous white pixels. Means, a stable portion detecting means for detecting a stable portion at the lateral edge of the image from the change amount of the continuous white pixel number, a presence / absence of the stable portion, and a change amount of the continuous white pixel number, and a contour line of the face. The contour edge continuity determining means for determining the continuity of the face and the face edge identifying means for identifying the lateral edge of the face based on the continuity determination, and the lateral edge of the identified face is detected by the eyeball. With the end point defining the horizontal width of the existing area, the vertical width setting means scans the image vertically from the lower position within the horizontal width set by the horizontal width setting means to detect two black areas. Means for controlling the presence of the eyeball based on the detected black area. 2. The driver's state detection device according to claim 1, wherein the vertical position of the zone is set. 3. The open / closed eye detection means uses the maximum number of the continuous black pixels as a reference to set a threshold value based on a maximum value and a minimum value in a plurality of images, and the continuous image for each image. The driver's state detection device according to claim 1 or 2, wherein the open / closed eyes are detected by comparing the maximum number of black pixels. 4. The driver state determination means determines that the driver is in an abnormal state when the eye closed state detected by the open / closed eye detection means continues for a predetermined time or more. Item 1. The driver state detection device according to item 1 or 2.