JP2000198369A - Eye state detecting device and doze-driving alarm device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve detecting and alarming accuracy by estimating the glaring state of a driver to correct the open/closed eye determination reference value. SOLUTION: This eye state detecting device is provided with a density detecting means CL2 for detecting density in a longitudinal picture element row of face image data inputted by an image input means CL1, a point extracting means CL3 for determining a pixel every rise of density to set an extraction point, a continuous data extracting means CL4 for extracting continuous data from the extraction points, eye position specifying means CL5, CL6 setting a specified area on the basis of the continuous data to specify the position of eyes, an eye open degree detecting means CL7 for detecting the open degree of eyes from the density change state of the continuous data, an open/ closed eye determination reference value setting means CL8 for setting the reference value for open/closed eye determination, an open/closed eye determination reference value correcting means CL9 for estimating the glaring state of a driver to correct the open/closed eye determination reference value, and an open/closed state determination means CL10 for judging the open/closed state of eyes using the reference value.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye condition detection device and a drowsiness alarm device for detecting a drowsiness state of a driver of a vehicle, a boat operator, an operator of a plant or the like, and issuing an alarm.

[0002]

2. Description of the Related Art An apparatus of this type using conventional image processing is disclosed, for example, in Japanese Patent Application Laid-Open No. 10-143669. In this configuration, the position of the eye is detected with respect to the grayscale image, and the eye is tracked while performing the open / closed eye determination based on the eye opening value.

[0003]

However, in the above-mentioned conventional apparatus, when performing the eye open / close determination, the open / closed eye determination reference value which is set and fixed in advance is used. When the driver hits near the eyes, even if the driver narrows his eyes in order to endure the glare, the closed eye determination is performed, and the accuracy of detection and alarm is reduced.

According to the present invention, a situation in which a driver feels dazzling is estimated by using density information around a face, and when the estimation is made, the open / closed eye determination reference value is corrected and detected. It is an object to improve the accuracy of the alarm.

[0005]

According to a first aspect of the present invention, there is provided an image inputting means for inputting a face image, and a density detecting means for detecting a density in a vertical pixel row of the image data of the face input by the image inputting means. Detecting means, a point extracting means for defining a pixel for each increase in the density detected by the density detecting means and setting it as an extraction point, and an extraction point of a pixel row adjacent to the extraction point extracted by the point extraction means A continuous data extracting means for extracting continuous data extending in the lateral direction of the face, and a predetermined area including the continuous data extracted by the continuous data extracting means, and an eye for specifying an eye position within the predetermined area. A position specifying unit, an eye opening detecting unit that detects the opening of the eye from the density change state of the continuous data of the eye position specified by the eye position specifying unit, and an output from the eye opening detecting unit. Time series of eye opening values The open / closed eye determination reference value setting means for learning the maximum value and the minimum value of the eye opening based on the change of the eye and setting a reference value used for the open / closed eye determination, and the driver dazzles from the density information in the predetermined area. An open / closed eye determination reference value correcting means for estimating a state in which the user feels sensation and correcting the reference value, and a reference value set by the open / closed eye determination reference value setting means or a reference value corrected by the open / closed eye reference value correction means And an open / closed state determining means for determining the open / closed state of the eye using the above.

According to a second aspect of the present invention, there is provided the eye state detecting apparatus according to the first aspect, wherein the open / closed eye determination reference value correction means has an average density of the predetermined area as the density information exceeding a predetermined value. In this case, it is characterized that it is estimated that the driver feels dazzling.

According to a third aspect of the present invention, there is provided the eye condition detecting apparatus according to the first aspect, wherein the open / closed eye determination reference value correcting means is configured to determine the right and left of the continuous data appearing in the predetermined area as the density information. When the average density of a predetermined section in the vertical direction passing through the central portion exceeds a predetermined value, it is estimated that the driver is in a state of feeling dazzling.

According to a fourth aspect of the present invention, there is provided the eye condition detecting apparatus according to the first aspect, wherein the open / closed eye determination reference value correcting means includes an average density of the predetermined area as the density information or a value within the predetermined area. The difference between the average density of a predetermined vertical section passing through the left and right central portions of the continuous data appearing at the brightest density value in the predetermined background area of the portion corresponding to the driver's background,
When the value exceeds a predetermined value, it is estimated that the driver feels dazzling.

According to a fifth aspect of the present invention, in the eye condition detecting apparatus according to the fourth aspect, the predetermined background area of the portion corresponding to the driver's background is determined according to the position of the predetermined area including the position of the eye. It is characterized by switching positions.

According to a sixth aspect of the present invention, there is provided the eye condition detecting apparatus according to the fourth or fifth aspect, further comprising a day / night determining means for determining day / night, and an infrared illuminating device for irradiating the vehicle interior with infrared light. The open / closed eye reference value correction unit, when it is determined that the night is based on the day / night determination result by the day / night determination unit, turns on the infrared illumination in the vehicle compartment and irradiates the predetermined area with infrared light to perform the correction. Is stopped, and when it is determined that it is daytime, it is estimated that the driver feels dazzling when the density difference is less than a predetermined value.

According to a seventh aspect of the present invention, there is provided the eye condition detecting apparatus according to any one of the first to sixth aspects, wherein the open / closed eye reference value correcting means is in a state where the driver feels dazzle. If the eye opening information output from the eye opening detecting means is not smaller than a predetermined value at the time of estimation, the reference value is not corrected.

According to an eighth aspect of the present invention, there is provided the eye state detecting apparatus according to any one of the first to seventh aspects, wherein the degree of arousal is determined based on a change in the open / closed state of the eye by the open / closed state determination means. It is characterized by comprising a judgment means, and an alarm means for issuing an alarm when the arousal level judgment means judges that the arousal level is lowered.

[0013]

According to the first aspect of the present invention, when the driver does not feel dazzling, the eye open / close judgment is performed based on the reference value set by the eye open / close judgment value setting means, and the eye state is determined by accurate judgment. Can be detected. In addition, when the driver's face is exposed to direct sunlight or the like, the situation where the driver feels dazzling is estimated by using the density information around the eyes,
When the estimation is made, the reference value of the open / closed eye determination can be corrected, for example, to a relatively small value by the open / closed eye determination reference value correction means. The state can be distinguished from the state where the opening value of the feeling eye is small, and more accurate eye state detection can be performed.

According to a second aspect of the present invention, in addition to the effect of the first aspect, the open / closed eye determination reference value correcting means determines whether or not the average density of the entire predetermined area exceeds a predetermined value. It is possible to estimate a state where the user feels glare, and it is possible to perform accurate and quick detection.

According to a third aspect of the present invention, in addition to the effect of the first aspect of the present invention, the open / closed eye determination reference value correcting means includes: a predetermined section in a vertical direction at a center in the left and right direction of continuous data appearing in the predetermined area; The state in which the driver feels dazzling can be estimated based on whether the average density exceeds a predetermined value, and more accurate detection can be performed.

According to a fourth aspect of the present invention, in addition to the effect of the first aspect of the invention, the open / closed eye determination reference value correction means may determine the average density of the entire predetermined area or the right and left of continuous data appearing in the predetermined area. A state in which the driver perceives glare depending on whether or not the difference between the average density in the predetermined vertical section at the center and the brightest density value in the predetermined background area in the portion corresponding to the driver's background exceeds a predetermined value. Can be estimated, and more accurate detection can be performed.

According to a fifth aspect of the present invention, in addition to the effect of the fourth aspect of the present invention, the position of the predetermined background area corresponding to the driver's background is switched according to the position of the predetermined area including the position of the eye. And detection can be accurately performed according to the movement of the driver.

According to a sixth aspect of the present invention, in addition to the effect of the fourth or fifth aspect, the open / closed eye reference value correcting means turns on the infrared light in the vehicle cabin based on the result of the day / night determination to bring the vehicle into the predetermined area. The correction can be stopped by irradiation with infrared light. Therefore, even if there is a night road lighting device or the like, accurate detection can be performed without being affected by the illumination device. For visible light of another vehicle, the average density of the entire predetermined area or the average density of a predetermined vertical section at the left and right central portions of continuous data appearing in the predetermined area, and a portion corresponding to the background of the driver When the difference from the brightest density value in the predetermined background area falls below the predetermined value, it is possible to estimate a state where the driver feels dazzling, and it is possible to perform accurate detection.

According to a seventh aspect of the present invention, in addition to the effect of any one of the first to sixth aspects, the open / closed eye reference value correcting means includes:
The eye opening information output from the eye opening detecting means can be considered, and even if it is estimated that the driver is in a state of dazzling in light environment, the eye opening value is Since the reference value is not reduced, unnecessary correction of the reference value can be prevented, and quick detection can be performed.

[0020] The invention of claim 8 provides, in addition to the effects of any of the inventions of claims 1 to 7, when the arousal level determination means determines that the arousal level has decreased from a change in the eye open / closed state. A warning can be issued by the warning means, and the drowsy driving can be suppressed.

[0021]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present apparatus can be used as a drowsy driving alarm for operators of railway vehicles, ships, plants and the like in addition to automobiles, but this embodiment will be described as applied to automobiles.

FIG. 1 is a functional block diagram of an eye condition detecting device and a drowsy driving alarm device to which one embodiment of the present invention is applied. The device comprises an image input means CL1, a density detecting means CL2, Inside point extraction means CL3, continuous data extraction means CL4, predetermined area setting means CL5 including eyes, area inside point extraction means CL6, eye opening detection means CL7, open / closed eye determination reference value setting means CL8, Open / closed eye determination reference value correction means CL9, open / closed state determination means CL10, arousal degree determination means C
L11 and alarm means CL12 are provided.

The image input means CL1 inputs a face image. The density detecting means CL2 detects a local increase in density in a vertical pixel row of the face image data input by the image input means CL1. The point extracting means CL3 in the image constitutes a point extracting means in the present embodiment, and defines a pixel for each increase in the density detected by the density detecting means CL2 as an extraction point. The continuous data extraction means CL4 extracts continuous data extending in the horizontal direction of the face when the extraction points extracted by the intra-image point extraction means CL3 approach each other in adjacent pixel rows.

The predetermined area setting means CL5 including the eye sets a predetermined area based on the continuous data extracted by the continuous data extraction means CL4 and specifies the area including the eye. The intra-region point extraction unit CL6 determines a pixel for each increase in the density of a pixel row in the vertical direction in a predetermined region including the position of the eye specified by the predetermined region setting unit CL5 including the eye, and sets the pixel as an extraction point. Then, the position of the eye is specified on the basis of continuous data that continuously extends in the lateral direction of the face at adjacent extraction points. Therefore,
The predetermined area setting means CL5 and the intra-area point setting means CL6 constitute eye position specifying means in the present embodiment.

The continuous data detected by the in-region point extracting means CL6 is based on the in-image point extracting means CL.
Although the detection is finer and more accurate than the continuous data detected in step 3, the processing can be accelerated by detecting the latter with the same roughness as the former.

The eye opening detecting means CL7 detects the eye opening from the density change state of the continuous data of the eye position specified by the eye position specifying means.

The open / closed eye determination reference value setting means CL8 determines the maximum value and the minimum value of the eye opening in the time series change of the eye opening output from the eye opening detecting means CL7. And sets a reference value used for open / closed eye determination. The open / closed eye determination reference value correction means CL9 estimates a state in which the driver feels dazzling from the density information in the predetermined area, and corrects the reference value for the open / closed eye determination.

The open / closed state determination means CL10 determines the open / closed state of the eye using the reference value set by the open / closed eye determination reference value setting means CL8 or the reference value corrected by the open / closed eye reference value correction means CL9. .

The arousal level judging means CL11 judges the arousal level based on the change of the open / closed state of the eye by the open / closed state judging means. The alarm means CL12 issues an alarm such as a buzzer when the alertness determination means CL11 determines that the alertness is lowered.

FIG. 2 is a configuration block diagram according to an embodiment of the present invention.

As shown in the figure, a TV camera 21 as the image input means CL1 is installed on an instrument of a car, and is provided for photographing a driver's face from the front.
In this embodiment, the input image of the TV camera 21 has 512 pixels in the horizontal direction (X) and 512 pixels in the vertical direction (Y) as shown in FIG.
It consists of 480 pixels.

The input image picked up by the TV camera 21 is stored in the image memory 23 via the A / D converter 22 as digital image input image data. The output of the image memory 23 is input to the image data operation circuit 24.

The image data arithmetic circuit 24 detects the density of the vertical pixel row of the face based on the input image data,
Points are extracted (extracted points) based on the increase in the density in the pixel row and its change state, and points adjacent to the adjacent pixel row in the pixel row direction are continuously extracted in the horizontal direction of the face. That is, in this embodiment, the image data arithmetic circuit 24 includes the pixel row density detecting means CL2, the point extracting means CL3, and the continuous data extracting means CL4.
Is composed. The output of the image data calculation circuit 24 is input to the eye position detection circuit 25.

The eye position detection circuit 25 detects an eye position by selecting an eye from the continuous data. That is, the eye position detection circuit 25 sets a predetermined area including the eye (eye tracking area), and extracts points from the increase in the density in the vertical direction in the predetermined area and its change state (extraction point). Then, consecutive data in the horizontal direction of the face are successively extracted from the points adjacent to each other in the pixel column direction of the adjacent pixel column. In this embodiment, the predetermined region setting unit CL5 for the eye and the intra-region point extraction unit CL6 are provided. It constitutes eye position specifying means. The output of the eye position detection circuit 25 is input to the eye opening detection circuit 26.

The eye opening detecting circuit 26 detects the eye opening from the density change state of the continuous data of the eye position specified by the eye position detecting circuit 25. That is, the eye opening detection circuit 26 constitutes the eye opening detecting means CL7. The output of the eye opening detection circuit 26 is input to the open / closed eye determination reference value setting circuit 27.

The open / closed eye determination reference value setting circuit 27 determines the maximum value of the eye opening degree in the time series change of the eye opening value output from the eye opening detection circuit 26 a plurality of times. The reference value used for learning the open / closed eyes is set by learning the minimum value, and this embodiment constitutes the open / closed eye determination reference value setting means CL8. The output of the open / closed eye determination value setting circuit 27 is input to the open / closed eye determination reference value correction circuit 28.

The open / closed eye determination reference value correction circuit 28 estimates the state in which the driver feels dazzling from the density information in the predetermined area, corrects the reference value for the open / closed eye determination, and opens / closes in this embodiment. The eye determination reference value correction means CL9 is configured.
The output of the open / closed eye determination reference value correction circuit 28 is input to the arousal level determination circuit 29.

The arousal degree determination circuit 29 determines the open / closed state of the eyes using the reference value set by the open / closed eye determination reference value setting circuit 27 or the reference value corrected by the open / closed eye reference value correction circuit 28. Then, the degree of arousal is determined from the change in the open / closed state of the eyes. Therefore, the wakefulness determination circuit 29 constitutes the open / closed state determination means CL10 and the wakefulness determination means CL11 in this embodiment. The output of the alertness determination circuit 29 is
Input to 0.

The warning device 30 issues a warning to alert the driver when the arousal level determination circuit 29 determines that the arousal level has decreased. The alarm device 30 constitutes the alarm means CL12 in the present embodiment.

Next, the flow of operation in the above configuration will be described with reference to the flowcharts of FIGS. First,
In step S301 (hereinafter, characters of “step” are omitted), a face portion is photographed by a TV camera,
In S302, an input image for one frame is converted into a digital signal by an AD converter and stored in an image memory.

Next, in step S303, it is checked whether an eye tracking area has been set. The eye tracking area indicates a predetermined area including the eye, and by detecting the position of continuous data appearing in this area, the detailed position of the eye is recognized, and the density data at the specified position is read. Eye opening is detected. Further, by setting a processing area in the next captured image based on the detailed position of the continuous data appearing in the predetermined area, it is possible to cope with a change in eye position.

If the eye tracking area has not been set,
In S304 and S305, a width in the horizontal direction (X direction) and a width in the vertical direction (Y direction) to be the tracking areas of the eyes are set. Details of the eye position detection will be described later with reference to the flowchart of FIG. 5 and the explanatory diagrams shown in FIGS.

If it is determined in S303 that the tracking area of the eye has been set, the eye opening and the detailed position detection are performed in S306. Details of this processing will be described later with reference to the flowchart of FIG. 12 and the explanatory diagrams shown in FIGS.

Thereafter, in step S307, it is determined whether or not the eye is correctly tracked based on the eye opening value detected in the previous step. If it is determined in step S307 that the eye has not been tracked correctly, the flow shifts to step S310 to clear the tracking area of the eye, and returns to step S301 to shift to processing of the next frame.
The details of the eye tracking will be described later with reference to an explanatory diagram shown in FIG.

If it is determined in step S307 that the eye is being tracked correctly, the flow shifts to step S308 to determine whether an open / closed eye determination reference value is set. If it is determined in step S308 that the open / closed eye determination reference value has not been set, the process advances to step S309 to set the open / closed eye determination reference value. The process of S309 is completed by outputting a plurality of eye opening values. Therefore, the process shifts from S308 to S309 until a predetermined number of eye opening values are output, and S3
A series of processing from 01 to S307 is repeated. The most common setting method of the open / closed eye determination reference value is a maximum value (when the eye is open) and a minimum value (when the eye is closed) of the degree of eye opening output in the plurality of processes. Is learned and the median value is used.

When the setting of the open / closed eye determination reference value is confirmed in S308, the flow shifts to S401 in FIG. In S401, by reading out the density information around the eyes, it is determined whether or not the driver is in a state of feeling dazzling. If it is determined in step S402 that the driver is not in a state of feeling dazzling, FIG.
Using the open / closed eye determination reference value set in S309 in S40
At step 3, the open / closed eye is determined. If it is determined in step S402 that the driver feels dazzling, the process proceeds to step S402.
To S, after correcting the open / closed eye determination reference value to a smaller value.
The flow shifts to 403, where an open / closed eye is determined. The details of the method of determining the state in which the driver feels dazzling in S401 will be described later with reference to the flowcharts of FIGS. 19, 22, and 25 and the explanatory diagrams illustrated in FIGS. 18, 20, 21, 23, and 24. .

In step S404, the tracking area of the eye is updated, and in step S405, the arousal level is determined based on the appearance pattern of the closed-eye output. If it is determined in S405 that the result of determining that the eyes are closed is output continuously and it is determined that long eyes are closed,
An alarm is issued to alert the driver. After the determination of the arousal level in S405, the process returns to S301 in FIG. 3 to input the next image and continue the same processing.

The details of the next eye position detection will be described.

The flow of the eye position detection process will be described with reference to the flowchart of FIG. First, in S501, as shown in FIG. 6, point extraction processing is performed on data on a line in the Y-axis direction, and after one line is completed, processing is shifted to the processing of the next adjacent line. It is determined whether or not the point extraction on the line has been completed. If it is determined in S501 that point extraction has not been performed on all lines, the process proceeds to S502. In this step S502, the arithmetic mean of the density values of one line in a predetermined direction is calculated. This processing is intended to eliminate small variations in the change in density value at the time of capturing image data, and to capture a global change in density value.

FIG. 7A shows the processing result of the arithmetic averaging operation of the line data of Xa in FIG. In S503 in FIG. 5, a differential operation is performed on the arithmetic average value that is the operation result of S502. FIG. 7B shows the processing result. In step 504 of FIG. 5, point extraction is performed using the differential value that is the result of the calculation in step 503. The method of extracting the point is a point (p1 to p1) at which the differential value changes from negative to positive.
p5) Referring to FIG. 7A, a point at which the graph becomes convex downward is extracted. Next, it is determined whether or not the change in the density value (q1 to q5) until the point is reached is equal to or less than a predetermined value, and whether or not the density value falls within a gray portion in FIG. The Y coordinate values (A1 to A3) are extracted for points having a change in value.

After this processing is completed for one line, in S505,
Switch to the processing of the next line. While repeating this process, for example, in the case of a line like Xb shown in FIG.
As can be seen from FIGS. 8A and 8B, there may be no extraction point.

When it is determined in S501 that the point extraction for all lines has been completed, points as shown in FIG. 9 are extracted. That is, two points A1 and A2 are extracted on the Xc line in FIG. 9, and on the Xd line,
This means that four points A1, A2, A3, and A4 have been extracted.

Thereafter, the flow shifts to S506, where the Y coordinate values of the extraction points (A1, A2, A3...) Of each adjacent line are compared, and if the Y coordinate value is within a predetermined value, the continuous data The group number, continuous start line number, and continuous data number are stored in memory. This specific processing will be described with reference to FIG.

In FIG. 10, line 1 has two extraction points having Y coordinate values of 192 and 229. Line 1
Since the line with the Y coordinate value of 192 does not have a line on the left, there is no continuous data at this stage, and the group number of the continuous data is "1". In addition, since there is no continuous data at this stage at the point of the Y coordinate value 229 for the same reason, the group number of the continuous data is set to the next “2”. Next, in line 2 on the right side, there are two extraction points whose Y coordinate values are 191 and 224. Since the point of the Y coordinate value 191 of the line 2 is a point within 10 with the Y coordinate value 192 of the line 1 on the left, the group number of the continuous data is set to “1”. At this time, the number of continuous data is two. When the same determination is performed at the point of the Y coordinate value 224 of the line 2, the group number of the continuous data is “2” and the number of continuous data is 2.

At the point of the Y coordinate value 360 of the next line 3, since there is no point within 360 and 10 in the line 2 on the left, the group number of continuous data is "3" and the number of continuous data is It becomes 1.

The continuous start line number in S506 refers to a line number having a point at which the number of continuous data is determined to be one.

In step S506, the continuity of the points on each line is determined until the processing is completed for all the lines, and the process proceeds to step S507.

In S507, the average value of the Y coordinate values of the points having the same continuous data group number is stored in the average value of the continuous points. This value can be used as a representative Y coordinate value of the group. Further, a continuous end line is obtained from the continuous start line and the number of continuous data, and an average value of the continuous start line and the continuous end line is stored.
This value can be used as the representative X coordinate value of the group.

In S508, the length of each continuous group, (X,
Y) The position of the eye can be specified by the determination based on the coordinate values.

Here, a specific eye position detecting method will be described with reference to FIG.

First, considering the characteristic amount of the eye, it can be defined that the eye has a horizontally long and upwardly convex arcuate shape. If the continuous data is narrowed down based on this definition, the eye becomes horizontal. , The difference between the Y coordinate values of the continuous start point and the continuous end point can be narrowed down to small continuous data, based on the condition that the number of continuous points continues for 5 or more points, and the condition of an arc shape.
When narrowing down the continuous data based on this determination, FIG.
Groups G1 to G6 as shown in FIG. 1A are extracted.

Next, considering the position of the representative coordinate values of X and Y of each group described above, as shown in FIG. 11B, the ZONE: L, Z
ONE: C and ZONE: R. This is because the left eye and the left eyebrow do not greatly separate in the X coordinate direction, and the right eye and the right eyebrow do not greatly separate in the X coordinate direction. In addition, continuous data due to the shadow under the nose,
Mouth continuous data is located near the center.

As described above, the data is further classified according to the degree of approach in the X coordinate direction, and the position of the eye can be easily detected by narrowing down the data. The continuous data included in ZONE: L includes the left eye and the left eyebrow.
E: If it is determined that the continuous data included in R is the right eye and the right eyebrow, the positions of the eyes are G3 and G4, and the coordinate values thereof can be specified. In this way, eye position detection can be performed for both eyes, but for the purpose of drowsiness detection, it seems that no driver would sleep with only one eye closed. Eye detection is limited to one eye (left eye). The reason for limiting to one eye (left eye) of the driver is not only to save computation processing time, but also in the case of a right-hand drive vehicle, the right eye, which may be exposed to direct light, has strong light. This is because the shape of the eye may be difficult to catch.

Next, the details of the eye opening detection in step S306 of FIG. 3 will be described.

FIG. 1 shows a method for detecting the eye opening value.
As shown in FIG. 4, the point Q where the density change from the white part of the skin to the black part of the eye is maximum, and the point R where the density change from the black part of the eye to the white part of the skin is maximum There is a method of calculating the interval, and a method of setting a binarization threshold, which will be described with reference to a flowchart of FIG.

Now, the details of the eye opening detection processing by setting the binarization threshold will be described.

First, the flow of a binarization threshold value setting method for converting a binarized image whose opening is detected will be described with reference to the flowchart of FIG. First, in step S1201, point extraction processing is performed on data on a line in the Y-axis direction as shown in FIG. 13A, and after one line is completed, processing proceeds to the next adjacent line, and all lines in the predetermined direction are processed. It is determined whether or not the point extraction on the line has been completed.

If it is determined in step S1201 that point extraction has not been performed on all lines, the process advances to step S1202. In step S1202, an arithmetic average of the density values of one line in a predetermined direction is performed. This processing is intended to eliminate a small variation in a change in density value at the time of capturing image data, and to provide a global change in density value. FIG. 13B shows the processing result of the arithmetic averaging operation of the line data of Xa in FIG.

In S1203 in FIG. 12, a differential operation is performed on the arithmetic mean value which is the operation result of S1202. The result of this processing is shown in FIG. S1204 in FIG.
Then, point extraction is performed based on the differential value that is the operation result of S1203. The method of extracting the point is a point P at which the differential value changes from negative to positive.
Extract points where the graph is convex to the left. Next, it is determined whether or not the change in the density value before and after the point is equal to or smaller than a predetermined value, and whether or not the density value falls within a gray portion in FIG. Is extracted.

If it is determined in S1205 that the extraction point P exists in the Xa column as shown in FIG. 13A, the flow shifts to S1206, where the maximum differential value before and after the point P is calculated.
The density values of the Y coordinate of the Q point and the R point, which are the minimum values, are stored as Nal (density value of the Y coordinate with the minimum differential value) and Nah (density value of the Y coordinate with the maximum differential value) in S1207. Move to.

In S1207, the process is switched to the process for the next line, and the same process is repeated until the end of the process for all lines is confirmed in S1201. That is, as shown in FIG. 14A, in the Xb column after the Xa column, the density values of the Y coordinate of the Q point and the R point which are the maximum and minimum values of the differential value before and after the extraction point P in the Xb column are represented by Nbl. And memory as Nbh. The minimum differential value N_l of the density change before the extraction point stored in each line is a portion where the density value changes most from a bright portion to a dark portion, and a portion darker than this density value, that is, the extraction point P Can be said to be the density value of the part corresponding to the eye. The maximum differential value N_h of the density change after the extraction point stored in each line is a portion where the density value changes most from a dark portion to a bright portion, and becomes bright from the extraction point P to this density value. It can be said that the portion is a density value corresponding to the eye. Therefore, the values of N_l and N_h are used as threshold setting information for reliably converting the eye part to a black pixel (0) and the skin part around the eye to a white pixel (1) in the binarization processing. Can be.

If it is confirmed in step S1201 that the processing for all the lines has been completed, the process advances to step S1208 to set a binarization threshold from the information on the density values (N_1, N_h) of the extraction points of each line. There are the following methods for setting the binarization threshold.

First, the density N_ of the minimum differential value of each line
The minimum value of 1 is updated and set based on that value.

Second, the density N_ of the maximum differential value of each line
The maximum value of h is updated and set based on that value.

Third, the density N_ of the minimum differential value of each line
Set based on the average value of l.

Fourth, the density N_ of the maximum differential value of each line
h is set based on the average value.

Fifth, the average value of all N_l and N_h of each line is set as a reference.

Sixth, the gradation up amount is corrected and set using the level of the differential value with reference to the continuous data of the eyes and the brightest density value at point P of each line to be extracted. (If the level of the differential value is large, the tone added to the brightest density value at point P is set large, and if the level of the differential value is small, the tone added to the brightest density value at point P is set small. Next, a method for detecting the eye opening will be described with reference to FIG.

Using the binarization threshold value obtained by the above-described method, binarization processing is performed by further limiting the area in the range where continuous data has appeared. The binarized image is as shown in FIG. 15 when the detection target person is normal and when he or she is dozing.
At this time, when the maximum number of consecutive continuations of the eye portion converted into black pixels in the vertical direction is counted, the value increases in a normal state (opened eyes) and decreases in a dozing state (closed eyes). In this way, the eye opening is detected.

Next, details of the eye tracking method in S404 of the flowchart of FIG. 4 will be described with reference to FIG.
In the initial state of this processing, that is, in the first frame, naturally,
Since the eye tracking area E has not been set, S30 in FIG.
4. In step S305, an eye tracking area is set. At this time, the center coordinates of the eye and the center coordinates of the tracking area E of the eye are as shown in FIG.
Match as shown.

After a series of processes, the process proceeds to the process of the second frame and proceeds to S303. Since the eye tracking area E has already been set here, the process proceeds to S306, where the eye opening and position detection are performed. Do. At this time, the detected positions of the eyes are as shown in FIG. The eye tracking area E in FIG. 16B is located at the position set in the first frame, whereas the current eye position is image data captured in the second frame. As a result, the center point of the eye is shifted from the tracking area E of the eye. However, as long as the eye is within the eye tracking area E, the eye opening and the detailed eye position detection can be performed. Thereafter, if the eye has been correctly detected, the process proceeds from step 307 to step 308 in FIG.
To Steps 401 to 403 in FIG.

In step S404, the tracking area E of the eye is updated based on the center coordinates of the eye calculated from the continuous data shown in FIG. Therefore, as long as the movement of the eye is not extremely fast, the tracking area E of the eye can be made to follow the movement.
FIGS. 16C and 16D show the positional relationship between the eye position and the eye tracking area E in the face image data captured in the third and fourth frames.

Further, the determination as to whether or not the eye has been correctly tracked in S307 is made based on the eye opening value. In short, once the photographer is identified, the eye opening value only changes in the range from when the eye is open to when the eye is closed, so if a value outside this range is output or the continuous data itself is lost Determines that there has been an eye tracking error, and S301
By clearing the output value of the eye opening and the tracking area E of the eye, the abnormal position is backed up by performing the eye position detection processing again in S304.

Next, the method of determining the state in which the driver feels dazzling in S401 of FIG. 4 will be described with reference to six examples.

In the first example, as shown in FIG. 17, the average density value of the entire eye tracking area E is read out, and whether the driver feels dazzle depending on whether or not the value exceeds a predetermined value is determined. Has been determined. For example, if the white density value is higher than a certain value, it is determined that the user is in a state of feeling glare.

In the second example, in order to further improve the accuracy of determining whether or not the driver feels dazzling, the range in which the density information around the eyes is read out is limited with respect to the first example. It was done. As shown in FIG. 18 (b), based on the representative coordinate values of the continuous data appearing in the eye tracking area E, the average density value is read out only for a vertically long section F at the center of the continuous data. It is determined whether or not the driver is in a state of dazzling based on whether or not the value exceeds a predetermined value.

In the third example, in order to further improve the accuracy of determining whether or not the driver feels glare, the density of the region corresponding to the background of the driver is compared with the first and second examples. Information is used.

When the driver's eyes are exposed to direct light, the area of the driver's background becomes completely dark due to automatic adjustment of the exposure of the camera. Therefore, by using the density information of the background in addition to the density information around the eyes, it is possible to further improve the determination accuracy of the state in which the driver feels dazzling.

The processing will be described with reference to the flowchart of FIG. In S1901, the average density value A shown in FIG. 20 is read out in the same manner as in Example 1 or Example 2 above.
At 02, the density value B of the brightest area H in the area corresponding to the driver's background shown in FIG. 20 is read. The reason why the brightest density value is used instead of the average density value in the area H corresponding to the background is that if hair or the like enters the same area H due to the movement of the driver's face, the average density value is easily affected by the hair. is there. The density values A and B are set in S1901 and S1902.
After the reading, the flow shifts to S1903 to determine the density difference. At this time, if the density difference between the density value A around the eyes and the density value B of the background is equal to or more than the predetermined tone difference GL, the flow shifts to S1904 to correct the open / closed eye determination reference value.

In the fourth example, in order to further improve the accuracy of determining whether or not the driver feels dazzling, in addition to the third example, when the face movement of the driver is large, This corresponds to the case where the position of the driver's face is shifted.

If the position of the driver's face shifts greatly and enters the region RH corresponding to the driver's background as shown in FIG. 21, the difference from the density value around the eyes cannot be determined accurately. The approach of the face to the area RH corresponding to the driver's background is determined using the representative coordinate values of the continuous data appearing in the eye tracking area E, and if it is determined that the face is approaching, the driver The area corresponding to the background is switched to LH. Here, the reason why the initial position of the region corresponding to the background of the driver is set to RH will be described. In the present embodiment, a right-hand drive vehicle is assumed, and the right side, which hits the inside of the vehicle, is less affected by the light from the side glass.

In the fifth example, daytime and nighttime judgments are added to the third example in order to further improve the accuracy of determining whether or not the driver feels glare. .

As shown in FIG. 24 (b), assuming that the vehicle is being processed at night or in a tunnel using infrared illumination, the driver's face is brightly projected by the illumination. However, the background part is only dark because of the distance from the lighting, and the difference between the density value around the eyes and the density value of the area corresponding to the background is very similar to the situation where the driver feels glare in the daytime. State. However, as a nighttime lighting device, the driver does not feel any glare because it uses infrared light that does not hinder driving.
At this time, the correction of the open / closed eye determination reference value must not be performed.

The processing in this case will be described with reference to the flowchart of FIG. In S2201, the average density value A shown in FIG. 23 is read based on the first or second example, and in S2202, the density value B of the brightest area H in the area corresponding to the driver's background shown in FIG. 23 is read. Thereafter, the flow shifts to S2203, where day / night determination is performed. In the present embodiment, as shown in FIG. 23, an area J for reading the average density value C is provided horizontally long at the lower part of the screen, and it is determined whether the average density value C is equal to or higher than a predetermined gradation by day / night. Do. In the present embodiment, a method of determining day or night based on density information on an image is described. However, an independent optical sensor or an ON / OFF state of a headlamp switch may be used.

If it is determined in S2203 that it is daytime, the process proceeds to S2
The process proceeds to S204, where the determination of the density difference is performed in the same manner as in the third example.
The process proceeds to 06, where the open / closed eye determination reference value is corrected. As the area gradually darkens from dusk to night, FIG.
As shown in (a), the average density value C of the area J in which day / night determination is performed also becomes dark. By this average density value C, S2
At 203, it is determined to be night, and the infrared illumination is turned on.
Therefore, for day / night determination based on the density value C, the infrared illumination is ON.
The determination is made based on whether or not the infrared illumination is ON. The determination from the nighttime to the daytime for turning off the infrared illumination is performed based on whether or not the density value B in the area corresponding to the background of the driver is continuously bright. The open / closed eye reference value correction circuit 28
Can turn on the infrared illumination in the vehicle cabin based on the day / night determination result, and stop the correction by irradiating the infrared light into the predetermined area F. Therefore, even if there is an illumination device for a road for night use, it is possible to perform accurate detection without being affected by the illumination device and performing unnecessary correction.

Next, a description will be given of a case where the driver feels dazzling with visible light such as headlights of an oncoming vehicle or a following vehicle at night. If it is determined in S2203 in FIG. 22 that it is night, the flow shifts to S2205, and similarly, the density value A of the area F around the eyes and the density value B of the background area H
Then, it is determined whether or not the difference exceeds a predetermined gradation difference GLN. At this time, if the density difference exceeds a predetermined value, open / closed eye determination is performed using a normal open / closed eye determination reference value. If the density difference is equal to or less than the predetermined value, it is determined that the driver feels dazzling with the headlights of the oncoming vehicle and the following vehicle, and S2 is determined.
The process proceeds to 206, where the open / closed eye determination reference value is corrected. The reason for this is that when the driver feels glare at the headlights of oncoming and following vehicles at night, the area around the face is also quite bright, so the background density around the eyes is very low. This is because the difference has become smaller.

In the sixth example, in order to further improve the accuracy of determining whether or not the driver feels dazzling, the determination of the eye opening value is added to the third example. Things.

In this example, in addition to the state in which there is a difference between the density values around the eyes and the background, it is determined whether or not the eye opening value itself is also small, thereby determining the open / closed eye determination reference value. The necessity of correction is determined again. According to this determination, even if it is estimated that the driver is in a state of feeling glare in the light environment, if the opening degree value of the eye itself is not small, it is determined that the driver does not feel glare Thus, unnecessary correction of the open / closed eye determination reference value can be prevented.

The processing is performed in S of the flowchart of FIG.
Only when the eye opening value is small, as in 2504, the correction of the open / closed eye determination reference value in S2505 is performed.

As described above, the open / closed eye determination reference value setting circuit 2
The open / closed state of the eye is determined by the arousal level determination circuit 29 using the reference value of the open / closed eye determination set in step 7 or the reference value corrected by the open / closed eye determination reference value correction circuit 28, and the change of the open / closed state of the eye To determine the degree of arousal. The alertness determination circuit 29
When it is determined that the arousal level has decreased, the warning device 30 issues a warning to alert the driver.

Therefore, by using the density information around the eyes, a situation in which the driver feels dazzle is estimated, and when the estimation is made, the reference value for determining whether the driver's eyes are open or closed is corrected to be smaller, so that the driver can fall asleep. It is possible to distinguish between a state in which the degree of opening of the falling eyes is small and a state in which the degree of opening of the eyes is small due to glare, and accurately prevent false alarms that occur when the driver feels glare. Can be. Therefore, the reliability of the drowsy driving warning device can be improved.

[Brief description of the drawings]

FIG. 1 is a functional block diagram according to an embodiment of the present invention.

FIG. 2 is a configuration block diagram according to an embodiment of the present invention.

FIG. 3 is a flowchart showing an overall operation.

FIG. 4 is a flowchart showing an overall operation.

FIG. 5 is a flowchart illustrating an operation of detecting an eye position.

FIG. 6 is an explanatory diagram related to eye position detection.

FIG. 7 is an explanatory diagram relating to eye position detection.

FIG. 8 is an explanatory diagram related to eye position detection.

FIG. 9 is an explanatory diagram related to eye position detection.

FIG. 10 is an explanatory diagram relating to eye position detection.

FIG. 11 is an explanatory diagram related to eye position detection.

FIG. 12 is a flowchart illustrating a method of calculating a binarization threshold for obtaining an eye opening value.

FIG. 13 is an explanatory diagram relating to a method of obtaining a binarization threshold.

FIG. 14 is an explanatory diagram relating to a method of obtaining a binarization threshold.

FIG. 15 is a diagram illustrating a method of outputting an eye opening value.

FIG. 16 is an explanatory diagram relating to an eye tracking method.

FIG. 17 is an explanatory diagram relating to a method of correcting the open / closed eye determination reference value of the first example.

FIG. 18 is an explanatory diagram relating to a method of correcting the open / closed eye determination reference value of the second example.

FIG. 19 is a flowchart illustrating an operation of correcting the open / closed eye determination reference value of the third example.

FIG. 20 is an explanatory diagram relating to a method of correcting the open / closed eye determination reference value of the third example.

FIG. 21 is an explanatory diagram relating to a method of correcting the open / closed eye determination reference value of the fourth example.

FIG. 22 is a flowchart illustrating an operation of correcting an open / closed eye determination reference value according to a fifth example.

FIG. 23 is an explanatory diagram regarding a method of correcting the open / closed eye determination reference value of the fifth example.

FIG. 24 is an explanatory diagram relating to a method of correcting the open / closed eye determination reference value of the fifth example.

FIG. 25 is a flowchart showing the operation of correcting the open / closed eye determination reference value of the sixth example.

[Explanation of symbols]

 CL1 Image input means CL2 Density detection means CL3 Point extraction means in image (point extraction means) CL4 Continuous data extraction means CL5 Predetermined area setting means (eye position identification means) CL6 Area point extraction means (eye position identification means) CL7 Eye opening degree detecting means CL8 Open / closed eye judgment reference value setting means CL9 Open / closed eye judgment reference value correcting means CL10 Open / closed state judgment means CL11 Arousal degree judgment means CL12 Alarm means 21 TV camera 22 A / D converter 23 Image memory 24 Image data 24 Image data Arithmetic circuit 25 Eye position detection circuit 26 Open / closed eye detection circuit 27 Open / closed eye determination reference value setting circuit 28 Open / closed eye determination reference value correction circuit 28 Arousal level determination circuit 29 Alarm device

Continuing on the front page (72) Inventor Takeshi Fukuda 31 Niheisu Odaira, Tone-cho, Kita-soma-gun, Ibaraki Prefecture (72) Inventor Masayuki Kaneda 2 Takaracho, Kanagawa-ku, Yokohama-shi, Kanagawa Nissan Motor Co., Ltd. (72) Inventor Masaji Owada 2 Takaracho, Kanagawa-ku, Yokohama-shi, Kanagawa F-term (reference) in Nissan Motor Co., Ltd. LL08 LL20

Claims (8)

[Claims] 1. An image input means for inputting a face image, a density detection means for detecting a density in a vertical pixel row of face image data input by the image input means, Point extraction means for defining a pixel for each increase in density and extracting points, and continuous data in which the extraction points extracted by the point extraction means and the extraction points of the pixel rows adjacent thereto extend in the horizontal direction of the face. Continuous data extracting means for extracting; an eye position specifying means for setting a predetermined area including the continuous data extracted by the continuous data extracting means and specifying an eye position within the predetermined area; and an eye position specifying means Eye opening detecting means for detecting the degree of opening of the eye from the density change state of the continuous data of the position of the eye specified by: a time series of eye opening values output from the eye opening detecting means The maximum eye opening based on the change An open / closed eye determination reference value setting means for learning a large value and a minimum value and setting a reference value used for open / closed eye determination; and estimating a state in which a driver feels dazzling from density information in the predetermined area, An open / closed eye determination reference value correction unit that corrects a value, and determines an eye open / closed state using a reference value set by the open / closed eye determination reference value setting unit or a reference value corrected by the open / closed eye reference value correction unit. An eye state detection device comprising: an open / closed state determination unit. 2. The eye state detection device according to claim 1, wherein the open / closed eye determination reference value correction means is configured to determine whether the average density of the predetermined area as the density information exceeds a predetermined value. An eye condition detection device for estimating a condition in which the user feels glare. 3. The eye state detection device according to claim 1, wherein the open / closed eye determination reference value correction means passes through a left and right central portion of continuous data appearing in the predetermined area as the density information. An eye state detection device, wherein when the average density of a predetermined section in a direction exceeds a predetermined value, it is estimated that the driver feels dazzling. 4. The eye state detection device according to claim 1, wherein the open / closed eye determination reference value correction means includes an average density of the predetermined area as the density information or continuous data appearing in the predetermined area. If the difference between the average density in a predetermined vertical section passing through the left and right central portions and the brightest density value in the predetermined background area of the portion corresponding to the driver's background exceeds the predetermined value, the driver feels glare An eye condition detection device, which estimates that 5. The eye state detection device according to claim 4, wherein a predetermined background area of a portion corresponding to a background of the driver switches a position according to a position of the predetermined area including the position of the eye. Characteristic eye condition detection device. 6. The eye state detection device according to claim 4, further comprising: a day / night determination unit that determines day / night; and an infrared illuminator that irradiates an infrared ray into a vehicle interior, wherein the open / closed eye reference value correction is performed. Means, when it is determined that it is night based on the day and night determination result by the day and night determination means, while turning on the infrared illumination in the vehicle interior, irradiating the predetermined area with infrared light and stopping the correction, An eye condition detecting device, wherein when it is determined that it is daytime, it is estimated that the driver feels dazzling when the density difference is less than a predetermined value. 7. The eye state detection device according to claim 1, wherein the open / closed eye reference value correction unit estimates that the driver is in a state of feeling dazzling. An eye condition detecting device, wherein the reference value is not corrected when the eye opening information output from the eye opening detecting means is not smaller than a predetermined value. 8. An eye state detection device according to claim 1, wherein said open / closed state determination means determines a degree of arousal from a change in the open / closed state of the eye. A drowsiness driving alarm device comprising: an alarm means for issuing an alarm when the arousal level determination means determines that the arousal level has decreased.