JP2000142164A - Eye condition sensing device and driving-asleep alarm device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an eye condition sensing device and a driving-asleep alarm device capable of perform sensing quickly and accurately. SOLUTION: An eye condition sensing device is equipped with eye position specifying means CL2, CL3, CL4, CL5 to specify the eye position on the basis of the continuous data of extraction points extracted from the image data of a face as input and eye opening sensing means CL6 and CL7 to sense the eye opening from the density varying condition of the continuous data, wherein the arrangement further includes a reference position deciding means CL8 to decide a reference position for the eye predictable as the eye existing based upon the continuous data, an eye tracing condition judging means CL9 to trace the continuous data of the eye in the specified region and judge whether the tracing condition is normal on the basis of the condition of the continuous data, and a restoring means CL10 to restore the specified region to the reference position when the means CL9 judges that the eye tracing goes in abnormality.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye condition detecting apparatus used for automatically controlling a mirror and a seat, and for specifying a position of an eye necessary for detecting a dozing state from an open / closed state of a driver's eyes. And a drowsiness alarm device capable of alarming a drowsiness.

[0002]

2. Description of the Related Art As a conventional drowsy driving warning device using image processing, there is, for example, one described in Japanese Patent Application Laid-Open No. Hei 10-143669. In this configuration, the position of the eye is detected for the grayscale image, and the eye is tracked while performing the open / closed eye determination based on the eye opening value. Further, the conventional apparatus is configured to recognize an eye tracking error based on the output state of the eye opening value and automatically return to the eye position detection processing.

[0003]

However, in the conventional apparatus, when the eye tracking fails and the eye position is detected again, the eye position is returned to the initial state and the eye position is detected. Has a problem that it takes a long processing time again. In addition, even if there is a state in which the initial eye position detection is successful and the eye open / closed eye determination is stably continued while tracking the eye, the risk of detection error is reduced by the eye position detection process in the initial state again. There is a problem that it accompanies.

[0004] It is an object of the present invention to provide an eye condition detecting device and a drowsy driving alarm device which can detect the condition more quickly and accurately.

[0005]

According to a first aspect of the present invention, there is provided a face image input means for inputting a face image, and a local density of density in a vertical pixel row of face image data input from the face image input means. An eye point specifying means for specifying a pixel for each image height and an extraction point, and specifying an eye position based on continuous data in which the extraction point of the pixel row adjacent to the extraction point approaches and extends in the lateral direction of the face; A predetermined area setting means for setting a predetermined area including the specified eye position; and specifying an eye position based on the continuous data within a predetermined area set by the predetermined area setting means, An eye condition detection apparatus comprising: an eye position detection means for detecting an eye position from a density change state, wherein an eye reference region in which an eye is predicted to be present is determined and stored based on the continuous data. A first eye reference region determining means; An eye tracking state determining unit that tracks continuous eye data in a predetermined area and determines whether the tracking state is normal based on the state of the continuous data; When it is determined to be abnormal,
A first return means for returning the predetermined area to the reference area is provided.

According to a second aspect of the present invention, there is provided a face image input means for inputting a face image, and a pixel for each local increase in density in a vertical pixel row of the face image data input from the face image input means. Eye position specifying means for specifying an eye position based on continuous data in which the extraction point of a pixel row adjacent to the extraction point is close to and extending in the lateral direction of the face; and A predetermined area setting means for setting a predetermined area including the position of the eye, and specifying the position of the eye based on the continuous data within the predetermined area set by the predetermined area setting means, from the density change state of the continuous data An eye condition detecting apparatus comprising: an eye opening detecting means for detecting an opening, wherein an eye reference region in which an eye is predicted to be present is determined based on the continuous data, and a first eye reference is stored. Area determining means; An eye tracking state determining unit that tracks continuous data of the eye and determines whether or not the tracking state is normal based on the state of the continuous data; and determines that the eye tracking is abnormal by the eye tracking state determining unit. When performed, left and right region setting means for setting a left and right region capable of detecting continuous data of the left and right eyes based on the reference region, and continuous data of determining whether continuous data appears in the left and right regions Appearance state determination means, and re-detection means for causing the eye opening degree detection means to perform the detection again based on the continuous data when it is determined that continuous data appears in the left and right regions are provided. It is characterized by having.

According to a third aspect of the present invention, there is provided a face image input means for inputting a face image, and a pixel for each local increase in density in a vertical pixel row of face image data input from the face image input means. Eye position specifying means for specifying an eye position based on continuous data in which the extraction point of a pixel row adjacent to the extraction point is close to and extending in the lateral direction of the face; and A predetermined area setting means for setting a predetermined area including the position of the eye, and specifying the position of the eye based on the continuous data within the predetermined area set by the predetermined area setting means, from the density change state of the continuous data An eye condition detection apparatus comprising: an eye position detecting means for detecting an opening degree; tracking eye continuous data in the predetermined area; and determining whether a tracking state is normal based on a state of the continuous data. Eye tracking state determination means A left and right region setting unit that sets a left and right region in which continuous data of the left and right eyes can be detected based on the reference region when the eye tracking state determination unit determines that the eye tracking is abnormal; A continuous data appearance state determining means for determining whether or not continuous data has appeared; and, when it is determined that the continuous data has appeared, the eye position specifying means based on the continuous data to perform the detection. A second eye reference area determining means for determining an eye reference area that can be predicted to have an eye based on the eye position information specified by the re-identifying means, and storing the same. A second return means for returning the predetermined area to the reference area when the eye tracking state determination means determines that the eye tracking is abnormal.

According to a fourth aspect of the present invention, there is provided the eye condition detecting apparatus according to the first, second, or third aspect, wherein the first or second reference region determining means repeats the eye position detecting process. It is characterized by learning a reference region of the eye.

According to a fifth aspect of the present invention, there is provided the eye condition detecting apparatus according to the first, second, or third aspect, wherein the first or second reference area determining means determines a time series of eye opening values. It is characterized in that a change state is determined and a reference region of the eye is learned.

According to a sixth aspect of the present invention, there is provided the eye state detecting apparatus according to the first, second, or third aspect, wherein the first or second reference area determining means performs a plurality of eye position detecting processes. When a more stable eye position is output, the reference region of the eye is determined.

According to a seventh aspect of the present invention, in the eye condition detecting apparatus according to the second aspect, the re-detecting means is capable of stably detecting the two consecutive areas in the left and right regions at predetermined intervals in the horizontal direction. Redetection is performed when data is present.

According to an eighth aspect of the present invention, there is provided the eye condition detecting apparatus according to the third aspect, wherein the re-identifying means is capable of stably detecting the two consecutive areas in the left and right regions at predetermined intervals in the horizontal direction. Redetection is performed when data is present.

The ninth aspect of the present invention is the second aspect of the present invention.
The eye condition detecting device according to the above, further comprising a determination extending means for extending the determination by a predetermined time when continuous data in the left and right regions cannot be detected stably.

According to a tenth aspect of the present invention, there is provided the second aspect of the present invention.
8. The eye state detection device according to 8, wherein a determination extending means for extending the determination by a predetermined time when continuous data in the left and right regions cannot be detected is provided.

The invention of claim 11 is the invention of claims 2, 3, 7,
8. The eye state detecting device according to claim 8, wherein when the continuous data in the left and right regions cannot be stably detected and when the continuous data in the left and right regions cannot be detected, the determination extending unit extends the determination for a predetermined time. And the determination extension means sets a longer extension time when continuous data cannot be detected than when it cannot be detected stably.

According to a twelfth aspect of the present invention, in the eye condition detecting apparatus according to any one of the first to eleventh aspects, the tracking state determining means moves the continuous data in the predetermined area and stays at the position for a predetermined time. It is characterized in that when it is determined that the eye tracking is abnormal.

According to a thirteenth aspect of the present invention, in the eye state detecting apparatus according to any one of the first to eleventh aspects, the tracking state determining means is configured to detect an eye tracking abnormality when continuous data in the predetermined area cannot be detected for a predetermined time. Is determined.

According to a fourteenth aspect of the present invention, in the eye state detecting device according to any one of the first to eleventh aspects, the tracking state determining means detects that the eye opening value exceeds a predetermined range and is constantly detected. When the eye tracking abnormality is determined.

According to a fifteenth aspect of the present invention, there is provided the eye state detecting apparatus according to any one of the first to fourteenth aspects, wherein the eye opening state output by the eye opening detecting means is determined, and the open / closed state of the eye is determined. An arousal level determining means for determining the arousal level from the change, and an alarm means for issuing an alarm when the arousal level is determined to be low by the arousal level determining means are provided.

[0020]

According to the first aspect of the present invention, the first eye reference area determining means determines an eye reference area in which an eye can be predicted based on continuous data specifying an eye position, and tracks the eye. When the state determination unit determines that the eye tracking is abnormal, the return unit can return the predetermined region for detecting the eye opening to the reference region. Therefore, when the detection is performed again, the processing from the initial state to the setting of the predetermined area can be omitted, and the processing can be performed quickly. In addition, there is no risk of a detection error in the omitted processing, and more accurate detection can be performed.

According to the second aspect of the present invention, the first eye reference area determining means determines an eye reference area in which the eye can be predicted based on the continuous data specifying the eye position, and determines the tracking state of the eye. When it is determined by the means that the eye tracking is abnormal, the left and right areas in which continuous data of the left and right eyes can be detected based on the reference area are set by the left and right area setting means, and the appearance state determining means continuously sets the left and right areas in the left and right areas. When it is determined that data has appeared, the eye opening degree detecting means can perform the detection again based on the continuous data. Therefore, when the detection is performed again, the processing from the initial state to the setting of the predetermined area can be omitted, and the processing can be performed quickly. In addition, there is no risk of a detection error in the omitted processing, and more accurate detection can be performed. Furthermore, the left-right symmetry condition of continuous data can be used, and more accurate detection can be performed.

According to the third aspect of the present invention, when the eye tracking state determination means determines that the eye tracking is abnormal, the left and right regions in which the continuous data of the left and right eyes can be detected based on the eye position continuous data of the initial detection. Set by the left and right area setting means, when it is determined that continuous data has appeared in the left and right areas, the re-specification means based on the continuous data the eye position identification means to perform the detection again, A second eye reference region determining unit determines an eye reference region that can predict that an eye is present based on the eye position information specified by the re-identifying unit, and the eye tracking state determining unit determines the eye tracking state. When it is determined that the eye is abnormal, the predetermined area for detecting the degree of eye opening can be returned to the reference area by the second return means. For this reason, in addition to the value corresponding to the opening value as information for determining whether the re-detected continuous data is an eye,
Since it is possible to use a left-right symmetry condition or the like of continuous data, it is possible to improve the detection accuracy of the reference region of the eye.
Further, since the region in which the eye is re-detected is limited to the left and right regions, the detection time of the reference position can be shortened, and quick processing can be performed.

According to the fourth aspect of the present invention, in addition to the effects of the first or second aspect of the present invention, the reference position of the eye can be learned by repeating the eye position detection processing, and more accurate processing can be performed. Can be.

According to the fifth aspect of the present invention, in addition to the effects of the first or second aspect, it is possible to learn the reference region of the eye by judging a time-series change state of the eye opening value. Accurate processing can be performed.

According to the sixth aspect of the present invention, in addition to the effects of the first or second aspect, when the eye position is output stably by a plurality of eye position detection processes, the reference region of the eye is output. Can be determined, and more accurate processing can be performed.

According to a seventh aspect of the present invention, in addition to the effect of the second aspect, re-detection is performed when there are two continuous data which can be stably detected at predetermined intervals in the horizontal direction in the left and right regions. Thus, more accurate processing can be performed by using the left-right symmetry condition of continuous data.

According to an eighth aspect of the present invention, in addition to the effect of the third aspect, re-detection is performed when there are two continuous data which can be stably detected at predetermined intervals in the horizontal direction in the left and right regions. Thus, more accurate processing can be performed by using the left-right symmetry condition of continuous data.

According to the ninth aspect of the present invention, there is provided the second aspect of the present invention.
In addition to the effects of the invention described in 8, when the continuous data in the left and right regions cannot be detected stably, the determination by the appearance state determination means can be extended for a predetermined time, and the continuous data can be captured more accurately, thereby achieving more accurate Good processing can be performed.

According to the tenth aspect, in the second aspect,
In addition to the effects of the inventions described in 7 and 8, when the continuous data in the left and right regions cannot be detected, the determination by the appearance state determination means can be extended for a predetermined time, and the continuous data can be more accurately captured, so that more accurate data can be obtained. Processing can be performed.

According to the eleventh aspect, in the second aspect,
In addition to the effects of the inventions described in 7 and 8, when the continuous data in the left and right areas is not stable and when there is no continuous data in the left and right areas, the determination by the appearance state determination means can be extended for a predetermined time. By capturing data more accurately, more accurate processing can be performed. Moreover, the determination extension means sets the extension time when the continuous data does not exist longer than when the continuous data is not stable, so that even when the continuous data does not exist, the continuous data is more accurately captured after the extension time, so that the accuracy is improved. Processing can be performed.

According to the twelfth aspect of the invention, in addition to the effects of the first to eleventh aspects, when continuous data in a predetermined area moves and stays at the position for a predetermined time, it is determined that the eye tracking is abnormal. Thus, even if continuous data such as eyeglasses, which are noises around the eyes due to a fast movement of the face, is likely to be tracked, the data can be accurately canceled and redetected with high accuracy.

According to the thirteenth aspect, in addition to the effects of the first to eleventh aspects, when continuous data in a predetermined area cannot be detected for a predetermined time, it can be determined that there is an eye tracking abnormality, and a change in the light environment. Even if there is, for example, highly accurate re-detection can be performed.

In the fourteenth aspect, in addition to the effects of the first to eleventh aspects, when the eye opening value exceeds the predetermined range and is constantly detected, it is determined that the eye tracking is abnormal. This makes it possible to perform highly accurate re-detection even when the face is largely moved sideways.

According to the fifteenth aspect, in addition to the effects of the first to fourteenth aspects, it is possible to accurately issue an alarm when it is determined that the arousal level is decreasing.

[0035]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present apparatus can be used as a drowsy driving alarm for operators of railway vehicles, ships, plants and the like in addition to automobiles, but this embodiment will be described as applied to automobiles.

FIG. 1 is a functional block diagram of an eye condition detecting device and a drowsy driving alarm device to which one embodiment of the present invention is applied. The device includes an image input means CL1 and a pixel row density detecting means CL2. Point extracting means CL3, continuous data extracting means CL4, eye position detecting means CL5, point extracting means CL6, eye opening detecting means CL7, first eye reference area determining means CL8, Eye tracking state determination means CL9, first return means CL10, arousal level determination means CL11, alarm means
With CL12.

The image input means CL1 inputs a face image. The pixel line density detecting means CL2 detects a local increase in the density of the input face image data in the vertical pixel line. The point extracting means CL3 detects points based on the increase in the density in the pixel row and its change state. The continuous data extracting means CL4 extracts continuous data in the horizontal direction of the face by continuously connecting points adjacent to each other in the pixel column direction. The eye position detecting means CL5 detects an eye position based on the continuous data. These pixel row density detecting means CL2, point extracting means CL3, continuous data extracting means CL4, and eye position detecting means CL5 constitute an eye position specifying means of the present invention.

The point extracting means CL6 sets a predetermined area including the specified eye position, determines a pixel for each increase in the density of the pixel row in the vertical direction in the predetermined area, sets the pixel as an extraction point, and sets an adjacent point. The position of the eye is specified based on continuous data extending in the horizontal direction of the face continuously at the extraction points. Note that the continuous data detected by the point extracting means CL6 is finer than the continuous data detected by the point extracting means CL3, and performs accurate detection.However, the latter is detected with the same roughness as the former. By doing so, it is possible to adopt a configuration in which the processing is accelerated. The eye opening detecting means CL7 detects the eye opening from the density change state of the continuous data in the predetermined area.
These point extracting means CL6 and eye opening detecting means CL7
These constitute the predetermined area setting means and the eye opening degree detecting means of the present invention.

The eye reference region determining means CL8 determines a reference position as an eye reference region in which an eye can be predicted based on the continuous data.

The eye tracking state determination means CL9 tracks continuous data of the eye within the predetermined area, and determines whether the tracking state is normal based on the state of the continuous data.

The first return means CL10 returns the predetermined area to the reference position when the eye tracking state determining means determines that the eye tracking is abnormal.

The arousal level determining means CL11 determines the state of the eye opening output from the eye opening determining means, and determines the arousal level from the change in the open / closed state.

The alarm means CL12 issues an alarm such as a buzzer when the alertness determining means determines that the alertness is lowered.

FIG. 2 is a configuration block diagram according to an embodiment of the present invention.

As shown in the figure, a TV camera 21 is provided as an image input means CL1 which is installed in an instrument of a car and takes a picture of a driver's face from the front.
In the present embodiment, the input image of the camera 21 is composed of 512 pixels in the horizontal (X) direction and 480 pixels in the vertical (Y) direction.

The input image captured by the TV camera 21 is stored in the image memory 23 via the A / D converter 22 as digital image input image data. The output of the image memory 23 is input to the image data operation circuit 24.

The image data arithmetic circuit 24 detects the density of the vertical pixel row of the face based on the input image data,
Points are extracted (extracted points) based on the increase in the density in the pixel row and its change state, and points adjacent to the adjacent pixel row in the pixel row direction are continuously extracted in the horizontal direction of the face. That is, the image data calculation circuit 24 constitutes the density detection means CL2 of the pixel row, the point extraction means CL3, and the continuous data extraction means CL4. The output of the image data calculation circuit 24 is input to the eye position detection circuit 25.

The eye position detecting circuit 25 detects an eye position by selecting an eye from the continuous data. That is, the eye position detecting circuit 25 is provided with the eye position detecting means CL.
5. The output of the eye position detection circuit 25 is input to the open / closed eye detection circuit 26.

The open / closed eye detection circuit 26 sets a predetermined area including the eyes (eye tracking area), and extracts points in the predetermined area from the increase in density in the vertical direction and its change state (extraction point). Extracting continuous data in the horizontal direction of the face by continuously connecting points adjacent to each other in the pixel column direction of the adjacent pixel column, and detecting the opening degree of the eye from the density change state of the continuous data in the range specified by the point I do. That is, the open / closed eye detection circuit 26 constitutes the eye opening degree detection means CL7.
The output of the open / closed eye detection circuit 26 is input to the backup circuit 27.

The backup circuit 27 determines a reference position of the eye that can be predicted to have an eye based on the continuous data, tracks the continuous data of the eye within the predetermined area, and based on the state of the continuous data. Determining whether or not the eye tracking state is normal, and returning the predetermined area to the reference position of the eye in which the eye tracking area has been learned in advance when the eye tracking abnormality is determined and the eye tracking failure is confirmed. . That is, the backup circuit 27 is provided with the first eye reference area determining means CL.
8, the eye tracking state determination means CL9, and the first return means CL10. The output of the backup circuit 27 is input to the arousal level determination circuit 28.

The arousal level determination circuit 28 determines the arousal level based on the detection result of the open / closed eye sent from the open / closed eye detection circuit 26. That is, the arousal level determination circuit 28 constitutes the arousal level determination means CL11. The output of the arousal level determination circuit 28 is input to the alarm device 29. The alarm device 29
When it is determined that the arousal level has decreased, an alarm is issued to call attention to the driver. That is, the alarm device 29 constitutes the alarm means CL12.

Next, the flow of the entire operation in the above configuration will be described with reference to the flowcharts of FIGS.
First, in step 301, a face portion is photographed by the TV camera 21, and in step 302, an input image for one frame is converted into a digital signal by an AD converter and stored in an image memory.

Next, in step 303, it is checked whether or not an eye tracking area has been set. The eye tracking area refers to an area set as a predetermined area including the eyes in the entire image, recognizes the detailed position of the eye by detecting the position of continuous data appearing in this area, and identifies the eye. The opening degree of the eye is detected by reading out the density data at the shifted position. Further, by setting a processing area in the next captured image based on the detailed position of the continuous data appearing in the predetermined area, it is possible to cope with a change in eye position.

If the eye tracking area has not been set,
In steps 304 and 305, a width in the horizontal direction (X direction) and a width in the vertical direction (Y direction) to be the tracking areas of the eyes are set. The details of eye position detection are shown in FIG.
This will be described later with reference to the flowchart of FIG.

If it is determined in step 303 that the eye tracking area has been set, then in step 306 the eye opening and detailed position detection are performed. Details of this processing will be described later with reference to the flowchart of FIG. 12 and the explanatory diagrams shown in FIGS.

Thereafter, in step 307, it is determined whether or not the eye opening value is normal, and if it is normal, the process proceeds to step 401 in FIG. In step 401, it is determined whether or not the reference position of the eye has been learned. Since the judgment in this step in the initial state is naturally not learned, the flow shifts to step 403 to start learning the reference position of the eye. The method of learning the reference position of the eye is based on the condition that the output state of the eye opening value continues to change in the range considered to be the size of the normal eye and that there is a position where the data stays. Store the reference position. In this case, data stays at a reference position of the eye for too long time (for example, 3
S) for more than a second.

At step 403, while learning the reference position of the eye, at step 404 an open / closed eye is determined based on the eye opening value, and at step 405, the tracking area of the eye is updated.
Further, in step 406, the arousal level is determined based on the appearance pattern of the closed-eye output. If it is determined in step 406 that the eyes are continuously closed and it is determined that long eyes are closed, an alarm is issued to alert the driver.

After the determination of the arousal level in step 406, the process returns to step 301 to input the next image, and the same processing is continued. In this processing flow, if the eye opening value is out of the normal eye size range and becomes extremely large (such as a person with dark eyebrows) or small (such as the line of the mouth) In step 307, it is recognized that the eye opening value is abnormal, and the process proceeds to step 308. In step 308,
It is confirmed by a plurality of processes whether the abnormal state of the eye opening value is temporary or stationary, and if it is determined that the state is abnormal, the tracking area of the eye is determined in step 309. clear.

Thereafter, Step 301 to Step 303
The same processing as described above is repeated, but in the determination of step 303, since the tracking area of the eye has been cleared due to the occurrence of a steady eye opening value abnormality, when this flow is performed, The eye position detection will be performed again.

After learning the reference position of the eye in step 403,
If it is determined in step 402 that the eye tracking has failed in the flow from step 401 to step 402, the process proceeds to step 407, and the tracking area to the reference position of the eye is returned to step 301. Then, the next image is input and the same processing is continued.

That is, if the eye tracking error is made due to the rapid movement of the driver's face, the deterioration of the light environment, or the noise around the eyes when wearing glasses, the movement amount and the Recognition of the tracking error based on the dwell time at the position and returning the eye tracking area to the reference position of the eye where the eye is most likely to be present when the driver is looking ahead of the vehicle and driving, thereby improving accuracy. High eye re-detection can be performed in an instant. Details of the operation state by this processing will be described later with reference to the explanatory diagrams shown in FIGS. Details of a specific eye tracking method will be described later with reference to the explanatory diagram of FIG.

Next, details of the eye position detection will be described.

The flow of the eye position detection process will be described with reference to the flowchart of FIG. First, in step 501, point extraction processing is performed on data on a line in the Y-axis direction, as shown in FIG. It is determined whether or not the point extraction has been completed.

If it is determined in step 501 that point extraction has not been performed on all lines, the process proceeds to step 502. In this step 502, arithmetic mean calculation of density values of one line in a predetermined direction is performed. This processing is intended to eliminate small variations in the change in density value at the time of capturing image data, and to capture a global change in density value. FIG. 7A shows a processing result of the arithmetic averaging operation of the line data of Xa in FIG.

In step 503 of FIG.
The differential operation is performed on the arithmetic average value that is the operation result of Step 2. FIG. 7B shows the processing result. In step 504 of FIG. 5, point extraction is performed using the differential value that is the result of the calculation in step 503. The method of extracting that point is
Points where the differential value changes from negative to positive (p1 to p5),
In FIG. 7A, a point where the graph becomes convex downward is extracted. Next, it is determined whether or not the change in the density value (q1 to q5) until the point is reached is equal to or less than a predetermined value, that is, whether or not the density value falls within a gray portion in FIG. The Y coordinate values (A1 to A3) are extracted for points having a change in density value. After this process is completed for one line, the process is switched to the process for the next line in step 505.

During the repetition of this processing, for example, in the case of a line like Xb shown in FIG. 6, there is a case where there is no extraction point as can be seen from FIGS. 8 (a) and 8 (b).

If it is determined in step 501 that point extraction for all lines has been completed, points as shown in FIG. 9 are extracted. That is, on the Xc line in FIG.
Two points of A2 are extracted, and four points of A1, A2, A3, and A4 are extracted on the Xd line.

Thereafter, the process proceeds to step 506 in FIG.
Extraction points of adjacent lines (A1, A2, A3 ...)
Are compared, and if the Y coordinate value is within a predetermined value, the continuous data group number, continuous start line number, and continuous data number are stored as continuous data. This specific processing will be described with reference to FIG.

Line 1 has Y coordinate values of 192 and 229
There are two extraction points. Y coordinate value of line 1 is 1
At point 92, there is no line adjacent to the left, and there is no continuous data at this stage, so the group number of the continuous data is "1". In addition, since there is no continuous data at this stage at the point of the Y coordinate value 229 for the same reason, the group number of the continuous data is set to the next “2”. Next, in line 2 on the right side, there are two extraction points whose Y coordinate values are 191 and 224. Since the point of the Y coordinate value 191 of the line 2 is a point within 10 with the Y coordinate value 192 of the line 1 on the left, the group number of the continuous data is set to “1”. At this time, the number of continuous data is two. When the same determination is performed at the point of the Y coordinate value 224 of the line 2, the group number of the continuous data is “2” and the number of continuous data is 2.

The continuous start line number in step 506 in FIG. 5 refers to a line number having a point where the number of continuous data is determined to be one.

Next, in FIG.
At 60 points, there are no points within 360 and 10 in line 2 on the left, so the group number of continuous data is “3” and the number of continuous data is 1.

At step 506 in FIG. 5, the continuity of the points on each line is determined until the processing is completed for all the lines, and the process proceeds to step 507.

In step 507, the average value of the Y coordinate values of the points having the same continuous data group number is stored in the average value of the continuous points. This value can be used as a representative Y coordinate value of the group. Further, a continuous end line is obtained from the continuous start line and the number of continuous data, and an average value of the continuous start line and the continuous end line is stored. This value can be used as the representative X coordinate value of the group.

At step 508, the length of each continuous group,
By determining based on (X, Y) coordinate values, the position of the eye can be specified.

Here, a specific eye position detecting method will be described with reference to FIG.

First, considering the characteristic amount of the eye, it can be defined that the shape is long and laterally convex and upwardly convex. When the continuous data is narrowed down based on this definition, the eye becomes horizontal. , The difference between the Y coordinate values of the continuous start point and the continuous end point can be narrowed down to small continuous data, based on the condition that the number of continuous points continues for 5 or more points, and the condition of an arc shape. When narrowing down the continuous data based on this determination, FIG.
Groups G1 to G6 as shown in FIG.

Next, considering the positions of the representative coordinate values of X and Y of each group described above, as shown in FIG.
From the degree of approach in the coordinate direction, ZONE: L, ZON
It can be classified into E: C and ZONE: R. This is because the left eye and the left eyebrow do not greatly separate in the X coordinate direction, and the right eye and the right eyebrow do not greatly separate in the X coordinate direction. Further, continuous data due to the shadow under the nose and continuous data of the mouth are located near the center.

As described above, the data is further classified according to the degree of approach in the X coordinate direction, and the position of the eye can be easily detected by narrowing down the data. The continuous data included in ZONE: L includes the left eye and the left eyebrow.
E: If it is determined that the continuous data included in R is the right eye and the right eyebrow, the positions of the eyes are G3 and G4, and the coordinate values thereof can be specified. In this way, eye position detection can be performed for both eyes, but for the purpose of dozing detection, it is assumed that no driver will sleep with only one eye closed. Eye detection is limited to one eye (left eye). The reason for limiting to one eye (left eye) of the driver is not only to save the calculation processing time, but also in the case of a right-hand drive vehicle, the right-handed vehicle is closer to the side window than the left eye and is likely to receive direct light. Eyes are more likely to be difficult to catch the shape of the eye due to the intensity of light,
Opening / closing eye detection accuracy is improved by limiting the probability of direct light to the left eye to be smaller than the right eye. Therefore, in the case of a left-hand drive vehicle, it is better to limit the vehicle to the right eye.

Next, the details of the eye opening detection in step S306 in FIG. 3 will be described.

FIG. 1 shows a method for detecting the eye opening value.
As shown in FIG. 4, the point Q where the density change from the white part of the skin to the black part of the eye is maximum, and the point R where the density change from the black part of the eye to the white part of the skin is maximum There is a method of calculating the interval, and a method of setting a binarization threshold, which will be described with reference to a flowchart of FIG.

Now, the details of the eye opening detection processing by setting the binarization threshold will be described.

First, the flow of a binarization threshold value setting method for converting into a binarized image for which the opening is detected will be described with reference to the flowchart of FIG. First, step 12
In FIG. 13A, point extraction processing is performed on the data on the line in the Y-axis direction as shown in FIG. 13A, and after one line is completed, the processing shifts to the processing of the next adjacent line. It is determined whether or not the point extraction has been completed. If it is determined in step 1201 that point extraction has not been performed on all lines, step 120
Move to 2.

In this step 1202, 1 in the predetermined direction
The arithmetic operation of the density value of the line is performed. This processing is intended to eliminate small variations in the change in density value at the time of capturing image data, and to capture a global change in density value. FIG. 13B shows the processing result of the arithmetic averaging operation of the line data of Xa in FIG.

In step 1203 of FIG. 12, a differential operation is performed on the arithmetic mean value which is the operation result of step 1202. This processing result is shown in FIG. In step 1204 of FIG. 12, point extraction is performed using the differential value that is the result of the operation in step 1203. The method of extracting the points is the point P, at which the differential value changes from negative to positive,
Referring to FIG. 13B, a point at which the graph becomes convex to the left is extracted. It is determined whether or not the change in density value before and after the next point is equal to or less than a predetermined value and whether or not the change is within a gray portion in FIG. The point P having is extracted.

In step 1205 of FIG. 12, FIG.
If it is determined that the extraction point P exists in the Xa column as shown in (a), the process proceeds to step 1206 in FIG. 12, where the Q and R, which are the maximum and minimum differential values before and after the point P, are set.
The density value of the Y coordinate of the point is stored as Nal (the density value of the Y coordinate at which the differential value is maximum) and Nah (the density value of the Y coordinate at which the differential value is maximum), and the process proceeds to step 1207.

At step 1207, the process is switched to the process for the next line, and the same process is repeated until the end of the process for all lines is confirmed at step 1201. That is, FIG.
As shown in FIG. 4A, in the Xb column after the Xa column, the density values of the Y coordinate of the Q point and the R point which are the maximum and minimum values of the differential value before and after the extraction point P in the Xb column are represented by Nbl and Nbh. Go to memory as. The minimum differential value N_1 of the density change before the extraction point stored in each line is a portion where the density value changes most from a bright portion to a dark portion, and a portion darker than this density value, that is, the extraction point P Can be said to be the density value of the part corresponding to the eye. The maximum differential value N_h of the density change after the extraction point stored in each line is a portion where the density value changes most from a dark portion to a bright portion, and the extraction point P
It can be said that a portion that becomes brighter from the point to this density value is a density value corresponding to the eyes. Therefore, the values of N_1 and N_h are used as threshold setting information for reliably converting the eye part into a black pixel (0) and the skin part around the eye into a white pixel (1) in the binarization processing. Can be.

If it is confirmed in step 1201 in FIG. 12 that the processing of all the lines has been completed, the process proceeds to step 1208, where the density values (N_1, N_
A binarization threshold is set from the information in h). The following method can be considered for setting the binarization threshold.

The minimum value of the density N_1 of the minimum differential value of each line is updated and set based on the updated value.

The maximum value of the density N_h of the maximum differential value of each line is updated and set based on the updated value.

The average value of the density N_1 of the minimum differential value of each line is set as a reference.

The average value of the density N_h of the maximum differential value of each line is set as a reference.

The average value of all N_1 and N_h of each line is set as a reference.

Based on the continuous data of the eyes and the brightest density value at point P of each line to be extracted, the gradation up amount is corrected and set using the level of the differential value. (If the level of the differential value is large, the tone added to the brightest density value at point P is set large, and if the level of the differential value is small, the tone added to the brightest density value at point P is set small. Next, a method for detecting the degree of eye opening will be described with reference to FIG.

Using the binarization threshold value obtained by the above-described method, binarization processing is performed by further limiting the area within the range where continuous data has appeared. The binarized image is as shown in FIG. 15 when the detection target person is normal and when he or she is dozing.
At this time, when the maximum number of consecutive vertical portions of the eye portion converted to black pixels is counted, the value increases in a normal state (opened eyes) and decreases in a dozing state (closed eyes). In this way, the eye opening is detected.

Next, the details of the method for judging the open / closed eye in step 404 in FIG. 4 will be described.

When the driver is specified, the output value of the degree of eye opening changes between the open state and the closed state. Therefore, the reference value for determining whether the eye is open or closed, that is, the threshold for determining whether the eye is open or closed, is within the range. As a result of examining how it is appropriate to set the threshold here, since some people who are dozing are not in a deep sleep state, some drivers can not completely close their eyes, so the median value of opening and closing the eyes is It would be desirable to have a threshold.

Next, step 40 in the flowchart of FIG.
Details of the eye tracking method in 5 will be described with reference to FIG. In the initial state of this processing, that is, in the first frame, since the eye tracking area is not set, the eye tracking area is determined in steps 304 and 305 in FIG. At this time, the center coordinates of the eye and the center coordinates of the tracking area of the eye are shown in FIG.
They match as shown in FIG.

After a series of processing is completed, the processing shifts to the processing of the second frame, and proceeds to step 303 in FIG. 3. Since the tracking area of the eye has already been set here, the processing shifts to step 306 to open the eye. And position detection. At this time, the detected positions of the eyes are as shown in FIG. FIG.
The eye tracking area in (b) is located at the position set in the first frame, whereas the current eye position is image data captured in the second frame. The points are shifted with respect to the tracking area of the eye. However, as long as the eye is within the eye tracking area, the eye opening and the detailed position detection of the eye can be performed.

Thereafter, if the eye has been correctly detected, the process proceeds from step 307 in FIG. 3 to step 4 in FIG.
01 to step 405. Step 40
In 5, the eye tracking area is updated based on the center coordinates in FIG. Therefore, as long as the movement of the eye is not extremely fast, the tracking area of the eye can be made to follow the movement. FIGS. 16C and 16D show the positional relationship between the eye position and the eye tracking area in the face image data captured in the third and fourth frames.

Next, an example of making an eye tracking error and a backup method at that time will be described with reference to FIGS. After the learning of the reference position of the eye, in the determination of the tracking state of the eye in step 402 in FIG. 4, the tracking area of the eye moves to a position separated from the reference position of the eye learned in advance by a predetermined amount or more, and the position It is determined that an eye tracking error has occurred when the eye has stayed for more than a predetermined time, and the process proceeds to step 407 to return the eye tracking area to the reference position of the eye.

This phenomenon will be described with a specific example.
In the driver wearing glasses as shown in FIG. 7, there is noise that can be candidate data of eyes around the eyes, and slight vertical movement of the face causes the eye tracking area to move to a position as shown in FIG. May come. At this time, the continuous data 1 corresponding to the eye to be originally detected falls outside the tracking area, and the continuous data 2 corresponding to the frame of the eyeglasses as noise enters the same area, and the continuous data 2 has to be selected. It may not be possible. If erroneous continuous data is detected in the area shown in FIG. 18, the tracking is performed on the continuous data, and the eye tracking area moves accordingly. In such a case, by performing the processing of step 402 and step 407 in FIG. 4, the eye tracking area can be automatically returned as shown in FIG.

Therefore, when the detection is performed again, the processing up to the detection of the eye tracking area can be omitted, and the processing can be performed quickly. Further, there is no risk of a detection error due to eye position detection in the initial state again, and highly accurate processing can be performed.

In the above-described embodiment, a description has been given of a backup method for a detection error in which continuous data of the eye always exists, the movement of which is confirmed, and stays at the position, but in steps 307 and 308 of FIG. When the abnormality of the eye opening value occurs constantly, it is necessary to detect the position of the eye from the initial state again. Explaining the case where such a state occurs, as shown in FIG. 20A, strong sunlight hits the right half of the driver's face,
When the contrast of the left eye, which has become a shadow part, is extremely low and cannot be confirmed as continuous data, or the driver moves his face quickly and greatly as shown in FIG.
If this state is continued for a long time and the tracking area of the eye cannot follow the movement and stays at that position, a state where the eye opening value is abnormal occurs constantly.

An embodiment for solving this problem will be described below.
This will be described with reference to the functional block diagram of FIG. 21, the flowcharts of FIGS. 22 to 24, and the explanatory diagrams of FIGS. 25 to 34.

In this embodiment, as shown in FIG. 21, image input means CL2101, density detection means CL2102,
Point extraction means CL2103, continuous data extraction means CL2104, eye position detection means CL2105, first eye reference area determination means CL2106, point extraction means CL2107, eye opening detection means CL2108, eye opening value status Determination means CL2109, area setting means CL2
110, appearance state determination means CL2111, re-detection means C
L2112, eye tracking state determination means CL2113, first
Return means CL2114, arousal level determination means CL211
5. An alarm means CL2116 is provided.

The image input means CL2101 inputs a face image. The density detecting means CL2102 detects the density of a vertical pixel row of the face image input from the image input means CL1201. The point extracting means CL21
In step 03, points are extracted based on the increase in density in the pixel row and the change state. The continuous data extracting means CL2104 extracts continuous data in the horizontal direction of the face by continuously connecting points adjacent to each other in the pixel column direction. An eye is selected from the eye position detecting means CL2105 and continuous data.

The eye reference region determining means CL2106
Determines a reference position as an eye reference region by a plurality of detection processes by the eye position detection means CL2105. The point extracting means CL2107 extracts points from the density increase in the vertical direction in a predetermined area including the eyes (eye tracking area) and the change state thereof. The eye opening detecting means CL2108 detects the detailed position of the eye by extracting continuous data extending in the horizontal direction of the face by continuously extracting those adjacent to the extraction point in the pixel column direction. Then, the range from which the density value is read is specified, and the degree of eye opening is detected from the density change state within the range. The eye opening value status determining means CL2109 detects a steady state of the eye opening value abnormality.

The area setting means CL2110 detects left and right symmetrical data for the purpose of detecting left-right symmetric data based on the reference position of the eye when the eye opening value state determining means CL2109 detects an abnormal state of the eye opening value. Set. The continuous data appearance state determination unit CL2111 determines the appearance state of the continuous data in the left and right regions for detecting left-right symmetric data.

The re-detection means CL 2112 is based on the appearance state determination result of the continuous data,
Redetection by 2108 is performed. The eye tracking state determination means CL2113 is a predetermined area including the eye (eye tracking area).
The data is tracked based on the appearance position of the continuous data corresponding to the eye in the inside, and the tracking state is determined. The first
The return means CL2114 of the eye tracking state determination means CL
When 2113 determines that the eye tracking is abnormal, a predetermined area including the eye (eye tracking area) is returned to the reference position of the eye.

The wakefulness determination means CL2115 determines the open / closed state of the eye from the openness value output from the eye openness detection means CL2108, and determines the wakefulness from the change in the open / closed state. The alarm means CL2116 issues an alarm when the alertness determination means CL2115 determines that the alertness is decreasing.

Next, the flow of operation in the above configuration will be described in detail with reference to the flowcharts of FIGS. 22 to 24 and the explanatory diagrams of FIGS. 24 to 34. It should be noted that the processing content in each step to be described below is mainly performed on portions different from the above-described embodiment.

In step 2203 of FIG. 22, it is confirmed whether or not the left and right regions are set for detecting symmetric continuous data. This region is set based on the reference position of the eye as shown in FIG. 25, and its size is a region that completely includes continuous data of both eyes.

Next, in step 2204, it is checked whether or not an eye tracking area has been set. If the eye tracking area is not set, step 2205
Then, the position of the eye is detected in the same manner as in the above-described embodiment. Thereafter, in step 2206, learning of the reference position of the eye is performed, and whether or not the learning is completed is determined in step 2207.
Confirm with.

An example of a method of learning the reference position of the eye will be described. In the result of detecting the position of the eye from a plurality of images as a whole, the reference position of the eye is determined when the position of the eye is detected at the same position a predetermined number of times. There is a method to determine the value. Therefore, when the eye reference value is being learned, the process returns from step 2207 to step 2201 to repeat the eye position detection a plurality of times.

If it is confirmed in step 2207 that the reference position of the eye has been learned, the flow shifts to step 2208 where the width in the horizontal direction (X direction) and the width in the vertical direction (Y direction) as the eye tracking area. Set.

Thereafter, in step 2210, it is determined whether or not the eye opening value is normal.
The process proceeds to step 2301 of FIG. Since the process of step 2301 is the same as that of the above-described embodiment, the description thereof is omitted.

Next, due to the cause shown in the example of FIG.
The processing content when an eye opening value abnormality occurs will be described.

In this state, it is recognized in step 2210 in FIG. 22 that the eye opening value is abnormal, and the flow shifts to step 2211. In step 2211, left and right regions are set to detect symmetric continuous data. The setting method and the size of the area are described in the processing content of step 2203.

Thereafter, the flow returns to step 2201 to input an image with the left-right symmetric data as a detection target.
The process proceeds from step 203 to step 2401 in FIG. In step 2401, continuous data is detected within the detection region of left-right symmetric data as shown in FIG.
Move to 02. In step 2402, it is determined whether or not the waiting time for appearance determination of continuous data has been exceeded.
When the flow enters this flow for the first time, the process goes to step 2403 because the waiting time is not set.

If the state in which the eye opening value is determined to be abnormal in step 2210 in FIG. 22 is a temporary phenomenon, detection of continuous data in the left and right regions for detecting left / right symmetric data is performed. Is performed, continuous data of both eyes can be detected stably as shown in FIGS. Therefore,
In step 2403, two or more stable appearances can be confirmed, and the flow shifts to step 2408.

In the determination in step 2403, the reason why there are two or more continuous data portions that appear stably is that the detection region of the left-right symmetric data is misaligned or the driver's face movement causes the continuous data of the eyes. This is because continuous data of eyebrows may also be included. In step 2408, if the vehicle is a right-hand drive vehicle, the eye position is detected again by setting the tracking area of the eye based on the continuous data of the left eye of the driver having relatively good light environment conditions. Then, step 240
By clearing the left and right regions for detecting the left-right symmetric data in 9, the flow returns to the flow of the open / closed eye determination and the arousal level determination.

Next, a case where the light environment is bad or a case where the driver is looking sideways for a relatively long time (for example, 1 to 2 minutes) will be described.

For the above reasons, step 2210 in FIG.
In the case where the eye opening value is abnormal in step 2403, the appearance of two or more continuous data cannot be confirmed stably in the left-right symmetric data detection area in step 2403 in FIG. . In step 2404, similarly to step 2402, even when the flow enters this flow for the first time, since the continuous data appearance determination waiting time is not set, the flow shifts to step 2405. In step 2405, the occurrence of continuous data is 0 to 2
It is checked whether it is constantly changing in the range of the individual.
27 to 32 show the case where the light environment is deteriorated.
This will be described with reference to FIG. The case where the driver's face direction is not fixed and the driver is looking around other than the front of the vehicle for a long time will be described with reference to FIGS. 33 and 34.

The deterioration of the light environment when an automobile is assumed is as follows.
When the vehicle is running, it is constantly changing, and FIG.
As shown in FIG. 29, the left eye portion of the driver is slightly shadowed but can be recognized as continuous data, or as shown in FIG. 29, the shadow of the driver's left eye portion is darkened as continuous data. The case where recognition is not possible or the case where direct light to the right eye is further increased and the right eye cannot be recognized as continuous data as shown in FIG. 31 are repeated. Therefore, in the appearance state of the continuous data, the number of appearances of the continuous data constantly changes as shown in FIG. 28, FIG. 30, and FIG. It should be noted that the upper limit of the number of appearances of continuous data includes the case where continuous data of eyebrows is included,
Assuming a driver wearing spectacles, the number may naturally be two or more.

Next, a situation in which the driver is looking at a place other than the front for a long time will be described. When the driver is facing sideways as shown in FIG. 33, only one piece of continuous data appears in the symmetrical continuous data detection region as shown in FIG. So the driver looks right,
The number of occurrences of continuous data constantly changes even when looking to the left or dozing around. Performing re-detection of an eye in such a situation may cause an erroneous detection of the eye. Therefore, the process proceeds to step 2410, and the determination is extended by a predetermined time by setting a continuous data appearance determination waiting time t1, The eye can be re-detected while waiting for the improvement of the light environment or the position of the driver's face calms down.

If it is not recognized in step 2403 in FIG. 24 that the appearance of two or more continuous data is stable, and if it is not recognized in step 2405 that the appearance of the continuous data is constantly changing, the flow advances to step 2406. The process proceeds to check whether continuous data does not appear at all even after the predetermined number of processes.

As a scene corresponding to step 2406, there is a case where the face of the driver is so moved that the driver's face is not captured in the camera. To give an example,
For example, when the driver is looking for something in the glove box. If it is considered that such a situation is present, the process proceeds to step 2411, where a continuous data appearance determination waiting time t2 is set, and the determination is extended by a predetermined time. The waiting time t2 is set based on the time required for driver replacement. In short, when the appearance of continuous data completely disappears, in addition to the case where the face moves greatly, the driver may be replaced.If the driver seems to have been replaced, the eye position detection from the initial state should be performed. It is necessary. Step 2
Regarding the waiting time t1 at 410, at least 1 to 1 is considered because the lighting environment is poor or the driver is sulking, which may last for a relatively long time.
It seems that setting in 2 minutes or more is preferable.

Step 2401 or Step 2411
After setting the continuous data appearance waiting time,
Based on the determination in step 04, it is determined whether or not there is stable appearance of two or more continuous data by passing through step 2403 within the waiting time without setting the waiting time again. If no stable two or more continuous data appear within this waiting time, the process proceeds from step 2402 to step 2.
The flow shifts to 407, where the tracking area of the eye, the detection area of the left-right symmetric data, and the waiting time for determination are cleared, and the process returns to the eye position detection from the initial state.

Next, still another embodiment will be described with reference to FIG.
5 and the flowchart of FIG.

In the above-described embodiment, the reference position of the eye is learned by detecting the output state of the eye opening value a plurality of times and the position of the eye. However, in the former method, if the output value of the eye opening contains noise, the continuous data that is not the eye is erroneously recognized as the eye and the reference position is set. In this case, even if the continuous eye data is correctly captured, the position may not be easily learned as the reference position of the eye. Further, in the latter method, it is necessary to repeat the eye position detection from the initial state in order to learn the reference position of the eye, so that it takes time to learn the reference position of the eye for the first time. is there.

Therefore, in the present embodiment, when the abnormality is recognized by tracking the eyes and determining the opening degree status, the left and right regions for detecting left-right symmetric data based on the position of the eyes detected in the processing. Is set, and the eye position is detected again from the appearance state of the continuous data appearing in the area. At this time, the reference position of the eye is learned based on the detected position data of the eye.

According to this method, in addition to the value corresponding to the opening value as information for determining whether or not the re-detected continuous data is an eye, the left-right symmetry condition of the continuous data and the eyes are opened. Since it is possible to use a feature having a convex shape on the continuous data that can be recognized in such a case, the learning accuracy of the reference position of the eye can be improved. Further, since the region in which the eye is re-detected is limited to the detection region of the left-right symmetric data, the learning time can be reduced.

Next, the details of the processing will be described with reference to the flowcharts of FIGS. 35 and 36. It should be noted that the processing content in each step to be described below is mainly performed on portions different from the above-described embodiment.

In step 3508 shown in FIG. 35, an abnormal state is recognized on both sides of the eye tracking and the opening value. The determination of the tracking state of the eye until the reference position of the eye is learned is performed based on the position where the eye position is detected. Step 3508
If no abnormality is confirmed in step 3512, the flow shifts to step 3512 to repeat the flow of repeating the determination of the arousal level from the output pattern of the closed eyes, as in the above-described embodiment.

If abnormalities are found in the eye tracking and the opening degree status in step 3508, the flow shifts to step 3509. In step 3509, it is determined whether or not the reference position of the eye has been learned. Immediately after the start of the process, the reference position of the eye has not been learned.
The process moves to 510. In step 3510, left and right regions for detecting left-right symmetric data are set. The region where the left-right symmetric data is to be detected is determined based on the position of the eye detected in step 3505. Then, Step 3510
Returning to the flow, the flow enters the flow of determining the appearance state of continuous data appearing in the left and right regions for detecting left-right symmetric data as in the above-described embodiment.

In step 3603 of FIG. 36, it is determined whether or not two or more continuous data appear stably in the detection area of the left-right symmetric data. If the appearance is confirmed, the flow proceeds to step 3608. Then, the position of the eye is re-detected based on the continuous data that has been shifted and appeared, and step 36 is performed.
At 09, learning of the reference position of the eye is performed. Subsequent step 3
At 610, it is determined whether learning of the reference position of the eye is OK. The OK determination also includes conditions such as a small variation in the re-detected eye position, and whether the opening degree value for the continuous data changes at a size corresponding to the eye.

If the above condition is satisfied in step 3610, the process returns to the flow for determining the arousal level by clearing the left and right regions for detecting left-right symmetric data in step 3611.

In the flow of the awakening degree determination after the return, if the eye tracking and the opening value abnormality are confirmed again in step 3508 in FIG. 35, the reference position of the eye is learned in step 3509. Step 3515
Move to In step 3515, it is determined whether or not the eye tracking and the abnormal state of the opening degree are steady. If it is determined in step 3515 that the abnormality is not a stationary abnormality, the process proceeds to step 3511, where the tracking area is returned to the reference position of the eye and backup can be performed in a short time as in the above-described embodiment.

If it is determined in step 3515 that the abnormality is a steady abnormality, the flow shifts to step 3516 to clear the eye tracking area and the reference position of the eye so that the eye position can be detected in the initial state. And return to step 3501.
This process is prepared on the assumption that the driver is replaced while the flow of the arousal level determination is running.

In the step 3603 in FIG. 36, if two or more continuous data are not confirmed to appear stably in the detection area of the left-right symmetric data, the process proceeds to step 3604, and the same as in the above-described embodiment. The waiting time for the appearance determination of the continuous data is set, and the confirmation of the appearance of two or more continuous data appearing stably is continued within the waiting time.

If the waiting time is exceeded in step 3602, the flow shifts to step 3607 to clear the eye tracking area, the left and right areas for detecting left-right symmetric data, and the judgment waiting time, thereby detecting the eye position in the initial state. And the process returns to step 3501.

It should be noted that this flow is performed when the eye position initially detected at step 3505 in FIG. 35 is incorrect, and when the eye is in the left and right regions for detecting left-right symmetrical data while learning the reference position of the eye. May be replaced.

As is clear from the above description, the present invention recognizes abnormal processing by monitoring the tracking state of the eye and the state of the opening value, and when there is an abnormality, moves to the reference position of the eye learned in advance. By returning the eye tracking area, accurate eye re-detection can be performed in a minimum necessary time even if there is an eye tracking error due to fast face movement or temporary deterioration of the light environment. Therefore, the processing time ratio of the open / closed eye determination in the entire processing time increases, so that the accuracy of dozing detection can be improved.

[Brief description of the drawings]

FIG. 1 is a functional block diagram of an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of one embodiment of the present invention.

FIG. 3 is a flowchart illustrating an overall operation of the embodiment.

FIG. 4 is a flowchart showing an overall operation of the embodiment.

FIG. 5 is a flowchart illustrating an operation of detecting an eye position.

FIG. 6 is an explanatory diagram related to eye position detection.

FIG. 7 is an explanatory diagram relating to eye position detection.

FIG. 8 is an explanatory diagram related to eye position detection.

FIG. 9 is an explanatory diagram related to eye position detection.

FIG. 10 is a table for explaining eye position detection.

FIG. 11 is an explanatory diagram related to eye position detection.

FIG. 12 is a flowchart illustrating a method of calculating a binarization threshold for obtaining an eye opening value.

FIG. 13 is an explanatory diagram relating to a method of obtaining a binarization threshold.

FIG. 14 is an explanatory diagram relating to a method of obtaining a binarization threshold.

FIG. 15 is a diagram illustrating a method of outputting an eye opening value.

FIG. 16 is an explanatory diagram relating to an eye tracking method.

FIG. 17 is an explanatory diagram relating to a method of restoring an eye tracking area.

FIG. 18 is an explanatory diagram relating to a method of restoring an eye tracking area.

FIG. 19 is an explanatory diagram related to a method of returning an eye tracking area.

FIG. 20 is an explanatory diagram relating to a problem of one embodiment.

FIG. 21 is a functional block diagram showing a configuration of another embodiment.

FIG. 22 is a flowchart showing an overall operation of another embodiment.

FIG. 23 is a flowchart showing the overall operation of another embodiment.

FIG. 24 is a flowchart showing the overall operation of another embodiment.

FIG. 25 is an explanatory diagram relating to a method for determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 26 is an explanatory diagram relating to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 27 is an explanatory diagram relating to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 28 is an explanatory diagram relating to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 29 is an explanatory diagram regarding a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 30 is an explanatory diagram relating to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 31 is an explanatory diagram relating to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 32 is an explanatory diagram relating to a method for determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 33 is an explanatory diagram relating to a method for determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 34 is an explanatory diagram relating to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 35 is a flowchart showing the overall operation of still another embodiment.

FIG. 36 is a flowchart showing the overall operation of the embodiment.

[Explanation of symbols]

 CL1 Image input means CL2 Face pixel density detection means CL3 Point extraction means CL4 Continuous data extraction means CL5 Eye position detection means CL6 Point extraction means in eye tracking area CL7 Eye opening detection means CL8 Eye reference position determining means CL9 Eye tracking state determining means CL10 Eye tracking area returning means CL11 Awakening degree determining means CL12 Alarm means 21 TV camera 22 A / D converter 23 Image memory 24 Image data arithmetic circuit 25 Eye Position detection circuit 26 Open / closed eye detection circuit 27 Backup circuit 28 Arousal level determination circuit 29 Alarm device

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 3D037 FA05 FB09 5B057 AA16 DA07 DA15 DB02 DC22 5C086 AA23 BA20 BA22 BA23 CA28 CB36 DA01 DA08 DA33 FA02 5H180 AA01 CC04 LL20

Claims (15)

[Claims] 1. A face image input means for inputting a face image, and a pixel is determined for each local increase in density in a vertical pixel row of the face image data input from the face image input means. Eye position specifying means for specifying an eye position based on continuous data in which the extraction point of the pixel row adjacent to the extraction point is close to and extending in the lateral direction of the face; and a predetermined position including the specified eye position A predetermined area setting means for setting an area, an eye position is specified based on the continuous data in the predetermined area set by the predetermined area setting means, and an eye opening is detected from a density change state of the continuous data. An eye state detection device including an eye opening degree detection unit, wherein a first eye reference region determination unit that determines and stores an eye reference region in which an eye can be predicted based on the continuous data, Continuous data of the eyes within the predetermined area The eye tracking state determining means for determining whether the tracking state is normal based on the state of the continuous data, and determining that the eye tracking state is abnormal by the eye tracking state determining means. An eye condition detection apparatus, comprising: a first return unit that returns the predetermined area to the reference area. 2. A face image input means for inputting a face image, and a pixel is determined for each local increase in density in a vertical pixel row of the face image data input from the face image input means. Eye position specifying means for specifying an eye position based on continuous data in which the extraction point of the pixel row adjacent to the extraction point is close to and extending in the lateral direction of the face; and a predetermined position including the specified eye position A predetermined area setting means for setting an area, an eye position is specified based on the continuous data in the predetermined area set by the predetermined area setting means, and an eye opening is detected from a density change state of the continuous data. An eye state detection device including an eye opening degree detection unit, wherein a first eye reference region determination unit that determines and stores an eye reference region in which an eye can be predicted based on the continuous data, Continuous data of the eyes within the predetermined area The eye tracking state determining means for determining whether the tracking state is normal based on the state of the continuous data, and determining that the eye tracking state is abnormal by the eye tracking state determining means. Left and right region setting means for setting left and right regions in which continuous data of both right and left eyes can be detected based on the reference region; and continuous data appearance state determining means for determining whether or not continuous data appears in the left and right regions. And re-detecting means for causing the eye opening detecting means to perform the detection again based on the continuous data when it is determined that continuous data appears in the left and right regions. Eye condition detecting device. 3. A face image input unit for inputting a face image, and a pixel is determined for each local increase in density in a vertical pixel row of the face image data input from the face image input unit. Eye position specifying means for specifying an eye position based on continuous data in which the extraction point of the pixel row adjacent to the extraction point is close to and extending in the lateral direction of the face; and a predetermined position including the specified eye position A predetermined area setting means for setting an area, an eye position is specified based on the continuous data in the predetermined area set by the predetermined area setting means, and an eye opening is detected from a density change state of the continuous data. An eye state detection device comprising: an eye opening degree detection unit; an eye state tracking apparatus that tracks continuous eye data in the predetermined area and determines whether the tracking state is normal based on the state of the continuous data. Tracking state determining means, and the tracking state of the eye When it is determined that the eye tracking is abnormal by the state determination means, left and right area setting means for setting left and right areas capable of detecting continuous data of the left and right eyes based on the reference area, and continuous data appears in the left and right areas. A continuous data appearance state determining unit for determining whether or not the continuous data has appeared, and causing the eye position specifying unit to perform the detection again based on the continuous data when it is determined that the continuous data has appeared. Specifying means, a second eye reference area determining means for determining and storing an eye reference area that can be predicted to have an eye based on the eye position information specified by the re-identifying means, and tracking the eye A second step of returning the predetermined area to the reference area when the state determination means determines that the eye tracking is abnormal;
An eye condition detecting device provided with a return means. 4. The eye state detecting device according to claim 1, wherein the first or second reference region determining means repeats an eye position detection process to determine an eye reference region. An eye state detection device characterized by learning. 5. The eye state detection device according to claim 1, wherein the first or second reference area determination unit determines a time-series change state of the eye opening value. An eye state detecting device for learning a reference region of an eye by using the method. 6. The eye state detection device according to claim 1, wherein the first or second reference region determination unit performs a stable eye detection process by performing eye position detection processing a plurality of times. An eye reference area determined when the position of the eye is output. 7. The eye condition detection device according to claim 2, wherein the re-detection unit includes two continuous data that can be stably detected at predetermined intervals in the horizontal direction in the left and right regions. An eye condition detection device for performing re-detection. 8. The eye condition detecting apparatus according to claim 3, wherein the re-identifying unit is configured to detect when two continuous data which can be stably detected at predetermined intervals in the horizontal direction in the left and right regions. An eye condition detection device for performing re-detection. 9. The eye state detection device according to claim 2, wherein the determination unit extends the determination for a predetermined time when continuous data in the left and right regions cannot be detected stably. An eye condition detection device comprising: 10. The eye state detection device according to claim 2, further comprising: a determination extension unit that extends the determination by a predetermined time when continuous data in the left and right regions cannot be detected. An eye condition detection device, characterized in that: 11. The eye condition detecting device according to claim 2, wherein the continuous data in the left and right regions cannot be detected stably and the continuous data in the left and right regions is detected. A determination extending means for extending the determination for a predetermined time when the determination is not possible, wherein the determination extending means sets a longer extended time when the continuous data cannot be detected stably than when the continuous data cannot be detected stably. Detection device. 12. The eye state detection device according to claim 1, wherein the tracking state determination unit tracks the eye when continuous data in the predetermined area moves and stays at the position for a predetermined time. An eye condition detection device characterized by determining that the condition is abnormal. 13. The eye state detection device according to claim 1, wherein the tracking state determination unit determines that an eye tracking abnormality is detected when continuous data in the predetermined area cannot be detected for a predetermined time. Characteristic eye condition detection device. 14. The eye state detection device according to claim 1, wherein the tracking state determination unit tracks the eye when the opening value of the eye exceeds a predetermined range and is constantly detected. An eye condition detection device characterized by determining that the condition is abnormal. 15. The eye state detection device according to claim 1, wherein the eye opening state output by the eye opening detecting means is determined, and the degree of arousal is determined from a change in the opening / closing state. A drowsy driving alarm device, comprising: an arousal level determining means for determining; and an alarm means for issuing an alarm when the arousal level determining means determines that the arousal level is decreasing.