JP3334583B2 - Awake state detection method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a detection method for detecting an awake state by measuring a blinking eye closing time as an eyelid opening / closing operation of a vehicle driver or the like.

[0002]

2. Description of the Related Art Conventionally, various devices have been known for detecting an awake state by measuring the time of closing the eyelid blink. For example, in comparison with the reference eye closing time, the measured eye closing time is integrated for a plurality of eye closing times longer than the reference eye closing time, and when the total value is equal to or greater than a specified value within a predetermined time, it is considered a dozing state. to decide. Therefore, when the eye closing time measured for the first time is equal to or longer than the specified value, it is possible to immediately detect the dozing state. Such an apparatus is disclosed in JP-A-7-1566.
No. 82 is disclosed.

The means for measuring the eye closing time is as follows:
The above publication also discloses a means using electrodes, a means using image processing, and the like. In the EOG in which the electrodes are arranged on the face of the subject to measure the eyeball potential, the eyelid closing time for the normal blinking of the eyelid is about 200 ms (1/50 second), so that the eyelid closing time is about 1/40 of the eye closing time. 5ms early
At the sampling interval of Therefore, in the case of EOG, the graph shape of the voltage change from the full opening of the eyelid to the full opening via the full closing is accurately measured, and the slope of the rising curve from the full opening of the graph shape and from the full closing to the full opening. Is differentiated twice, a peak value of the voltage value with respect to both time axes can be obtained in a steep state, and the interval between the two is detected as the eye closing time.

However, in the above-described EOG system, arranging electrodes on the face of the driver of the vehicle is troublesome during driving. Therefore, as the blink detection device, it is desirable to dispose a camera that does not contact the subject in front of the driver's seat. The publication also discloses an example in which a face image of a subject is photographed using a CCD imaging body to detect opening and closing of eyelids. In the case of this CCD imaging body, in order to perform image processing at low cost using a normal CCD device, the sampling interval is 33.3 ms because a 30 frame / s video rate similar to a normal video signal is used. About 1/3
Images are viewed at 0 second intervals, and are not followed by shorter eyelid movements.

However, as described above, the eye closing time for a normal blink is about 200 ms (1/50 second).
When sampling is performed at intervals of 33.3 ms, the normal eye closing time can be obtained by six pieces of sampling data. The eye-closing time for inviting dozing is naturally longer than the normal eye-closing time, and should consist of six or more data, and can be used with a normal CCD image pickup body.

[0006]

Since the above-mentioned CCD imaging body performs sampling at an interval of 33.3 ms, the eyelids and / or the face are subjected to physiological vibration or vehicle vibration during this 1/30 second. The signal is delicately added and vibrates, and a high frequency component and a low frequency component are superimposed on the video signal, resulting in a complicated waveform. Therefore, since the waveform does not monotonically increase or decrease, even if the differentiation is performed twice during the closing curve by the video signal, not only two peaks but also a large number of waveforms are generated, and the accurate closing time of the eye is measured. Can not.

SUMMARY OF THE INVENTION In view of the above circumstances, an object of the present invention is to provide a method for accurately detecting an eyelid blink closing time using an eye closing curve based on a video signal.

[0008]

Means for solving the problems are as described in claims 1 to 3 . In the invention described in claim 1, “the image of the driver's eyelid open / closed state is captured by the CCD imager” may be that the entire face or half of the face of the vehicle driver may be observed.
It may be in the vicinity of the eyelids including the eyebrows and eyelids, which means that any observation pattern that can detect a voltage change by opening and closing the eyelids may be used.

For this purpose, a CCD camera is arranged at a position where the field of view of the CCD image pickup body (CCD camera) is not kicked by the steering wheel, and the entire face or at least a portion including the eyelids is observed directly or by reflection using a mirror. Configure as possible. Then, for example, as shown in FIG. 3, the CCD camera 1 is disposed on either side of the handle 22, and the angle of view 21 of the camera is such that the part 20 a including one eyelid and eyebrows is observed, and the eyes are closed. In such a case, it is desirable that the voltage is high in many white portions except for the eyebrows, and the voltage is high due to the black eye when the eye is opened, so that the output difference between the closed eye and the open eye is large.

The sampling waveform described in claim 1 can be obtained by such a CCD camera. However, during the operation of the vehicle, the observation pattern fluctuates in a long cycle due to irregularities on the road and a shock absorber function. Therefore, it is necessary to subtract these low frequency components as noise from the sampling waveform. Therefore, in the present invention, for example, as shown in a graph 3 'in FIG.
The waveform 43 'is obtained by subtracting the low frequency component 42' from the sampling waveform 40 '.

If the surface luminance of the observation pattern is constant, the low frequency component removal processing of the sampling waveform should stabilize the baseline. The time to return to the open state again has an interval of 200 ms even in the case of a normal blink, during which a difference may occur in the surface luminance of the observation pattern. When the low frequency component removal processing is performed, the voltage value of the output waveform is detected so as to be less than the one and located below the base line 45 'in the other eye open state.

Accordingly, by removing the lower envelope component of the waveform formed below the base line 45 'by using, for example, a peak hold circuit or the like, the waveform 44 as shown in the graph 6' of FIG. 'Is formed, and the base line can be further stabilized.

The low-frequency component removal processing and the lower envelope removal processing are defined as baseline stabilization processing.
Is a reference position of a formed waveform and a threshold voltage, and has an appropriate value including zero.

Then, the baseline stable waveform is selected by a predetermined threshold voltage, and the time axis of the baseline stable waveform that exceeds the eye closing time is measured to detect the eyelid closing time. Therefore, even when noise is included in the sampling waveform captured by the CCD camera of the present invention, the eye closing time can be accurately detected.

Further, as described in the second aspect, "performing the waveform shape stabilizing process on a monotonically increasing or decreasing shape" means that a jagged noise shape is obtained by superimposing a sawtooth wave on a sine wave. Means to stabilize the shape to a shape that monotonously increases or decreases like a sine wave from the waveform on which the waveform is superimposed.

[0016] The observation pattern of the vehicle driver is located when vibration due to physiological vibration of the driver, high-frequency vibration of the vehicle or the like is synthesized, and there is no synthesized vibration during a sampling time of 1/30 second. By storing a position slightly deviated from the position of the observation pattern, a waveform having a jagged noise waveform is formed, for example, as shown in graph 1 of FIG.

With this noise waveform, it is difficult to determine which position of the jagged waveform is the actual start time and the end time of closing the eyelids. Therefore, by subtracting the output of the high-frequency vibration component from this output waveform, it is possible to obtain a waveform that monotonically increases as the eyelid closes, and a shape that monotonically decreases as the eyelid opens,
A waveform having a stable shape is formed.

In this case, in order to stabilize the waveform shape, high-frequency component removal processing of the sampling waveform is performed, and a waveform formed thereby is re-sampled at predetermined time intervals. This effectively forms the waveform. When the high frequency component removal processing of the sampling waveform is performed, the jagged noise of the waveform can be eliminated. 33 ms to improve the accuracy in the time axis direction in consideration of the subsequent arithmetic processing
The resampling process is performed at shorter intervals, and the interpolation process is performed.

Further, according to these inventions, after the baseline stabilization processing, a gain correction waveform is formed by performing gain correction on the vertex of the closed-eye output of the waveform formed by the baseline stabilization processing, and thereafter, the gain correction waveform is obtained. It is preferable to detect the eyelid closing time by comparing the waveform with a predetermined threshold voltage and eye closing time.

As described above, if the noise component is superimposed due to the change in luminance of the observation pattern and the baseline stabilization process is performed, the noise component may be buried in the voltage threshold and not detected as a blinking candidate. Therefore, for example, as shown in a graph 6 'of FIG. 6, the waveforms Vt1 and Vt after the baseline stabilization processing are not so much larger than the threshold voltage Vsp.
2 is Vt1 'and Vt2' as shown in the graph 7 ', and the vertices of the output at the time of closing the waveform are corrected for gain, and rise up beyond the threshold voltage to be buried in the threshold. The eye-closing time can be accurately detected without any error.

[0021]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below in detail with reference to an embodiment shown in the drawings. However, unless otherwise specified, dimensions, materials, shapes, relative positions, and the like of the components described in this embodiment are not intended to limit the scope of the present invention, but are merely illustrative examples. Not just.

FIG. 1 is a first electrical block diagram of a control device according to the present invention, and FIG. 3 is an explanatory diagram showing a state in which a CCD camera observes an observation pattern of a face of a vehicle driver. 3, the eyelid movement detecting means 1 is constituted by a CCD camera, and is arranged in the dashboard on the left hand side of the steering wheel 22 with the observation pattern 20a of the driver's face 20 as a target as shown in FIG. And connected to the waveform shape stabilization processing means 2.

The waveform shape stabilizing means 2 comprises a low-pass filter 6 and a resampling means 7, and the low-pass filter 6 is 1/30
The high frequency component of the first waveform sampled at the second interval is cut off, and the resampling unit 7 resamples the first waveform at the interval of 5 ms. Connected to

The baseline stabilization processing means 3 comprises a low-pass filter 8 and a lower envelope extraction means 9, and the low-pass filter 8 extracts a low-frequency component of the first waveform and extracts the low-frequency component from the first waveform. The lower envelope extracting means 9 is configured by a lower peak hold circuit or the like,
By subtracting the lower envelope portion from the first waveform subtracted by the low frequency component, a second waveform is formed in which the reference baseline on the voltage axis is stabilized in the time axis direction.

The baseline stabilizing means 3 is connected to a comparator 10 constituting a first comparing means. The comparator 1
A value of 0 compares the threshold value 11 with the second waveform, and outputs a waveform exceeding the threshold value as a third waveform.

The first comparing means 4 is connected to a second comparing means 5 comprising a blinking waveform candidate detecting means 12 and a blinking waveform detecting means 13, and the blinking waveform candidate detecting means 12 determines the third waveform by a predetermined method. A waveform which is shorter than ts as compared with the time width ts is eliminated, and the blinking waveform detecting means 13 selects a waveform T satisfying tmin ≦ T ≦ tmax from a waveform exceeding the predetermined time width ts.

Next, an embodiment according to the present invention will be described with reference to the waveform diagrams of FIGS. 4 and 5 and the flowchart of FIG. In FIG. 7, sampling is performed by a CCD camera at a time interval of about 1/30 second (30). The observation pattern of the vehicle driver by the CCD camera is an observation pattern that would be positioned in the absence of the synthetic vibration during the sampling time, in which the physiological vibration of the driver, the vibration due to the high-frequency vibration of the vehicle, etc. were synthesized. In order to memorize the position slightly shifted from the position, a first waveform 40 with many jaggies is formed as shown in the graph 1 of FIG.

With this waveform, it is difficult to judge which position of the jagged waveform is the actual start time and the end time of closing the eyelid. Therefore, an oversampling filter (31) for removing high frequency components and resampling is performed. That is, first, the output due to the high frequency vibration component is subtracted from the first waveform. In addition, in order to improve the accuracy in the time axis direction in consideration of the subsequent arithmetic processing, the second sampling is performed again at intervals shorter than 33 ms, for example, at intervals of 5 ms, and interpolation processing is performed. A stabilized second waveform 41 (graph 2 in FIG. 4) is formed.

Next, a baseline stabilization process is performed by removing the extremely low frequency component (32) and removing the envelope component (33).
The base line is a basic line extending in the time axis direction indicating a completely opened state of the eyelid, and both have the same voltage value even when the eyelid changes from the open state to the closed state and returns to the open state again. It is necessary. During the operation of the vehicle, the observation pattern fluctuates at a low frequency 42 having a long cycle as shown in graph 3 of FIG. 4 due to the influence of road irregularities and shock absorber function. Accordingly, a low-frequency component is subtracted from the second waveform 41 to obtain a waveform 43 shown in the graph 4 of FIG.

If the surface luminance of the observation pattern is constant, baseline stabilization should be performed by performing the low-frequency component removal processing of the second waveform. The time to return to the eye-open state again after passing through, even in the case of a normal blink, there is an interval of 200 ms, during which a difference may occur in the surface luminance of the observation pattern,
If a difference occurs, when the baseline stabilization process is performed in one open state, the voltage value of the output waveform is detected so as to be vertically displaced from the one base line in the other open state.

Therefore, as shown in graph 4 of FIG. 4, the lower envelope component of the waveform formed below this base line is
The peak is extracted using a peak hold circuit or the like, the lower peak of the waveform is removed, and the lower peak is output at a voltage value higher than the peak value, and a waveform having a large jump as shown in a graph 6 of FIG. ) And a small waveform is formed between the waveforms, and the baseline can be further stabilized.

Then, the third waveform 44 is selected by a predetermined threshold voltage Vs (34), and further, as shown in a graph 7, T is satisfied by tmin ≦ T ≦ tmax.
1, T2 and T3 are selected as blink candidates. Further, the time axis of the waveforms of T1 to T3 is measured, and the T2 waveform exceeding a predetermined eye closing time is detected as the eyelid closing time of the eyelid to be careful.

As described in detail above, in the case of the first embodiment, although the first sampling waveform is not simple, the waveform shape stabilization processing means 2, the baseline stabilization processing means 3, and the first,
The eye closing time can be accurately detected by the second comparing means 4 and 5. Then, since changes in the eyelids are observed and processed by the CCD camera, the electrodes and the like do not come into contact with the driver's face, and there is no bother.

Next, a second embodiment according to the present invention will be described. FIG. 2 is a second electrical block diagram of the control device according to the present invention, and the same parts as those in FIG. 1 are denoted by the same reference numerals. FIG. 6 is a waveform diagram, FIG. 8 is a flowchart, and FIG.
9 is a flowchart showing a subroutine at the time of gain correction.
The difference from the first embodiment is that in the second embodiment, the waveform shape stabilizing means 2 in the first embodiment is different from the first embodiment.
And between the baseline stabilization processing means 3 and the first comparison means 4, the vertex of the output of the baseline stabilization processing means 3 when the eyes are closed is gain-corrected, and the vertex of each waveform is gained up. The point is that gain correction means for forming the third waveform is provided.

The gain correction means 14 samples a waveform within a predetermined time Ts (for example, n seconds) and performs a baseline stabilization process on a waveform 44 'having a plurality of peaks and valleys (see graph 6' in FIG. 6). Is temporarily stored, an average amplitude value Aave of the peak voltage value Vk larger than the temporary threshold voltage Vsp previously obtained from the experimental data is obtained, and 1 / Aave = G
(Gain), the waveform 46 within the predetermined time Ts is multiplied by the gain G, and the waveform 46 including Vt ′ and Vk ′ (graph 7 ′ in FIG. 6)
Cf.). Then, the temporary threshold Vsp is updated by the following equation: Vsp = Aave × K
For example, (1 / Vave) = K = 1/2 = 0.5
Is set.

Next, a second embodiment according to the present invention will be described with reference to the waveform diagram of FIG. 6 and the flowchart of FIG. In FIG. 8, sampling is performed at a time interval of about 1/30 second by the CCD camera (50), and a first waveform 40 'shown in a graph 3' of FIG. 6 is formed.

Next, a baseline stabilization process is performed by removing the extremely low frequency component (51) and the envelope component (52), and a long period from the first waveform 40 'as shown in the graph 3' of FIG. Is subtracted from the low frequency component 42 'to obtain a waveform 43' shown in a graph 4 '.

Then, a lower envelope component formed below the base line 45 'of the waveform 43' is extracted by using a peak hold circuit or the like, and the lower peak of the waveform is removed. A high voltage is output and a graph 6 'in FIG. 6 is output.
The waveform 44 'shown in FIG. 7 is formed, and the baseline can be stabilized. When the waveform 44 'subjected to the baseline stabilization processing is obtained, the process proceeds to the gain correction step 53.

In the flow chart 9 of the gain correction step 53, of the waveform 44 ', a blink waveform Vk1, Vk2, Vk3, Vk4 (graph 6' in FIG. 6) larger than the temporary threshold Vsp is applied for a predetermined time Ts (two seconds). (60),
The extracted blink waveforms Vk1, Vk2, Vk3, Vk4
The average amplitude Aave of the amplitudes A1, A2, A3, and A4 of Aave = (A1 + A2 + A3 + A4) / N, N =
4 (61), and the reciprocal of this Aave is (1 / Aa
ve) = G (gain) (62).

The waveform Vt of the past predetermined time Ts (2 seconds)
The waveforms Vt1, Vk1, Vk2, Vk3, Vk4, and Vt2 are multiplied by the G (gain) (63), and gain-up waveforms Vt1 'and Vk shown in graph 7' of FIG.
A waveform 46 shown in a graph 7 'having 1', Vk2 ', Vk3', Vk4 ', and Vt2' is obtained, and Vsp = Aave
The temporary threshold value Vsp is updated (64) from × K, and the process proceeds to step 54 in FIG.

Then, the waveform 46 is selected based on a predetermined threshold voltage Vs (54), and further the graph 7 'is selected.
As shown in (2), T ′ that satisfies tmin ≦ T ≦ tmax is selected as a blink candidate. Further, the time axis of the waveform of T 'is measured, and a waveform exceeding a predetermined eye closing time is detected as the eye closing time of the eyelid to be careful.

As described in detail above, the second embodiment includes the gain correcting means for forming the waveform by correcting the vertex of the closed-eye output of the baseline stabilizing processing means. When the noise component is superimposed due to the luminance change of the observation pattern and the baseline stabilization process is performed, the waveform component which is buried in the voltage threshold and is not detected as a blink candidate is gain-corrected, and FIG. As shown in the graph 7 ', the waveform can be formed upright beyond the threshold voltage. Thus, the eyelid closing time can be accurately detected.

The gain correction means 14 used in the second embodiment can be provided between the baseline stabilization processing means 3 and the first comparison means 4 in the first embodiment. As a result, a waveform buried in the threshold voltage due to a change in luminance of the observation pattern surface by the CCD camera can be detected early.

[0044]

FIG. 6 shows a low-frequency waveform 4 as shown in a graph 3 '.
When the width of tb is measured using 2 'as a threshold value and the eye closing time is set, the tb below it is actually the eye closing time, and ta
Is larger than tb and ta> tb. And
If the eye closing ratio R is defined as R = tb / ta, the slope of the waveform becomes gentle because the time from full opening to full closing becomes slower as the arousal level decreases, and the eye closing ratio R becomes
Become smaller.

Therefore, assuming that the proportionality constant is α, the estimated arousal level L and the closed eye ratio R are in a proportional relationship of L = αR, and the estimated arousal level L can be obtained from the closed eye ratio R. Estimated arousal Le measured by EOG with proportional constant α = 1
og, the estimated awakening degree Lg with gain correction and the estimated awakening degree Ln without gain correction in the second embodiment of the present application. In the early stage of driving, Leog: 67.3, Ln: 91.
8, Lg: 63.6, when arousal level is low, Leog: 30.9, Ln: 87.
6, Lg: 52.7.

From this result, in the case of the estimated arousal level Ln without gain correction, the estimated arousal level Leg with gain correction is much higher than the estimated arousal level Leog measured by the highly accurate EOG. , A value close to the value of the estimated alertness Leog measured by EOG,
It can be seen that the arousal level estimation accuracy is improved.

As described in detail above, in both the first embodiment and the second embodiment, a sampling waveform due to the blinking of the observation pattern including the eyelids is obtained by the CCD camera, and this sampling waveform is subjected to low-frequency component removal and , After removing the lower envelope component of the waveform formed thereby to obtain a baseline stable waveform that forms a waveform only above the baseline extending in the time axis direction at the reference position on the voltage axis, Since the eyelid closing time is detected by measuring the time axis of the baseline stable waveform that exceeds the value voltage and the eye closing time, even if the sampling waveform by the CCD camera is not simple, the eye closing time is accurately detected. be able to.

[0048]

As described above, according to the first aspect of the present invention, even when the sampling waveform obtained by the CCD camera is not simple, the eye closing time can be accurately detected. The invention described in claim 2 is a C
Since the waveform shape stabilization processing is performed on the sampling waveform by the CD camera before the baseline stabilization processing, a waveform having a stable shape can be obtained, and the eye closing time can be detected more accurately.

According to a third aspect of the present invention, the high-frequency component removal processing of the sampling waveform by the CCD camera is performed, and the resolution of the time axis is increased by the re-sampling processing of performing the waveform formed at a predetermined time interval. Therefore, a more accurate eye closing time can be obtained.

According to a fourth aspect of the present invention, after the baseline stabilization processing, a gain-corrected waveform in which the vertex of the closed-eye output of the waveform formed by the baseline stabilization processing is gain-corrected is formed. Since the eye-closing time of the eyelid is detected by comparing the gain correction waveform with a predetermined threshold voltage and eye-closing time later, the luminance change of the observation pattern by the CCD camera despite the long eye-closing time. Accordingly, a waveform in which the noise component is superimposed and buried in the threshold voltage can be detected.

[Brief description of the drawings]

FIG. 1 is an electrical block diagram showing a first embodiment according to the present invention.

FIG. 2 is an electrical block diagram showing a second embodiment according to the present invention.

FIG. 3 is an explanatory diagram showing how a CCD camera observes an observation pattern of a face of a vehicle driver.

FIG. 4 is an explanatory diagram of a first half showing a waveform forming state according to the first embodiment.

FIG. 5 is an explanatory diagram of the latter half showing a waveform forming state according to the first embodiment.

FIG. 6 is an explanatory diagram showing a waveform forming state according to a second embodiment.

FIG. 7 is a flowchart illustrating the operation of the first embodiment.

FIG. 8 is a flowchart illustrating the operation of the second embodiment.

FIG. 9 is a flowchart showing a subroutine at the time of gain correction.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Eyelid movement detection means 2 Waveform shape stabilization processing means 3 Baseline stabilization processing means 4 First comparison means 5 Second comparison means

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-9-277849 (JP, A) JP-A-6-270711 (JP, A) JP-A 8-153288 (JP, A) JP-A-9-278 147120 (JP, A) JP-A-63-258226 (JP, A) JP-A-2-57433 (JP, A) JP-A-6-333183 (JP, A) JP-A-63-46928 (JP, A) 63-125624 (JP, U) (58) Fields investigated (Int. Cl. ⁷ , DB name) B60K 28/06 G06T 1/00 G08B 21/00 G08G 1/16 H04N 7/18

Claims (3)

(57) [Claims] 1. An image of an eyelid opening / closing state of a driver or another observed person is taken by a CCD imaging body, and after obtaining a sampling waveform in which a temporal change due to eyelid blinking is captured by a voltage change so that the eyelid closing becomes a peak value. The sample waveform is subjected to baseline stabilization processing by removing low-frequency components and removing the lower envelope component of the waveform formed thereby, and the reference position on the voltage axis is completely closed by the eyelids.
Taking the voltage value indicating the state, give the group <br/> line stable waveform from baseline formed by extending the voltage value in the time axis direction takes the waveform of only the upper side, from the base line stability waveform An awake state detection method, comprising: measuring a time axis of the baseline stable waveform that exceeds a predetermined threshold voltage and an eye closing time to detect an eyelid closing time. 2. The method according to claim 1, wherein the sampling waveform is converted from a waveform having a jagged noise shape before the baseline stabilization processing.
2. The awake state detecting method according to claim 1, wherein a waveform shape stabilization process is performed on a shape that is monotonically increased or decreased like a sine wave by removing a noise shape . 3. The waveform shape stabilizing process is a re-sampling process in which a high-frequency component removal process of the sampling waveform is performed, and a waveform formed thereby is performed at predetermined time intervals. The arousal state detection method described in the above.