JP5262819B2 - Awakening level estimation device - Google Patents

Description

  The present invention relates to an arousal level estimation apparatus that estimates an awakening level of an operator.

  Based on the frequency distribution of blinking time in the initial stage of driving, a threshold for determining long blinking is set, and the ratio of the number of long blinking to the total number of blinking within a predetermined time is calculated. A technique for detecting a decrease is known (see, for example, Patent Document 1).

JP-A-9-277849

  In the above technique, since there is a single threshold value for determining long blinks, for example, long closed eyes that appear when the degree of arousal is greatly reduced and a little late that appears when the degree of reduction in arousal is small There was a problem that it was not possible to distinguish between blinks and it was not suitable for estimating the arousal level in multiple stages.

  The problem to be solved by the present invention is to provide a wakefulness estimation device capable of multistage wakefulness estimation with high accuracy.

  The present invention classifies the closed eye measurement time into a plurality of closed eye time ranges defined according to the arousal level, calculates a determination value according to the arousal level according to a determination value calculation method according to the arousal level, The above-mentioned problem is solved by determining to which awakening level the awakening level of the operator belongs based on the determination value. Although not particularly limited, the operator includes, for example, a driver who drives a vehicle, an operator (operator) who operates an electronic device such as a personal computer, and a precision device.

  According to the present invention, the eye-closing time range is defined according to the arousal level, and the determination value is calculated using the calculation method according to the arousal level, so that the determination accuracy is improved at each arousal level. It is possible to estimate the multi-stage arousal level with high accuracy.

FIG. 1 is a block diagram showing the overall configuration of the arousal level estimation apparatus according to the first embodiment of the present invention. FIG. 2 is a perspective view showing the installation position of the camera in the first embodiment of the present invention. FIG. 3 is a circuit diagram showing another example of the open / close eye detection device according to the first embodiment of the present invention. 4 is a perspective view showing an installation position of a light emitting diode of the open / close eye detection device shown in FIG. FIG. 5 is a conceptual diagram showing the correspondence between the eye-closing time range and the arousal level in the first embodiment of the present invention. FIG. 6 is a graph showing an example of a change in the closed eye rate in the process in which the arousal level decreases with time. FIG. 7 is a flowchart (part 1) showing the processing of the arousal level estimation apparatus according to the first embodiment of the present invention. FIG. 8 is a flowchart (part 2) showing the process of the arousal level estimation apparatus according to the first embodiment of the present invention. FIG. 9 is a flowchart showing a subroutine of step S101 (blink learning) in FIG. FIG. 10A is a graph showing an example of a change in the appearance frequency of the eye closing time with the lapse of time of the operator X, and is a graph showing only eyes closed for A to B seconds. FIG. 10B is a graph showing an example of a change in the appearance frequency of the eye closing time with the passage of time of the operator X, and is a graph showing eye closing for C seconds or more. FIG. 11A is a graph showing an example of a change in the appearance frequency of the eye closing time with the lapse of time of the operator Y, and is a graph showing only eyes closed for A to B seconds. FIG. 11B is a graph showing an example of a change in the appearance frequency of the eye closing time with the passage of time of the operator Y, and is a graph showing eye closing for C seconds or more. FIG. 12A is a graph showing an example of a change in the appearance frequency of the eye-closed time after correction of the operator Y, and is a graph showing only the eyes closed for A to B ′ seconds. FIG. 12B is a graph showing an example of a change in the appearance frequency of the eye closing time after correction of the operator Y, and is a graph showing the eye closing for C ′ seconds or more. FIG. 13A is a graph showing an example when the frequency of appearance of the eye-closing time is counted every 30 seconds, and is a graph showing only the blinking time in normal times. FIG. 13B is a graph illustrating an example when the frequency of appearance of the eye closing time is counted every 30 seconds, and is a graph illustrating the eye closing time longer than the blinking in normal times. FIG. 14 is a graph showing an example when the frequency of appearance of the closed eye time is counted every minute. FIG. 15 is a graph showing an example when the frequency of appearance of the closed eye time is counted every two minutes. FIG. 16 is a graph showing an example when the frequency of appearance of the closed eye time is counted every 5 minutes. FIG. 17 is a graph showing an example when the frequency of appearance of the closed eye time is counted every 10 minutes. FIG. 18 is a graph showing an example when the frequency of appearance of the closed eye time is counted every 15 minutes. FIG. 19 is a graph for comparing the case where the closed eye time is expressed by the frequency of appearance and the case where the closed eye rate is expressed by the closed eye rate. FIG. 20 is a graph for comparing the case where the eye closure time is represented by the frequency of appearance, the case where the eye closure time is represented by the eye closure rate, and the case where the eye closure time is represented by a weight value. FIG. 21 is a conceptual diagram illustrating the processing in steps S111 and S117 of the flowcharts shown in FIGS. FIG. 22 is a graph for explaining the processing in steps S112 and S118 of the flowchart shown in FIG. FIG. 23 is a graph for explaining the processing in steps S114 and S120 of the flowchart shown in FIG. FIG. 24 is a block diagram showing the overall configuration of the arousal level estimation apparatus according to the second embodiment of the present invention. FIG. 25 is a graph showing an example of open / closed eye misjudgment due to lack of extension. FIG. 26 is a graph showing an example of synchrony between the aspect ratio of the mouth and the opening and closing of the eyes in the absence of extension. FIG. 27 is a flowchart showing a first process performed by the arousal level estimation apparatus according to the second embodiment of the present invention. FIG. 28 is a graph for explaining a method of setting an exclusion period in the second embodiment of the present invention. FIG. 29 is a flowchart showing a second process performed by the awakening level estimation device according to the second embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
FIG. 1 is a block diagram showing the overall configuration of the arousal level estimation apparatus in this embodiment, FIG. 2 is a perspective view showing the installation position of the camera in this embodiment, and FIG. 3 is another example of the open / closed eye detection apparatus in this embodiment. FIG. 4 is a perspective view showing an installation position of a light emitting diode of the open / close eye detection device of FIG.

  As shown in FIG. 1, the arousal level estimation device 1 according to the present embodiment includes an open / closed eye detection device 110, a closed eye time measurement circuit 120, and a wakefulness level determination circuit 130. This arousal level estimation device 1 can estimate the driver's arousal level (awakening level) in the following four stages.

(I) The first stage of the first arousal level (the degree of low arousal level is “small”)
(II) Late stage of the first arousal level (the degree of low arousal level is “small”)
(III) Early stage of the second arousal level (the degree of arousal level decrease “Large”)
(IV) Late stage of the second arousal level (the degree of arousal level decrease “Large”)

  The first arousal level (arousal level decrease degree “small”) is a level where the driver's arousal level is lower than the normal arousal state, and the second arousal level (arousal level decrease level “large”) is The driver's arousal level is lower than the first arousal level. In addition, the early stage of the arousal level is a stage in which the driver's degree of arousal level is relatively small among the arousal levels, and the latter stage of the arousal level is the first stage in the same arousal level. It is a stage where the degree of decrease in the driver's arousal level is greater than that. Note that the number of stages of the arousal level is not particularly limited. Moreover, although this embodiment demonstrates the example using the awakening degree estimation apparatus 1 about the driver who drives a vehicle, it is awakening in this example about the operator (operator) etc. who operate electronic devices, such as a personal computer, or precision equipment. The degree estimation device 1 may be used.

  The open / close eye detection device 110 is a device that detects opening / closing of the eyes of the driver 300, and includes a camera 111 and an image processing circuit 112. The camera 111 has an image sensor such as a CCD or a CMOS, and as shown in FIG. 2, for example, is installed in front of the instrument panel so that the face 310 of the driver 300 can be imaged. The image processing circuit 112 performs image processing on the face image captured by the camera 111, thereby detecting the position of the eyes of the driver 300 in the face image and determining whether to open or close the eyes. Note that a specific processing method for eye position detection and open / close determination is described in Japanese Patent Laid-Open No. 10-275212, and a description thereof will be omitted here.

  Instead of the image processing technique, as shown in FIG. 3, the open / close eye detection device may include a light emitting diode 211, a photodiode 212, and an amplifier 213. In this method, as shown in FIG. 4, near-infrared rays are irradiated to the eyes of the driver 300 by the light emitting diodes 211 attached to the lower part of the spectacle frame 200, and the difference in reflectance between the upper eyelid and the eyeball part is measured. Is sensed as a difference in current value, and this difference is amplified by the amplifier 213 to detect the open / closed eye.

  Returning to FIG. 1, the eye-closing time detection circuit 120 has a timer (not shown) that is turned on / off when the eye of the driver 300 detected by the opening / closing eye detection device 110 is closed and opened. The eye-closing time detection circuit 120 turns on the timer when the eye-opening eye detection device 110 detects the eye-closing and starts timing, and when the eye-opening eye detection device 110 detects the eye opening, turns the timer off and ends the time counting. The time (eye closing time) from the eyes closed to the eyes of the driver 300 is measured.

  As shown in FIG. 1, the arousal level determination circuit 130 functionally includes a normal eye-closing time setting unit 131, a range defining unit 132, a classification unit 133, a determination value detection unit 134, and a determination unit 135.

  Based on the closed eye time (closed eye measurement time) measured by the closed eye time measurement circuit 120, the normal closed eye time setting unit 131 sets the driver's closed eye time (normal closed time) in the normal arousal state (normal state). . The normal eye closing time is the eye closing time associated with the normal blink of the driver in the normal awake state. Since there is a large individual difference in the length of normal blinking, in this embodiment, the normal eye-closing time is set for each individual by the normal eye-closing time setting unit 131 to absorb the individual difference.

  The range defining unit 132 defines a plurality of closed eye time ranges according to the arousal level. In the present embodiment, as shown in FIG. 5, the range defining unit 132 corresponds to the first arousal level, and includes a first eye-closing time range R1 configured with eye-closing time of C to F seconds, A second eye-closing time range R2 corresponding to an arousal level of 2 and composed of eye-closing time of E seconds or more is defined. In addition, the range defining unit 132 determines the lower limit value of the first eye closing time range R1 by excluding the eye closing time (A to B seconds) less than or equal to the normal eye closing time from the first eye closing time range R1. FIG. 5 is a conceptual diagram showing the correspondence between the eye-closing time ranges R1 and R2 and the arousal level in the present embodiment.

  As described above, the first eye-closing time range R1 corresponding to the first arousal level having a relatively high arousal level is configured with a relatively short eye-closing time (C to F seconds), and the second low in arousal level. By configuring the second eye closing time range R2 corresponding to the level of arousal level with a relatively long eye closing time (E seconds or more), multistage arousal level estimation can be performed with high accuracy. Note that the eye closing times A to O seconds used in the present embodiment have a relationship of increasing time in alphabetical order.

  In the present embodiment, as shown in FIG. 5, the upper limit value (F seconds) of the first eye closing time range R1 is larger than the lower limit value (E seconds) of the second eye closing time range R2. (F> E), the first closed eye range R1 and the second closed eye range R2 partially overlap. For this reason, multistage arousal level estimation can be performed with higher accuracy. In the present embodiment, the upper limit value of the first eye closing time range R1 and the lower limit value of the second eye closing time range R2 are set as fixed values set in advance by experiments or the like. These may be variable depending on the situation.

  The classification unit 133 classifies the closed eye time measured by the closed eye time measurement circuit 120 into the first closed eye time range R1 or the second closed eye time range R2.

  The determination value calculation unit 134 calculates a determination value according to the arousal level from the closed eye time classified by the classification unit 133 according to the determination value calculation method defined according to the arousal level.

  For example, when the first determination value according to the first arousal level is calculated, the determination value calculation unit 134 accumulates the closed eye time classified into the first eye-closing time range R1 by the classification unit 133. Is calculated as the first determination value.

  On the other hand, when calculating the second determination value according to the second arousal level, the determination value calculation unit 134 calculates the closed eye time classified into the second eye closing time range R2 by the classification unit 133 as the time. A value weighted according to the length is substituted, and the cumulative value of the eye closing time after the substitution is calculated as the second determination value.

  Here, FIG. 6 is a graph showing an example of a change in the closed eye rate in the process in which the arousal level decreases with time. The solid line in the figure is the cumulative eye closure rate for every 30 seconds by accumulating all eye closure times, and the alternate long and short dash line in the figure is the cumulative eye closure time excluding the blinking time. The rate is calculated. As shown in the figure, the eye closure rate does not increase unilaterally as the degree of arousal decreases, but gradually increases while repeating a large increase / decrease. It is difficult to accurately estimate the degree in multiple stages.

  On the other hand, in the present embodiment, since the determination value is calculated using the calculation method according to the arousal level as described above, it is possible to improve the determination accuracy at each arousal level and to increase the accuracy with high accuracy. The awakening level of the stage can be estimated.

  Based on the determination value calculated by the determination value calculation unit 134, the determination unit 135 determines to which wakefulness level (above (I) to (IV)) the driver's wakefulness belongs. At this time, the determination unit 135 determines whether or not it belongs to the previous stage of the arousal level based on a single determination value at the same arousal level, and based on the plurality of determination values It is determined whether it belongs to the later stage.

  Next, the process of the arousal level estimation apparatus 1 in the present embodiment will be described. 7 and 8 are flowcharts showing the processing of the arousal level estimation apparatus in this embodiment, and FIG. 9 is a flowchart showing a blink learning subroutine.

  When the processing by the arousal level estimation device 1 is started, first, in step S101 in FIG. 7, learning of a normal blinking time that is different for each driver is executed.

  Here, the necessity of learning the blinking time will be described with reference to FIGS. 10A to 12B. FIG. 10A to FIG. 11B divide the eye closure time measured in the process of the arousal level decreasing with the passage of time for the operators X and Y by a certain time (A to B seconds: blink equivalent time, C to D seconds). , E to F seconds,...), And the frequency of appearance of each section per minute.

  FIG. 10A shows only the frequency of appearance of the eyes of the operator X from A to B seconds, and FIG. 10B shows the frequency of appearance of the eyes of the same operator X for C seconds or more. As shown in FIG. 10B, as the awakening level of the operator X decreases, the appearance of a slightly late eye closing time increases, and it is possible to confirm a tendency that gradually longer eye closing occurs gradually.

  On the other hand, FIG. 11A shows only the frequency of appearance of the eyes of the operator Y for A to B seconds, and FIG. 11B shows the frequency of appearance of the eyes of the same operator Y for C seconds or more. As shown in the XI part of FIG. 11B, many closed eyes of C to D [seconds] that are considered to be slightly late blinking immediately after the start of measurement are generated. Comparing FIG. 10B and FIG. 11B, it can be seen that the normal blink time of the operator Y is longer than the normal blink time of the operator X, and there is an individual difference in the normal blink time.

  Therefore, between B and C so that the time length of C to D seconds appearing in 5 minutes (part XI in FIG. 11B) immediately after the start of measurement in FIG. 11B is included in the normal blink time of the operator Y. When the threshold is corrected to B′−C ′, the change in the appearance frequency of the closed eye time with the lapse of time of the operator Y is as shown in FIGS. 12A and 12B. Note that FIG. 12A shows only the appearance frequency of the eyes of the operator Y from A to B ′ seconds, and FIG. 12B shows the appearance frequency of the eyes C or more of the same operator Y.

  As can be understood from FIG. 12B, by changing the threshold value between B and C, the change in the appearance frequency of the eye-closing time with the lapse of time of the operator Y also shows the same tendency as the operator X shown in FIG. 10B. It becomes like this. By such correction, it is possible to absorb individual differences in the blinking time during normal times, and it is possible to accurately grasp the arousal level.

  Hereinafter, the subroutine of step S101 (learning of blinking time) in FIG. 7 will be described with reference to FIG.

  In step S201 of FIG. 9, the arousal level determination circuit 130 starts measuring the learning period (first predetermined time) using a timer (not shown), and determines whether or not the learning period has elapsed in step S202. The learning period is, for example, 5 minutes immediately after the driver gets on.

  If it is determined in step S202 that the learning period has not elapsed (NO in step S202), in step S203, the arousal level determination circuit 130 captures the closed eye measurement time measured by the closed eye time measurement circuit 120.

  Next, in step S204, it is determined whether or not the closed eye measurement time is a second or less. The a seconds is a time value shorter than b seconds (step S207) and c seconds (step S212), which will be described later (a <b, c). In this step S204, the driver's normal blink is relatively small. It is determined whether or not it is fast.

  If it is determined in step S204 that the closed eye measurement time is a second or less (YES in step S204), it is determined in step S205 whether the normal closed eye time is not set.

  If it is determined in step S205 that the normal closing time has not been set (YES in step S206), in step S206, α seconds corresponding to the a seconds are set as the normal closing time, and then in step S202. Return.

  On the other hand, if it is determined in step S205 that the normal closed time has already been set (NO in step S206), the process returns to step S202, and as long as the closed eye measurement time is equal to or less than a second, steps S203 to S205 are performed. Repeat the loop.

  For example, when a slightly late blink is mixed even during normal times like the above-mentioned operator Y, the eye-closed measurement time exceeds a second. Therefore, in this embodiment, if it is determined that the closed eye measurement time is longer than a seconds (NO in step S204), it is determined in step S207 whether the closed eye measurement time is equal to or less than b seconds. This b seconds is a time value shorter than c seconds (step S212) described later (b <c).

  If it is determined in step S207 that the closed eye measurement time is less than b seconds (YES in step S207), the number of appearances of the closed eye measurement time between a and b seconds is counted up (step S208), and step S209 is performed. In step S1, it is determined whether the number of appearances is greater than the first predetermined number. As described above, it is possible to suppress erroneous determination of blinking in a normal state by setting the appearance of multiple blinks as a condition. Note that a to b seconds in the present embodiment correspond to an example of a predetermined time range in the present invention, and the first predetermined number of times in the present embodiment corresponds to an example of a predetermined number of times in the present invention.

  If it is determined in step S209 that the number of appearances is greater than the first predetermined number (YES in step S209), it is further determined in step S210 whether the normal eye-closing time is not set or is set to α seconds. . If the normal eye-closing time is not set or is set to α seconds (YES in step S210), β seconds corresponding to b seconds are set as the normal eye-closing time in step S211, and thereafter The process returns to step S202.

  On the other hand, when the number of appearances of the closed eye measurement time between a seconds and b seconds is equal to or less than the first predetermined number (NO in step S209), β seconds and γ seconds (described later) are set as the normal eye closure time. If it is determined (NO in step S210), the normal eye-closing time is not set to β seconds, and the process returns to step S202.

  For example, when blinking during normal times is slower than the above-described operator Y, the eye-closed measurement time exceeds b seconds. Therefore, in this embodiment, when it is determined that the closed eye measurement time is longer than b seconds (NO in step S207), it is determined in step S212 whether the closed eye measurement time is c seconds or less. This c second is set to a maximum value that takes into account individual differences in the normal blink time, and even if a closed eye measurement time larger than c seconds appears, the blink learning in this embodiment is particularly processed. No (NO in step S212).

  If it is determined in step S212 that the closed eye measurement time is equal to or shorter than c seconds (YES in step S212), the number of appearances of the closed eye measurement time between b seconds and c seconds is counted up (step S213). In S214, it is determined whether or not the number of appearances is greater than a second predetermined number. As described above, an erroneous determination of blinking in a normal state can be suppressed by setting the appearance of multiple blinks as a condition. Note that b to c seconds in the present embodiment correspond to an example of the predetermined time range in the present invention, and the second predetermined number of times in the present embodiment corresponds to an example of the predetermined number of times in the present invention.

  In general, as the blinking time is delayed, the number of corresponding drivers decreases. Therefore, the second predetermined number in step S214 is set to be larger than the first predetermined number in step S209 described above. Thereby, the misjudgment of the blink in normal time can further be suppressed.

  If it is determined in step S214 that the number of appearances of the eye-closed measurement time between b seconds and c seconds is greater than the second predetermined number (YES in step S214), γ corresponding to c seconds is determined in step S215. Second is set as the normal eye-closing time, and then the process returns to step S202.

  On the other hand, when it is determined in step S214 that the number of appearances of the eye-closed measurement time between b seconds and c seconds is equal to or less than the second predetermined number (NO in step S214), the normal eye-close time is set to γ seconds. Without returning, step S202 is returned to.

  If it is determined in step S202 that the learning time has elapsed while repeating the subroutine shown in FIG. 9 (YES in step S202), the normal set time set in step S206, S211 or S215 is determined in step S216. Then, the blink learning subroutine is completed. If the driver's blink time has already been learned, the blink learning shown in FIG. 9 (that is, step S101 in FIG. 7) may be omitted.

  The range defining unit 132 of the arousal level estimation circuit 130 defines the first and second closed eye time ranges R1 and R2 as shown in FIG. 5, and sets the normal closed time set as described above to the first closed eye. Excluding from the time range R1, the lower limit value of the first eye-closing time range R1 is determined.

  Returning to FIG. 7, when the blink learning is completed, in step S102, the arousal level determination circuit 130 starts measuring the determination period (second predetermined time) by a timer (not shown), and the determination period elapses in step 103. Determine whether or not. The determination period is 2 minutes or less, preferably 30 seconds to 1 minute.

  Here, the reason for setting the length of the determination period will be described with reference to FIGS. 13A to 18. FIG. 13A to FIG. 18 are graphs showing an example in which the same closed eye appearance frequency data is counted for different determination periods. 13A and 13B are 30 seconds, FIG. 14 is 1 minute, FIG. 15 is 2 minutes, FIG. 16 is 5 minutes, FIG. 17 is 10 minutes, and FIG. 18 is 15 minutes. In FIGS. 14 to 18, the appearance frequency of the normal blink time (A to B seconds) is omitted, and only the appearance frequency of the eye closure time longer than the normal blink is illustrated.

  As shown in FIGS. 13A to 18, the longer the determination period, the easier it is to grasp the correlation between the decrease in wakefulness and the longer increase in eye-closing time. There is a problem that the timing is delayed. Therefore, in this embodiment, in order to estimate the arousal level at an early stage, as described above, the determination period is set to 2 minutes or less, preferably 30 seconds to 1 minute.

  Returning to FIG. 7, when it is determined in step S103 that the determination period has not elapsed (NO in step S103), the arousal level determination circuit 130 is measured by the eye-closing time measurement circuit 120 in step S104. Capture the closed eye measurement time.

  Next, in step S105, it is determined whether or not the closed eye measurement time belongs to the first closed eye time range R1 (C to F seconds). If it is determined in step S105 that the closed eye measurement time does not belong to the first closed eye time range R1 (NO in step S105), the process proceeds to step S108.

  On the other hand, if it is determined in step S105 that the closed eye measurement time belongs to the first closed eye time range R1 (YES in step S105), the first determination corresponding to the first arousal level is performed in step S106. The value is calculated based on the closed eye measurement time. In step S106, the first determination value is calculated by accumulating the eye closure measurement times that occur within the determination period and that belong to the first eye closure time range R1.

  When the degree of decrease in the arousal level is small, as shown in FIG. 14, it starts to occur from C to F seconds, which is a slightly late eye closing time. Therefore, in the present embodiment, when the degree of decrease in the arousal level is small, only the eye closure time of C to F seconds is an accumulation target, so that the estimation accuracy is improved without being affected by a long eye closure of G seconds or more. be able to.

  In the present embodiment, since it is not affected by the long eye-closing time as described above, as shown in the XIXA part and the XIXB part of FIG. 19, the closed eye rate is calculated using the cumulative value of the eye-closing time rather than the appearance frequency. It is possible to perform estimation with higher sensitivity when handled. FIG. 19 is a graph for comparing the case where the closed eye time is expressed with the appearance frequency and the case where the closed eye time is expressed with the closed eye rate, and FIG. 19A is a graph where the closed eye time is expressed with the appearance frequency. FIG. 19B is a graph showing the same data as FIG. 19A in terms of the closed eye rate.

  Next, in step S107, the first determination value calculated in step S106 is stored in a memory (not shown).

  Next, in step S108, it is determined whether or not the closed eye measurement time belongs to the second closed eye time range R2 (E seconds or more). If it is determined in step S108 that the closed eye measurement time does not belong to the second closed eye time range R2 (NO in step S108), the process returns to step S103.

  On the other hand, if it is determined in step S108 that the closed eye measurement time belongs to the second closed eye time range R2 (YES in step S108), the second determination corresponding to the second arousal level is performed in step S109. The value is calculated based on the closed eye measurement time.

  In this step S109, among the closed eye measurement times that occur within the determination period, those belonging to the second closed eye time range R2 are divided into fixed times as shown in Table 1 below, and according to the time length of each division. The second determination value is calculated by substituting each weighted measurement time with the weighted value and calculating the cumulative value of the replacement value.

  In Table 1 above, representative eye closure times for each eye closure time category are set to 0.4 to 1.2 seconds and ∞ for convenience, and the eye closure time is longer than the standard with 1 second eye closure as a reference. (In Table 1, “M to N” category and “O to” category), the value after replacement is set longer than the actual closed eye measurement time. On the other hand, for the sections with shorter eye closure time than the reference ("E to F" section, "G to H" section, and "I to J" section in Table 1), values after replacement than the actual closed eye measurement time Is set short. In general, when the driver is awake enough to struggle with drowsiness, the eyes are closed for about 1 second. Therefore, in this embodiment, the above standard is set to 1 second. The 1-second closed eye in the present embodiment corresponds to an example of a third predetermined time in the present invention, but the third predetermined time is not particularly limited to 1 second.

  As described above, for a section with a long eye closing time, the effect of erroneously determining that the eye movement as shown in Table 2 below is closed can be reduced as much as possible by shortening the replacement value from the actually measured value. it can. In addition, for sections with a short eye closure time, the detection sensitivity can be increased by making the replacement value longer than the actual measurement value.

  In general, a slightly longer closed eye appears more frequently than one very long closed eye appears in a situation where the driver is not conscious of lowering the arousal state. Thus, in the present embodiment, by converting the actual measurement value to the replacement value as described above, it is easy to detect a slightly longer closed eye that frequently appears while minimizing the influence of a long eye that becomes noise.

  For example, FIGS. 20A to 20C are all graphs based on the same eye-closing time data, FIG. 20A is a graph simply representing the frequency of appearance for each category, and FIG. FIG. 20C is a graph represented by the above-described replacement value in this example. FIG. 20C is a graph representing the closed eye rate calculated by accumulating the eye closure time. When comparing FIGS. 20A to 20C, it is the calculation method of the present embodiment that makes it easy to detect a slightly long closed eye that frequently appears while minimizing the influence of a long closed eye that causes noise. I understand.

  Returning to FIG. 7, when the calculation of the second determination value in step S109 is completed, in step S1110, the second determination value is stored in the memory.

  The above steps S103 to S110 are repeated until the determination period elapses. When the determination period elapses (YES in step S103), the process proceeds to step S111, and as shown in FIG. The second determination value is updated with the first determination value and the second determination value calculated in the current determination period. FIG. 21 is a conceptual diagram for explaining the processing in steps S111 and S117.

  Next, in step S112 in FIG. 8, it is determined whether or not the first determination value calculated in the latest single determination period is greater than the first threshold value TH1. As shown in FIG. 22, when it is determined in step S112 that first determination value TH1 is larger than first threshold value TH1 (YES in step S112), in step S113, the driver's arousal level is It determines with belonging to the first stage of 1 arousal level (awakening level fall degree "small"), and progresses to step S113. On the other hand, if it is determined in step S112 that the first determination is less than or equal to the first threshold TH1 (NO in step S112), the process proceeds to step S114 without performing the determination in step S113. FIG. 22 is a graph for explaining the processing in steps S112 and S118.

  In step S114, it is determined whether or not the second determination value calculated in the latest single determination period is greater than the second threshold value TH2. As shown in FIG. 23, when it is determined in step S114 that the second determination value is larger than second threshold value TH2 (YES in step S114), in step S115, the driver's arousal level is second. Is determined to belong to the first stage of the awakening level (awakening level “high”), and the process proceeds to step S116. On the other hand, if it is determined in step S114 that the second determination value is equal to or smaller than second threshold value TH2 (NO in step S114), the process proceeds to step S116 without performing the determination in step S115. FIG. 23 is a graph for explaining the processing in steps S114 and S120.

  In steps S112 and S114, as shown in FIG. 6, the eye-closed rate repeatedly increases and decreases as the degree of arousal decreases, so the driver's arousal level is determined using the determination value in a single determination period. Yes.

  In step S116 of FIG. 8, as shown in FIG. 21, it is determined whether or not the first and second determination values over a predetermined number of determination periods (the three most recent determination periods in this example) are stored in the memory. to decide. If the predetermined number of determination values is not stored (NO in step S116), the processing of steps S102 to S110 in FIG. 7 is repeated until the predetermined number of determination values is stored (step S124).

  On the other hand, when it is determined in step S116 that a predetermined number of determination values are stored in the memory (YES in step S116), in step S117, as shown in FIG. In chronologically, the judgment value of the oldest judgment period is updated with the latest judgment value. In addition, when this step S117 is entered for the first time, this update is not performed.

  Next, in step S118, the first determination value over a predetermined number of determination periods is compared with the third threshold value TH3. As shown in FIG. 22, when the first determination value is larger than the third threshold value TH3 in all determination periods (YES in step S118), in step S119, the driver's arousal level is the second value. It is determined that it belongs to the later stage of the arousal level (awakening level lowering degree “small”), and the process proceeds to step S120.

  On the other hand, if it is determined in step S118 that the first determination value in one determination period is not more than the third threshold value TH3 (NO in step S118), the determination in step S119 is not performed, and the process proceeds to step S120. move on.

  In step S120, the second determination value is compared with the fourth threshold value TH4 over a predetermined number of determination periods. As shown in FIG. 23, when the second determination value is larger than fourth threshold value TH4 in all determination periods (YES in step S120), in step S121, the driver's arousal level is the second threshold value. It is determined that it belongs to the later stage of the arousal level (awakening level lowering degree “high”), and the process proceeds to step S122.

  On the other hand, if it is determined in step S120 that the second determination value in one determination period is not more than the fourth threshold value TH4 (NO in step S120), the determination in step S121 is not performed and the process proceeds to step S122. move on.

  Note that in steps S118 and S120, the closed eyes that have increased instantaneously with a decrease in the driver's arousal level generally become continuous. Therefore, the driver's arousal level is determined using determination values in a plurality of determination periods. Judgment.

  Next, in step S122, it is determined whether any arousal level has been determined. If it is determined in step S122 that the arousal level has been determined (YES in step S122), in step S123, the lowest arousal level already determined in step S123 is determined by the driver. It is determined as the current arousal level. That is, in steps S122 and S123, from the two arousal levels of the first stage of the first arousal level or the first stage of the second arousal level until the judgment values for the predetermined number of judgment periods are stored. When the driver's current arousal level is selected and the determination values for a predetermined number of determination periods are stored, the current arousal level is selected from the four arousal levels (I) to (IV) above. .

  The arousal level estimation apparatus 1 repeats steps S103 to S123 described above, and executes a countermeasure corresponding to the arousal level, for example, via voice or the like.

  As described above, in this embodiment, as shown in Table 3 below, the eye-closing time ranges R1 and R2 are defined according to the arousal level, and the first and the second methods are calculated using the calculation method according to the arousal level. Since the second determination value is calculated, the determination accuracy can be improved at each arousal level, and the multi-level arousal level can be estimated with high accuracy.

  In the present embodiment, the blinking time during normal times (normal eye closing time) is excluded from the eye closing time ranges R1 and R2, so that the influence of individual differences can be reduced.

  Further, in the present embodiment, when setting the normal blink time, the number of appearances is increased as the eye-closing time becomes longer, so that the accuracy of the blink learning can be improved.

  In this embodiment, the higher the arousal level corresponding to the arousal level, the shorter the closed eye time range corresponding to that level, and the lower the corresponding arousal level, Since the corresponding eye-closing time range is composed of long eye-closing times, multi-stage wakefulness estimation can be performed with high accuracy.

  In the present embodiment, the upper limit value of the first eye closing time range R1 is set to be larger than the lower limit value of the second eye closing time range R2, and the first eye closing time range R1 and the second eye closing time range R2 are set. Are partially overlapped, so multistage arousal level estimation can be performed with higher accuracy.

  In the present embodiment, the cumulative value of the closed eye measurement time classified into the first closed eye time range R1 is obtained, or the closed eye measurement time classified into the second closed eye time range R2 is determined according to the time length. By substituting the weighted values and obtaining the cumulative value of the replacement values, the determination value corresponding to each arousal level is calculated, so that multistage arousal level estimation can be performed with high accuracy.

  In this embodiment, when the closed eye measurement time is shorter than the specific closed eye time (third predetermined time), the closed eye time after replacement is made longer than the closed eye time before replacement, and the closed eye measurement time is specified. When the eye closing time is longer than the eye closing time after the replacement, the eye closing time after the replacement is made shorter than the eye closing time before the replacement, so that it is possible to improve the robustness against noise of erroneous determination of the opening and closing eyes.

  Moreover, in this embodiment, since the determination period (second predetermined time) is 2 minutes or less, the arousal level can be estimated early.

  Further, in the present embodiment, based on the number of determination values corresponding to the degree of decrease in the driver's arousal level, whether the driver's arousal level belongs to the early stage or the late stage at the same arousal level. Since the determination is made, multistage arousal level estimation can be performed with high accuracy.

Second Embodiment
FIG. 24 is a block diagram showing the overall configuration of the arousal level estimation device 1 ′ in the present embodiment.

  As shown in FIG. 24, the present embodiment is different from the first embodiment in that it includes a missing extension detection circuit 140, but the other configurations are the same as those of the first embodiment. Hereinafter, only differences between the second embodiment and the first embodiment will be described, and portions having the same configuration as the first embodiment will be denoted by the same reference numerals and description thereof will be omitted.

  The lack detection circuit 140 according to the present embodiment calculates the aspect ratio of the mouth shape extracted from the driver's face image by the image processing circuit 112, and when the aspect ratio exceeds the fifth threshold value TH5, the driver Is determined to be missing. The fifth threshold value TH5 is set to an aspect ratio of the mouth that cannot be a normal conversation, based on the feature of stretch that opens the mouth greatly.

  In addition to the processing described with reference to FIGS. 7 to 9 in the first embodiment, the wakefulness level determination device 1 ′ according to the present embodiment performs the following two processes using the lack detection circuit 140. .

  Absence is a phenomenon caused by the driver's defense response to a decrease in arousal level, and tends to occur in a concentrated manner in a predetermined time zone as shown in the XXV part of FIG. In this extension, as shown in the XXVI part of FIG. 26, the eyes become narrower as the mouth is greatly opened, so there is a possibility that the eye is erroneously determined as a long closed eye as shown in the XXV part of FIG. is there. FIG. 25 is a graph showing an example of erroneous determination of open / closed eyes due to extension, and FIG. 26 is a graph showing an example of synchrony between the aspect ratio of the mouth and the opening / closing of the eyes in extension.

  Therefore, in the present embodiment, as a first process using the absence detection circuit 140, a process of excluding a certain period after detection of the absence from the awakening level determination target is performed. FIG. 27 is a flowchart showing a first process performed by the arousal level estimation apparatus 1 ′ according to this embodiment.

  In the first process, as shown in FIG. 27, first, in step S301, the aspect ratio of the driver's mouth is calculated, and then in step S302, the aspect ratio is compared with the fifth threshold value TH5.

  If the aspect ratio is larger than the fifth threshold value TH5 in step S302 (YES in step S302), as shown in FIG. 28, a fixed length is set starting from the aspect ratio exceeding the fifth threshold value TH5. This period is excluded from the awakening level determination target. FIG. 28 is a graph for explaining the setting method of the exclusion period in this embodiment.

  In general, the failure tends to be repeated many times at a fixed period, and the arousal level tends to decrease greatly after a predetermined time elapses after the failure. In view of such a characteristic of the elongation, in the present embodiment, as the second process using the elongation detection circuit 140, when the frequency of the occurrence of the elongation increases, the second threshold TH2 (in FIG. Step S114) and the fourth threshold TH4 (see Step S120 in FIG. 8) are lowered to increase the determination sensitivity.

  FIG. 29 is a flowchart showing a second process by the arousal level estimation apparatus 1 ′ in the present embodiment.

  In the second process, as shown in FIG. 29, in step S401, the measurement of the determination period is started by a timer (not shown), and it is determined whether or not the determination period has elapsed in step S402. The determination period in this example may be the same as or different from the determination period in the first embodiment.

  If it is determined in step S402 that the determination period has not elapsed (NO in step S402), in step S403, the defect detection circuit 140 performs defect detection.

  On the other hand, if it is determined in step S402 that the determination period has elapsed (YES in step S402), it is determined in step S404 whether or not a defect has been detected within the determination period. If it is determined that no deficit is detected within the determination period (NO in step S404), the counter that counts the deficit continuous detection period is reset in step S411, and the process returns to step S402.

  On the other hand, if it is determined in step S404 that a deficit has been detected (YES in step S404), the period in which the deficit is continuously detected in step S405 is counted up.

  Next, in step S406, the continuous detection period of the absence is compared with a predetermined length. If it is determined in step S406 that the continuous failure detection period does not exceed the predetermined length (NO in step S406), the process returns to step S402.

  On the other hand, when it is determined in step S406 that the continuous detection period for the absence of extension exceeds the predetermined length (YES in step S406), the current time is stored in step S407, and the second and fourth times are stored in step S408. The threshold values TH2 and TH4 are decreased.

  Then, the second and fourth thresholds TH2 and TH4 are kept lowered until a predetermined time has elapsed from the time stored in step S407, and the determination sensitivity for the later stage of each arousal level is increased. When the predetermined time has elapsed (step S409), the second and fourth threshold values TH2 and TH4 are returned to the original values (step S410), and the series of processes is terminated. Note that the predetermined time used in step S409 of the present embodiment corresponds to an example of a fourth predetermined time in the present invention.

  In the present embodiment, as in the first embodiment, the closed eye time ranges R1 and R2 are defined according to the arousal level, and the first and second determination values are calculated using a calculation method according to the arousal level. Since the calculation is performed, the determination accuracy can be improved at each arousal level, and the multi-level arousal level can be estimated with high accuracy.

  Further, in the present embodiment, as in the first embodiment, the normal blinking time (normally closed eye time) is excluded from the closed eye time ranges R1 and R2, so that the influence due to individual differences can be reduced.

  Further, in the present embodiment, as in the first embodiment, when setting the normal blink time, the longer the eye-closing time, the more the number of appearances is required, so the accuracy of the blink learning is improved. Can do.

  Further, in this embodiment, as in the first embodiment, the higher the wakefulness level corresponding to the wakefulness level, the shorter the closed eye time range corresponding to that level, and the corresponding wakefulness level of the wakefulness level. The lower the degree, the longer the eye-closing time range corresponding to the level is configured, so that multistage arousal level estimation can be performed with high accuracy.

  Further, in the present embodiment, similarly to the first embodiment, the upper limit value of the first eye closing time range R1 is set to be larger than the lower limit value of the second eye closing time range R2, and the first eye closing time range R1. And the second eye-closing time range R2 partially overlap, so that multistage arousal level estimation can be performed with higher accuracy.

  Further, in the present embodiment, as in the first embodiment, the cumulative value of the closed eye measurement time classified into the first closed eye time range R1 is obtained, or the closed eye measurement time classified into the second closed eye time range R2. Is replaced with a weighted value according to the length of time, and the cumulative value of the replacement value is calculated to calculate a determination value according to each arousal level. Estimation can be performed.

  In the present embodiment, similarly to the first embodiment, when the closed eye measurement time is shorter than the specific closed eye time (third predetermined time), the closed eye time after replacement is set to be greater than the closed eye time before replacement. If the closed eye measurement time is longer than the specific closed eye time, the closed eye time after replacement is made shorter than the closed eye time before replacement, so that the robustness against noise of erroneous determination of open / closed eyes can be improved. .

  In the present embodiment, as in the first embodiment, the determination period (second predetermined time) is set to 2 minutes or less, so that the arousal level can be estimated early.

  Further, in the present embodiment, as in the first embodiment, based on the number of determination values corresponding to the degree of decrease in the driver's arousal level, the driver's arousal level is the same as the first level or the second level in the same arousal level. Since it is determined whether it belongs to one of the stages, multi-stage wakefulness estimation can be performed with high accuracy.

  Furthermore, in this embodiment, since a certain period after the absence detection is excluded from the determination target of the arousal level, the influence of the noise in the erroneous determination of the open / closed eyes due to the extension is suppressed, and multistage arousal estimation is performed with high accuracy. be able to.

  Furthermore, in the present embodiment, when the frequency of occurrence of the absence increases, the second threshold value TH2 and the fourth threshold value TH4 are reduced over a predetermined time, so that a large reduction in the arousal level that occurs after the occurrence of the failure is high. Can be captured with accuracy.

  The closed eye time measuring circuit 120 in the present embodiment corresponds to an example of the closed eye time measuring unit in the present invention, and the normally closed eye setting unit 131 in the present embodiment corresponds to an example of the normally closed eye time setting unit in the present invention. The range defining unit 132 in the present embodiment corresponds to an example of the range defining unit in the present invention, the classification unit 133 in the present embodiment corresponds to an example of the classifying unit in the present invention, and the determination value calculation unit 134 in the present embodiment. It corresponds to an example of a determination value calculation unit in the present invention, and the determination unit 135 in the present embodiment corresponds to an example of a determination unit in the present invention. Further, the lack detection circuit 140 in the present embodiment corresponds to an example of the lack detection means in the present invention.

  The embodiment described above is described for facilitating the understanding of the present invention, and is not described for limiting the present invention. Therefore, each element disclosed in the above embodiment is intended to include all design changes and equivalents belonging to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1,1 '... Arousal degree estimation apparatus 110 ... Opening / closing eye detection apparatus 111 ... Camera 112 ... Image processing circuit 120 ... Eye closing time measuring circuit 130 ... Arousal degree level determination circuit 131 ... Normal eye closing time setting part 132 ... Range prescription | regulation part 133 ... Classification unit 134 ... Determination value calculation unit 135 ... Determination unit 140 ... Missing detection circuit 200 ... Eyeglass frame 210 ... Opening / closing eye detection device 211 ... Light emitting diode 212 ... Photo diode 213 ... Amplifier 300 ... Driver 310 ... Face

Claims (21)

A wakefulness estimation device for estimating the wakefulness of an operator,
Eye-closing time measuring means for measuring an eye-closing time when the operator is in an eye-closed state;
Range defining means for defining a plurality of closed eye time ranges according to a plurality of arousal levels indicating the awakening level of the operator in stages;
Classification means for classifying the closed eye measurement time measured by the closed eye time measurement means into the closed eye time range;
In accordance with a determination value calculation method defined according to the arousal level, a determination value calculation unit that calculates a determination value according to the arousal level from the closed eye measurement time classified by the classification unit;
A wakefulness level estimation device comprising: determination means for determining to which wakefulness level the wakefulness level of the operator belongs based on the determination value. The arousal level estimation apparatus according to claim 1,
A normal eye-closing time setting means for setting a normal eye-closing time in the normal arousal state of the operator based on the eye-closing time measured by the eye-closing time measuring means within a first predetermined time;
The range defining means excludes the normal eye closing time from the eye closing time range. The arousal level estimation apparatus according to claim 2,
The normal eye-closing time setting means sets the normal eye-closing time according to the predetermined time range when the number of appearances of the eye-closed measurement time belonging to the predetermined time range becomes equal to or greater than a predetermined number of times within the first predetermined time. And
The predetermined number of times is set so as to increase as the upper limit value of the predetermined time range increases. The arousal level estimation apparatus according to any one of claims 1 to 3,
The range defining means defines the closed eye range corresponding to the arousal level with a short closed time as the awakening level corresponding to the arousal level is higher. The arousal level estimation apparatus according to any one of claims 1 to 4,
The range defining means defines the closed eye range corresponding to the arousal level with a long closed time as the awakening level corresponding to the arousal level is lower. The arousal level estimation apparatus according to any one of claims 1 to 5,
The plurality of closed eye time ranges are:
A first eye-closing time range corresponding to a first arousal level that is lower than the normal arousal state of the operator;
At least a second eye-closing time range corresponding to a second arousal level that is lower than the first arousal level,
The classifying means classifies the closed eye measurement time measured by the closed eye time measuring means within a second predetermined time into the first or second closed eye time range. The arousal level estimation apparatus according to claim 6,
The wakefulness estimation device, wherein an upper limit value of the first closed eye time range is larger than a lower limit value of the second closed eye time range. The arousal level estimation apparatus according to claim 6 or 7,
The determination value calculating unit obtains a first determination value corresponding to the first arousal level by obtaining a cumulative value of the closed eye measurement time classified by the classification unit into the first closed eye time range. An arousal level estimation device characterized by calculating. The arousal level estimation apparatus according to any one of claims 6 to 8,
The determination value calculating unit replaces the closed eye measurement time classified into the second closed eye time range by the classification unit with a value weighted according to a time length, and obtains a cumulative value of the replacement value. Thus, the second determination value corresponding to the second arousal level is calculated. The arousal level estimation apparatus according to claim 9,
When the closed eye measurement time is shorter than a third predetermined time, the determination value calculation means makes the closed eye time after replacement longer than the closed eye measurement time before replacement, and the closed eye measurement time is the third eye measurement time. An arousal level estimation apparatus characterized in that, when the time is longer than the predetermined time, the closed eye time after replacement is shorter than the closed eye measurement time before replacement. The arousal level estimation apparatus according to any one of claims 6 to 10,
The second predetermined time is 2 minutes or less. The arousal level estimation apparatus according to any one of claims 1 to 11,
The determination means determines, based on the number of the determination values corresponding to the degree of decrease in the operator's arousal level, which stage the operator's arousal level belongs to at the same arousal level. A device for estimating arousal level. The apparatus for estimating arousal level according to claim 12,
The determination value calculation means calculates a first determination value according to a first arousal level that is lower than the normal arousal state of the operator.
The determination means includes
Based on the single first determination value, it is determined whether the wakefulness of the operator belongs to the first stage of the first wakefulness level,
A wakefulness level estimation device that determines whether or not the wakefulness level of the operator belongs to a later stage of the first wakefulness level based on a plurality of the first determination values. The wakefulness estimation device according to claim 13,
The determination means determines that the wakefulness level of the operator is the first stage of the first wakefulness level when the single first determination value exceeds a first threshold value. Awakening level estimation device. The wakefulness estimation device according to claim 13 or 14,
The determination means determines that the wakefulness level of the operator is a later stage of the first wakefulness level when all of the plurality of first determination values exceed the second threshold value. A device for estimating arousal level. The wakefulness estimation device according to claim 15,
Further comprising an extension detection means for detecting the extension of the operator;
The awakening level estimation device, wherein the classifying unit lowers the second threshold for a fourth predetermined time from when the absence detection is detected by the extension detection unit. The arousal level estimation apparatus according to any one of claims 13 to 16,
The determination value calculating means calculates a second determination value corresponding to a second arousal level, which is lower than the first arousal level,
The determination means includes
Based on the single second determination value, it is determined whether or not the wakefulness of the operator belongs to the first stage of the second wakefulness level,
An arousal level estimation apparatus that determines whether or not the wakefulness level of the operator belongs to a later stage of the second arousal level based on a plurality of the second determination values. The wakefulness estimation apparatus according to claim 17,
The determination unit determines that the wakefulness level of the operator is in the first stage of the second wakefulness level when the single second determination value exceeds a third threshold value. Awakening level estimation device. The wakefulness estimation device according to claim 18,
The determination means determines that the wakefulness level of the operator is in a later stage of the second wakefulness level when any of a plurality of second determination values exceeds a fourth threshold value. Awakening level estimation device. The wakefulness estimation device according to claim 19,
Further comprising an extension detection means for detecting the extension of the operator;
The determination means reduces the fourth threshold for a fourth predetermined time after detection of the absence by the absence detection means. The arousal level estimation apparatus according to any one of claims 1 to 20,
Further comprising an extension detection means for detecting the extension of the operator;
The awakening level estimation device, wherein the classification means excludes the closed eye measurement time measured by the closed eye time measurement means while the absence detection means detects the absence from the classification target.

Images