WO2023105970A1

WO2023105970A1 - Estimation device, estimation method, and program

Info

Publication number: WO2023105970A1
Application number: PCT/JP2022/039910
Authority: WO
Inventors: 知里今村; 豊居; 孝好古山
Original assignee: パナソニックＩｐマネジメント株式会社
Priority date: 2021-12-10
Filing date: 2022-10-26
Publication date: 2023-06-15

Abstract

This estimation device (2) comprises: an eye-closing state detection unit (12) which detects an eye-closing time of the eyes of a driver; a drowsiness determination unit (14) which determines the level of drowsiness of the driver; and a micro-sleep estimation unit (16) which estimates the driver to be in a micro-sleep state on the condition that the eye-closing time is detected as equal to or greater than a first time and smaller than a second time by the eye-closing state detection unit (12), and the level of drowsiness is determined as equal to or greater than a first threshold by the drowsiness determination unit (14).

Description

Estimation device, estimation method and program

The present disclosure relates to an estimation device, an estimation method, and a program.

It is known that when a driver becomes very drowsy while driving a vehicle, the driver may fall into a momentary state of sleep called "microsleep" in the stage before falling asleep completely. there is From the viewpoint of danger avoidance, accident prevention, etc., a detection device for detecting such a micro-sleep state of a driver has been proposed (see, for example, Patent Document 1). This type of detection device detects the driver's micro-sleep state based on the driver's closed eye state.

Japanese Patent Application Publication No. 2018-508870

However, with conventional detection devices, there is a problem that, for example, when the driver intentionally closes his or her eyes because the contact lens is displaced, the micro-sleep state of the driver may be erroneously detected.

Therefore, the present disclosure provides an estimation device, an estimation method, and a program capable of accurately estimating the micro-sleep state.

An estimating device according to an aspect of the present disclosure is an estimating device for estimating that a driver of a vehicle is in a micro-sleep state, comprising: an eye-closed state detection unit that detects an eye-closed time of the driver; a drowsiness determination unit that determines a drowsiness level of the driver; and the eye-closed state detection unit that detects that the eye-closed time is longer than or equal to a first time and less than a second time, and the drowsiness determination unit detects the drowsiness. a micro-sleep estimator for estimating that the driver is in a micro-sleep state on condition that the level is determined to be equal to or greater than a first threshold.

In addition, these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM (Compact Disc-Read Only Memory), Any combination of systems, methods, integrated circuits, computer programs and storage media may be implemented.

According to the estimation device and the like according to one aspect of the present disclosure, it is possible to accurately estimate the micro-sleep state.

1 is a block diagram showing the configuration of an estimation device according to Embodiment 1; FIG. 4 is a flowchart showing the flow of operations of the estimation device according to Embodiment 1; FIG. 12 is a block diagram showing the configuration of an estimation device according to Embodiment 2; FIG. 9 is a flow chart showing the flow of operations of an estimation device according to Embodiment 2; FIG. 12 is a block diagram showing the configuration of an estimation device according to Embodiment 3; FIG. 11 is a flow chart showing the flow of operations of an estimation device according to Embodiment 3. FIG. FIG. 12 is a block diagram showing the configuration of an estimation device according to Embodiment 4; FIG. 14 is a flow chart showing the operation flow of an estimation device according to Embodiment 4; FIG. 12 is a block diagram showing the configuration of an estimation device according to Embodiment 5; 14 is a flow chart showing the flow of operations of an estimation device according to Embodiment 5. FIG. FIG. 13 is a block diagram showing the configuration of an estimation device according to Embodiment 6; 14 is a flow chart showing the flow of operations of an estimation device according to Embodiment 6. FIG. FIG. 13 is a block diagram showing the configuration of an estimation device according to Embodiment 7; FIG. 13 is a flow chart showing the flow of operations of an estimation device according to Embodiment 7. FIG. FIG. 22 is a block diagram showing the configuration of an estimation device according to Embodiment 8; 20 is a flow chart showing the operation flow of an estimation device according to Embodiment 8. FIG. FIG. 23 is a block diagram showing the configuration of an estimation device according to Embodiment 9; 29 is a flow chart showing the operation flow of an estimation device according to Embodiment 9. FIG. FIG. 23 is a block diagram showing the configuration of an estimation device according to Embodiment 10; 29 is a flow chart showing the operation flow of an estimation device according to Embodiment 10. FIG. FIG. 23 is a block diagram showing the configuration of an estimation device according to Embodiment 11; FIG. 22 is a flow chart showing the operation flow of an estimation device according to Embodiment 11; FIG. FIG. 23 is a block diagram showing the configuration of an estimation device according to Embodiment 12; 29 is a flow chart showing the flow of operations of an estimation device according to Embodiment 12. FIG. FIG. 23 is a block diagram showing the configuration of an estimation device according to Embodiment 13; FIG. 22 is a flow chart showing the flow of the first operation of the estimating device according to the thirteenth embodiment; FIG. FIG. 22 is a flow chart showing the flow of the second operation of the estimation device according to the thirteenth embodiment; FIG.

An estimating device according to a first aspect of the present disclosure is an estimating device for estimating that a driver of a vehicle is in a micro-sleep state, and includes an eye-closed state detection unit that detects an eye-closed time of the driver. a drowsiness determination unit that determines a drowsiness level of the driver; and the eye-closed state detection unit that detects that the eye-closed time is greater than or equal to a first time period and less than a second time period, and the drowsiness determination unit detects a microsleep estimating unit estimating that the driver is in a microsleep state on condition that the drowsiness level is determined to be equal to or greater than a first threshold.

According to this aspect, the micro-sleep estimation unit estimates the micro-sleep state of the driver in consideration of the detection result of the closed-eye time by the closed-eye state detection unit and the determination result of the drowsiness level by the drowsiness determination unit. As a result, the microsleep state can be estimated with high accuracy.

Further, in the estimation device according to the second aspect of the present disclosure, in the first aspect, the first time is 0.5 seconds, and the second time is 3 seconds. good.

According to this aspect, when the driver falls into a momentary sleep state for 0.5 seconds or more and less than 3 seconds, and the drowsiness level is equal to or higher than the first threshold, it is determined that the driver is in the micro-sleep state. can be estimated.

Further, in the estimation device according to the third aspect of the present disclosure, in the first aspect or the second aspect, the micro-sleep estimating unit further confirms the estimation result that the driver is in the micro-sleep state. It may be configured to calculate a first reliability, which is an index indicating likelihood.

According to this aspect, it is possible to improve the accuracy of estimating the micro-sleep state.

Further, in the estimation device according to the fourth aspect of the present disclosure, in the third aspect, the eye-closed state detection unit further determines that the eye-closed time is greater than or equal to the first time and less than the second time. A second reliability, which is an index indicating the certainty of the detection result, is calculated, and the drowsiness determination unit further calculates the second reliability, which is an index indicating the certainty of the determination result that the drowsiness level is equal to or greater than the first threshold. A third confidence may be calculated, and the microsleep estimator may be configured to calculate the first confidence based on the second confidence and the third confidence. .

According to this aspect, it is possible to further improve the accuracy of estimating the micro-sleep state.

Further, in the estimating device according to the fifth aspect of the present disclosure, in the third aspect, the estimating device further includes an open/closed state detection unit that detects an eye opening/closing speed of the driver, and the microsleep estimation the eye-closed state detection unit detects that the eye-closed time is greater than or equal to the first time period and less than the second time period, and the drowsiness determination unit detects that the drowsiness level is greater than or equal to the first threshold value; and the opening/closing state detection unit detects that the opening/closing speed is less than a second threshold, it is estimated that the driver is in the micro-sleep state. may

According to this aspect, the micro-sleep estimating unit includes the detection result of the closed-eye time by the closed-eye state detection unit, the sleepiness level determination result by the drowsiness determination unit, and the opening/closing speed detection result by the open/close state detection unit. Consider and estimate the driver's micro-sleep state. This makes it possible to estimate the microsleep state more accurately.

Further, in the estimation device according to the sixth aspect of the present disclosure, in the fifth aspect, the eye-closed state detection unit further determines that the eye-closed time is greater than or equal to the first time and less than the second time. A second reliability, which is an index indicating the certainty of the detection result, is calculated, and the drowsiness determination unit further calculates the second reliability, which is an index indicating the certainty of the determination result that the drowsiness level is equal to or greater than the first threshold. After calculating a certain third degree of reliability, the opening/closing state detection unit further calculates a fourth degree of reliability, which is an index indicating the likelihood of the detection result that the opening/closing speed is less than the second threshold. and the microsleep estimator may be configured to calculate the first reliability based on the second reliability, the third reliability, and the fourth reliability.

Further, in the estimation device according to the seventh aspect of the present disclosure, in the fifth aspect or the sixth aspect, the drowsiness determination unit may be configured to include a function as the open/closed state detection unit.

According to this aspect, the configuration of the estimation device can be simplified.

Further, in the estimation device according to the eighth aspect of the present disclosure, in the seventh aspect, the drowsiness determination unit may further include a function as the closed-eyes state detection unit.

According to this aspect, the configuration of the estimation device can be further simplified.

Further, in the estimating device according to the ninth aspect of the present disclosure, in the third aspect, the estimating device further includes an open/closed state detection unit that detects an eye-closing speed and an eye-opening speed of the driver, The microsleep estimating unit detects that the closed-eyes time is longer than or equal to the first time period and less than the second time period by the closed-eye state detecting unit, and the drowsiness determining unit determines that the drowsiness level is equal to or greater than the first time. and the open/closed state detection unit detects that the eye-closing speed is less than the third threshold, and the open-close state detection unit detects that the eye-opening speed is less than the fourth threshold. It may be configured to presume that the driver is in a micro-sleep state on the condition that something is detected.

According to this aspect, the microsleep estimating unit detects the closing speed and the opening speed of the eyes by the open/closed state detection unit, in addition to the detection result of the closed eye time by the closed eye state detection unit and the determination result of the drowsiness level by the drowsiness determination unit. The results are also taken into account to estimate the driver's microsleep state. This makes it possible to estimate the microsleep state more accurately.

Further, in the estimating device according to the tenth aspect of the present disclosure, in the ninth aspect, the eye-closed state detection unit further determines that the eye-closed time is greater than or equal to the first time and less than the second time. A second reliability, which is an index indicating the certainty of the detection result, is calculated, and the drowsiness determination unit further calculates the second reliability, which is an index indicating the certainty of the determination result that the drowsiness level is equal to or greater than the first threshold. After calculating a certain third reliability, the open/closed state detection unit further calculates a fifth reliability, which is an index indicating the likelihood of the detection result that the eye-closing speed is less than the third threshold, and , calculating a sixth reliability that is an index indicating the likelihood of the detection result that the eye-opening speed is less than the fourth threshold, and the microsleep estimator calculates the second reliability, the The first reliability may be calculated based on the third reliability, the fifth reliability, and the sixth reliability.

Further, in the estimating device according to the eleventh aspect of the present disclosure, in the third aspect, the microsleep estimating unit determines the first reliability according to the drowsiness level determined by the drowsiness determining unit. You may comprise so that it may calculate.

According to this aspect, it is possible to improve the calculation accuracy of the first reliability.

Further, in the estimating device according to the twelfth aspect of the present disclosure, in any one of the first to third aspects, the estimating device further includes binocular detection for detecting both eyes of the driver. and the microsleep estimating unit detects that the eye-closed time is longer than or equal to the first time period and is less than the second time period by the eye-closed state detecting unit, and the drowsiness determining unit detects that the drowsiness level is is determined to be equal to or greater than the first threshold, and the binocular detection unit detects both eyes of the driver, so as to estimate that the driver is in a micro-sleep state. may be configured.

According to this aspect, the microsleep estimation unit considers the detection result of the binocular detection unit in addition to the detection result of the closed-eye time by the closed-eye state detection unit and the determination result of the drowsiness level by the drowsiness determination unit. , to estimate the driver's microsleep state. This makes it possible to estimate the microsleep state more accurately.

Further, in the estimating device according to the thirteenth aspect of the present disclosure, in any one of the first to third aspects, the estimating device further includes a blink count detection that detects the number of blinks of the driver. and the microsleep estimating unit detects that the eye-closed time is longer than or equal to the first time period and is less than the second time period by the eye-closed state detecting unit, and the drowsiness determining unit detects that the drowsiness level is is determined to be equal to or greater than the first threshold, and the number of times of blinking of the driver per unit time is detected by the blinking number detection unit to be increased or decreased by a fifth threshold or more. , the driver is assumed to be in a micro-sleep state.

According to this aspect, in addition to the eye-closed time detection result by the eye-closed state detection unit and the drowsiness level determination result by the drowsiness determination unit, the detection result of the number of blinks detection unit is also taken into consideration, and the microsleep of the driver is performed. Estimate the state. This makes it possible to estimate the microsleep state more accurately.

Further, in the estimating device according to the fourteenth aspect of the present disclosure, in any one of the first to third aspects, the estimating device further acquires life log information regarding the life of the driver. a life log information acquiring unit, wherein the microsleep estimating unit detects that the closed-eyes time is longer than or equal to the first time and less than the second time by the closed-eye state detecting unit; on the condition that the drowsiness level is determined to be less than the first threshold by and the life log information that affects the driver's microsleep state is acquired by the life log information acquisition unit. It may be configured to presume that the driver is in a micro-sleep state.

According to this aspect, in addition to the eye-closed time detection result by the eye-closed state detection unit and the drowsiness level determination result by the drowsiness determination unit, the life log information acquired by the life log information acquisition unit is also considered, Estimate the driver's microsleep state. This makes it possible to estimate the microsleep state more accurately.

Further, in the estimating device according to the fifteenth aspect of the present disclosure, in any one of the first to third aspects, the estimating device further includes facial feature information indicating facial features of the driver. The microsleep estimating unit detects that the closed-eyes time is longer than or equal to the first time and shorter than the second time by the closed-eye state detecting unit, and the drowsiness On the condition that the determination unit determines that the drowsiness level is less than the first threshold value, and that the facial feature information acquisition unit acquires the facial feature information that affects the micro-sleep state of the driver. , the driver is assumed to be in a micro-sleep state.

According to this aspect, in addition to the detection result of the closed-eye time by the closed-eye state detection unit and the determination result of the drowsiness level by the drowsiness determination unit, the facial feature information acquired by the facial feature information acquisition unit is also considered, Estimate the driver's microsleep state. This makes it possible to estimate the microsleep state more accurately.

Further, in the estimating device according to the sixteenth aspect of the present disclosure, in any one of the first to third aspects, the estimating device further includes: The micro-sleep estimating unit detects that the closed-eyes time is longer than or equal to the first time and less than the second time by the eye-closed state detecting unit, and the drowsiness determining unit detects on the condition that the drowsiness level is determined to be less than the first threshold, and that the head motion detection unit detects the head motion that affects the driver's micro-sleep state; It may be configured to presume that the driver is in a micro-sleep state.

According to this aspect, the detection result of the head movement detection unit is taken into consideration in addition to the detection result of the closed eye time by the closed eye state detection unit and the determination result of the drowsiness level by the drowsiness determination unit. Estimate sleep state. This makes it possible to estimate the microsleep state more accurately.

Further, in the estimating device according to the seventeenth aspect of the present disclosure, in the third aspect, the estimating device further detects a situation that affects the estimation of the driver's micro-sleep state by the micro-sleep estimating unit. and the microsleep estimating unit may change the first reliability in consideration of the detection result of the erroneous estimation condition detecting unit.

Further, in the estimation device according to the eighteenth aspect of the present disclosure, in any one aspect of the first to third aspects, the estimation device further includes: An erroneous estimation situation detection unit that detects an erroneous estimation situation that is a situation that affects state estimation, and the microsleep estimating unit detects, when the erroneous estimation situation is detected by the erroneous estimation situation detection unit, It may be configured not to perform the estimation of the micro-sleep state of the driver.

Further, in the estimating device according to the nineteenth aspect of the present disclosure, in any one of the first to third aspects, the estimating device further detects a driving situation of the vehicle by the driver. a driving situation detection unit, and based on the driving situation detected by the driving situation detection unit, the first time and/or the second time used for detecting the eye-closed time in the eye-closed state detection unit; and an eye-closing time changing unit for changing.

Further, in the estimation device according to the twentieth aspect of the present disclosure, in any one of the first aspect to the eleventh aspect, the eye-closed state detection unit includes image information of the driver captured by the imaging unit. and the drowsiness determination unit may determine the drowsiness level based on the driver's biological information detected by a biological sensor.

According to this aspect, it is possible to easily detect the closed eye time by using the image information of the driver captured by the imaging unit. In addition, by using the driver's biological information detected by the biological sensor, the drowsiness level can be easily determined.

Further, in the estimation device according to the twenty-first aspect of the present disclosure, in any one of the first aspect to the eleventh aspect, the closed-eyes state detection unit includes image information of the driver captured by the imaging unit. and the drowsiness determination unit may determine the drowsiness level based on the image information.

According to this aspect, by using the image information of the driver imaged by the imaging unit, it is possible to easily detect the closed eye time and easily determine the drowsiness level.

An estimation method according to a twenty-second aspect of the present disclosure is an estimation method for estimating that a driver of a vehicle is in a micro-sleep state, comprising: (a) detecting an eye closure time of the driver's eyes; (b) determining the drowsiness level of the driver; (c) detecting that the eye-closing time is longer than or equal to a first time period and less than a second time period in (a); and assuming that the driver is in a micro-sleep state, provided that the drowsiness level is determined to be greater than or equal to a first threshold in b).

According to this aspect, the driver's micro-sleep state is estimated in consideration of the eye closure time detection result and the drowsiness level determination result. As a result, the microsleep state can be estimated with high accuracy.

A program according to the twenty-third aspect of the present disclosure is a program that causes a computer to execute the estimation method according to the twenty-second aspect described above.

In addition, these comprehensive or specific aspects may be realized by a system, method, integrated circuit, computer program, or a recording medium such as a computer-readable CD-ROM. Any combination of programs or recording media may be used.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that the embodiments described below are all comprehensive or specific examples. Numerical values, shapes, materials, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are examples, and are not intended to limit the present disclosure. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in independent claims representing the highest concept will be described as optional constituent elements.

(Embodiment 1)
[1-1. Configuration of estimation device]
First, the configuration of the estimation device 2 according to Embodiment 1 will be described with reference to FIG. FIG. 1 is a block diagram showing the configuration of an estimation device 2 according to Embodiment 1. As shown in FIG.

As shown in FIG. 1, the estimation device 2 according to Embodiment 1 is a device for detecting the micro-sleep state of the driver of the vehicle. An estimation device 2 and a sensor group 3 are mounted on the vehicle. The vehicle is, for example, a motor vehicle such as a passenger car, bus or truck. In addition, the vehicle is not limited to an automobile, and may be, for example, a construction machine or an agricultural machine.

The sensor group 3 includes, for example, various sensors for detecting information about the vehicle and/or the driver sitting in the driver's seat of the vehicle. Specifically, the sensor group 3 includes, for example, an imaging unit 4, a biosensor 6, a vehicle state sensor 5, and the like.

The imaging unit 4 is a camera for imaging the driver sitting in the driver's seat of the vehicle. As the imaging unit 4, for example, a camera using a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a camera using a CCD (Charge Coupled Device) image sensor can be applied. The imaging unit 4 outputs image information obtained by imaging the driver to the estimation device 2 .

The biosensor 6 is a sensor for detecting the biometric information of the driver sitting in the driver's seat of the vehicle (eg, blood pressure, body temperature, respiratory rate, heart rate, amount of muscle activity, etc.). The biological sensor 6 outputs the detected biological information to the estimating device 2 .

The vehicle state sensor 5 is a sensor for detecting the speed and acceleration of the vehicle. The vehicle state sensor 5 outputs vehicle state information indicating the detected speed, acceleration, etc. to the estimation device 2 .

The estimation device 2 includes a sensor information acquisition unit 7 , a closed-eye state detection unit 12 , a drowsiness determination unit 14 , and a microsleep estimation unit 16 . Note that the estimation device 2 may include one or more sensors included in the sensor group 3 described above as a component.

The sensor information acquisition unit 7 acquires various types of information output from various sensors included in the sensor group 3 and outputs the acquired various types of information to the closed-eyes state detection unit 12 and the drowsiness determination unit 14 . In the present embodiment, a case will be described in which the sensor information acquisition unit 7 acquires image information output from, for example, the imaging unit 4 and outputs the acquired image information to the closed-eye state detection unit 12 . Further, in the present embodiment, a case will be described in which the sensor information acquisition unit 7 acquires, for example, biometric information output from the biosensor 6 and outputs the acquired biometric information to the drowsiness determination unit 14 .

The eye-closed state detection unit 12 detects the closed-eye time of the driver based on various information from the sensor information acquisition unit 7, for example, image information from the sensor information acquisition unit 7. Specifically, the eye-closed state detection unit 12 detects the closed-eye time of the driver by analyzing the image of the driver's eyes included in the image information. Here, the closed eye time means the time during which the driver's eyes are closed, more specifically, the time from when the driver's eyelids begin to close until the eyelids close and reopen. The eye-closed state detection unit 12 outputs the detection result of the eye-closed time to the microsleep estimation unit 16 . In the present embodiment, the eye-closed state detection unit 12 detects the closed-eye time of the driver based on the image information from the sensor information acquisition unit 7, but is not limited to this. may be used to detect the closing time of the driver's eyes.

The drowsiness determination unit 14 determines a drowsiness level indicating the degree of drowsiness of the driver based on various information from the sensor information acquisition unit 7, for example, biological information from the sensor information acquisition unit 7. The sleepiness level is represented, for example, by numerical values in five stages from "1" to "5". It is assumed that the higher the numerical value of the drowsiness level, the higher the degree of drowsiness of the driver. Specifically, sleepiness level "1" seems to be not sleepy at all, sleepiness level "2" seems to be slightly sleepy, sleepiness level "3" seems to be sleepy, sleepiness level "4" seems to be quite sleepy, and sleepiness level "5". is classified as very sleepy. The sleepiness determination unit 14 outputs the sleepiness level determination result to the microsleep estimation unit 16 .

In the present embodiment, the drowsiness determination unit 14 determines the drowsiness level of the driver based on the biological information from the sensor information acquisition unit 7, but is not limited to this. The drowsiness level of the driver may be determined based on the image information. In this case, the drowsiness determination unit 14 analyzes the image of the driver's eyes included in the image information, and determines the drowsiness level of the driver based on, for example, the degree of eyelid opening, which is an index indicating the degree of opening of the eyelids. You may Alternatively, the drowsiness determination unit 14 may determine the drowsiness level of the driver using, for example, deep learning.

The microsleep estimator 16 has an estimator 18 . The estimating unit 18 detects that the eye-closed time is 0.5 seconds (an example of a first time) or more and less than 3 seconds (an example of a second time) by the eye-closed state detection unit 12, and the drowsiness determination unit 14 determines that the drowsiness level is equal to or higher than "4" (an example of the first threshold), it is estimated that the driver is in the micro-sleep state.

In the present embodiment, the condition for the eye closing time is set to 0.5 seconds or more and less than 3 seconds, but is not limited to this. Any time can be set. Also, in the present embodiment, the sleepiness level condition is set to "4" or higher, but is not limited to this, and may be set to, for example, "5" or higher, and the first threshold can be set arbitrarily.

The estimation result of the estimation unit 18 is output to, for example, the CAN (Controller Area Network) of the vehicle. As a result, for example, when it is estimated that the driver is in a micro-sleep state, control is performed such that an alarm is sounded to awaken the driver, or the vehicle is degraded to stop the vehicle safely. is done. Note that the retraction operation means, for example, an operation of controlling the steering to bring the vehicle to the edge of the roadway (road shoulder), or controlling the engine or brakes to decelerate the vehicle.

[1-2. Operation of estimation device]
Next, operation of the estimation device 2 according to Embodiment 1 will be described with reference to FIG. FIG. 2 is a flow chart showing the operation flow of the estimation device 2 according to the first embodiment.

As shown in FIG. 2, the sensor information acquisition unit 7 acquires image information output from the imaging unit 4 (S11), and outputs the acquired image information to the closed-eye state detection unit 12. Further, the sensor information acquisition unit 7 acquires the biological information output from the biological sensor 6 (S11), and outputs the acquired biological information to the drowsiness determination unit .

The eye-closed state detection unit 12 detects the eye-closed time based on the image information from the sensor information acquisition unit 7 and outputs the detection result of the eye-closed time to the microsleep estimation unit 16 . Also, the drowsiness determination unit 14 determines the drowsiness level based on the biological information from the sensor information acquisition unit 7 and outputs the result of the drowsiness level determination to the microsleep estimation unit 16 .

When the eye-closed state detection unit 12 detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds (S12), and the sleepiness determination unit 14 determines that the sleepiness level is "4" or more (S13 ), the estimation unit 18 of the microsleep estimation unit 16 estimates that the driver is in the microsleep state (S14).

[1-3. effect]
In the present embodiment, the estimating unit 18 of the micro-sleep estimating unit 16 estimates that the driver is in the micro-sleep state, taking into consideration the driver's eye closure time and the driver's drowsiness level. As a result, the microsleep state can be estimated with high accuracy.

(Embodiment 2)
[2-1. Configuration of estimation device]
The configuration of an estimation device 2A according to Embodiment 2 will be described with reference to FIG. FIG. 3 is a block diagram showing the configuration of an estimation device 2A according to Embodiment 2. As shown in FIG. In addition, in the present embodiment, the same reference numerals are given to the same constituent elements as in the first embodiment, and the description thereof will be omitted.

As shown in FIG. 3, the estimation apparatus 2A according to Embodiment 2 includes an image information acquisition section 8 and a biometric information acquisition section 10 instead of the sensor information acquisition section 7 described in Embodiment 1 above. . In each embodiment described below, an example of estimating a micro-sleep state using the imaging unit 4 and the biosensor 6 of the sensor group 3 shown in FIG. 1 will be described.

The image information acquisition unit 8 acquires image information output from the imaging unit 4 . The image information acquisition unit 8 outputs the acquired image information to the closed-eye state detection unit 12 .

The biometric information acquisition unit 10 acquires biometric information output from the biosensor 6 . The biometric information acquisition unit 10 outputs the acquired biometric information to the drowsiness determination unit 14 .

Based on the image information from the image information acquisition unit 8, the eye-closed state detection unit 12 detects the eye closure time of the driver. In the present embodiment, the eye-closed state detection unit 12 detects the closed-eye time of the driver based on the image information from the image information acquisition unit 8, but is not limited to this. may be used to detect the closing time of the driver's eyes. Alternatively, the eye-closed state detection unit 12 may detect the closed-eye time of the driver based on the biometric information from the biometric information acquisition unit 10 . In this case, the biological information output from the biological sensor 6 such as a myoelectric sensor can be used as the biological information from the biological information acquisition unit 10 .

The drowsiness determination unit 14 determines a drowsiness level indicating the degree of drowsiness of the driver based on the biometric information from the biometric information acquisition unit 10 .

In the present embodiment, the drowsiness determination unit 14 determines the drowsiness level of the driver based on the biometric information from the biometric information acquisition unit 10, but is not limited to this. The drowsiness level of the driver may be determined based on the image information. In this case, the drowsiness determination unit 14 analyzes the image of the driver's eyes included in the image information, and determines the drowsiness level of the driver based on, for example, the degree of eyelid opening, which is an index indicating the degree of opening of the eyelids. You may Alternatively, the drowsiness determination unit 14 may determine the drowsiness level of the driver using, for example, deep learning.

Also, in the estimation device 2A according to the second embodiment, the configuration of the microsleep estimation unit 16A is different from that in the first embodiment. Specifically, the microsleep estimator 16A has a reliability calculator 20 in addition to the estimator 18 described in the first embodiment.

The reliability calculation unit 20 calculates a reliability (an example of a first reliability) that is an index indicating the probability of the estimation result of the estimation unit 18 that the driver is in the micro-sleep state. The reliability is calculated in two stages, for example, "low" and "high". The reliability calculation unit 20 may calculate the reliability according to the drowsiness level determined by the drowsiness determination unit 14. For example, the reliability may be calculated so that the higher the drowsiness level, the higher the reliability. good. Although the microsleep estimation unit 16A has the reliability calculation unit 20 in the present embodiment, the reliability calculation unit 20 may be omitted without being limited to this.

The estimation result of the estimation unit 18 and the calculation result of the reliability calculation unit 20 are output to, for example, the CAN of the vehicle. As a result, for example, when it is estimated that the driver is in the micro-sleep state and the reliability is "low", control is performed to sound an alarm to awaken the driver. Further, for example, when it is estimated that the driver is in the micro-sleep state and the reliability is "high", the vehicle is controlled to degenerate in order to safely stop the vehicle.

[2-2. Operation of estimation device]
Next, operation of the estimation device 2A according to Embodiment 2 will be described with reference to FIG. FIG. 4 is a flow chart showing the operation flow of the estimation device 2A according to the second embodiment.

As shown in FIG. 4, the image information acquisition unit 8 acquires the image information output from the imaging unit 4 (S101), and outputs the acquired image information to the closed-eye state detection unit 12. Also, the biometric information acquisition unit 10 acquires biometric information output from the biosensor 6 ( S<b>101 ), and outputs the acquired biometric information to the drowsiness determination unit 14 .

The eye-closed state detection unit 12 detects the eye-closed time based on the image information from the image information acquisition unit 8, and outputs the detection result of the eye-closed time to the microsleep estimation unit 16A. In addition, the drowsiness determination unit 14 determines the drowsiness level based on the biometric information from the biometric information acquisition unit 10, and outputs the determination result of the drowsiness level to the microsleep estimation unit 16A.

When the eye-closed state detecting unit 12 detects that the closed-eye time is less than 0.5 seconds or longer than 3 seconds (NO in S102), the estimating unit 18 of the microsleep estimating unit 16A determines that the driver is in microsleep. It is estimated that it is not in the state (S103). In this case, the reliability calculator 20 of the microsleep estimator 16A does not calculate the reliability.

Returning to step S102, when the closed-eye state detection unit 12 detects that the closed-eye time is 0.5 seconds or more and less than 3 seconds (YES in S102), the drowsiness determination unit 14 sets the drowsiness level to "4". If it is determined to be above (YES in S104), the reliability calculation unit 20 of the microsleep estimating unit 16A calculates the reliability “high” (S105), and the estimating unit 18 of the microsleep estimating unit 16A estimates that the driver is in a micro-sleep state (S106).

Returning to step S104, when the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S104), the reliability calculation unit 20 of the microsleep estimation unit 16A sets the reliability to "low". (S107), and the estimation unit 18 of the microsleep estimation unit 16A estimates that the driver is in the microsleep state (S106).

Note that the microsleep estimation unit 16A may not have the reliability calculation unit 20, and in this case, steps S105 and S107 may be omitted. In this case, when the drowsiness determination unit 14 determines that the drowsiness level is "4" or higher (YES in S104), the process proceeds to step S106, and the estimation unit 18 estimates that the driver is in the micro-sleep state. You may When the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S104), the process proceeds to step S103, where the estimation unit 18 estimates that the driver is not in the micro-sleep state. may

[2-3. effect]
In the present embodiment, the estimating unit 18 of the microsleep estimating unit 16A estimates that the driver is in the microsleep state, taking into consideration the driver's eye closure time and the driver's drowsiness level. As a result, even if the driver intentionally closes his or her eyes for a moment, for example, due to displacement of the contact lens, if the driver's drowsiness level is low, it can be determined that the driver is in a micro-sleep state. The reliability of the estimation result can be calculated to be low. As a result, it is possible to accurately estimate the microsleep state.

(Embodiment 3)
[3-1. Configuration of estimation device]
The configuration of the estimation device 2B according to Embodiment 3 will be described with reference to FIG. FIG. 5 is a block diagram showing the configuration of an estimation device 2B according to Embodiment 3. As shown in FIG. In addition, in the present embodiment, the same reference numerals are given to the same constituent elements as in the above-described second embodiment, and the description thereof will be omitted.

As shown in FIG. 5, the estimation device 2B according to the third embodiment differs from the first embodiment in the configurations of the eye-closed state detection unit 12B, drowsiness determination unit 14B, and microsleep estimation unit 16B.

The eye-closed state detector 12B has an eye-closed time detector 22 and a reliability calculator 24 . The eye-closed time detection unit 22 detects the eye-closed time of the driver based on the image information from the image information acquisition unit 8 . Note that the eye-closed time detection unit 12B may detect the eye-closed time of the driver using, for example, deep learning. The eye-closed time detection unit 22 outputs the detection result of the eye-closed time to the microsleep estimation unit 16B. The reliability calculation unit 24 calculates the reliability (an example of the second reliability) that is an index indicating the probability of the detection result of the eye closure time detection unit 22 that the eye closure time is 0.5 seconds or more and less than 3 seconds. calculate. Further, the reliability calculation unit 24 calculates the reliability, which is an index indicating the probability of the detection result of the eye closure time detection unit 22 that the eye closure time is less than 0.5 seconds or 3 seconds or more. The reliability is calculated as a numerical value between 0% and 100%, for example. The reliability calculation unit 24 outputs the reliability calculation result to the microsleep estimation unit 16B.

The drowsiness determination unit 14B has a drowsiness level determination unit 26 and a reliability calculation unit 28. The drowsiness level determination unit 26 determines a drowsiness level indicating the degree of drowsiness of the driver based on the biological information from the biological information acquisition unit 10 . Note that the drowsiness level determination unit 26 may determine the drowsiness level of the driver using, for example, deep learning. The sleepiness level determination unit 26 outputs the sleepiness level determination result to the microsleep estimation unit 16B. The reliability calculation unit 28 calculates a reliability (an example of a third reliability) that is an index indicating the likelihood of the determination result of the drowsiness level determination unit 26 that the drowsiness level is "4" or higher. Further, the reliability calculation unit 28 calculates the reliability, which is an index indicating the likelihood of the determination result of the sleepiness level determination unit 26 that the sleepiness level is less than "4". The reliability is calculated as a numerical value between 0% and 100%, for example. The reliability calculation unit 28 outputs the reliability calculation result to the microsleep estimation unit 16B.

The reliability calculation unit 20B of the microsleep estimation unit 16B determines whether the driver is A reliability, which is an index indicating the probability of the estimation result of the estimation unit 18 that the state is the micro-sleep state, is calculated. Further, the reliability calculation unit 20B of the micro-sleep estimation unit 16B determines the estimation result of the estimation unit 18 that the driver is not in the micro-sleep state, based on the calculation result of the reliability calculation unit 24 of the closed-eye state detection unit 12B. Reliability, which is an index indicating likelihood, is calculated. The reliability is calculated as a numerical value between 0% and 100%, for example.

[3-2. Operation of estimation device]
Next, operation of the estimation device 2B according to Embodiment 3 will be described with reference to FIG. FIG. 6 is a flow chart showing the operation flow of the estimation device 2B according to the third embodiment.

As shown in FIG. 6, the image information acquisition unit 8 acquires the image information output from the imaging unit 4 (S201), and outputs the acquired image information to the closed-eye state detection unit 12B. Also, the biological information acquisition unit 10 acquires the biological information output from the biological sensor 6 (S201), and outputs the acquired biological information to the drowsiness determination unit 14B.

The eye-closed time detection unit 22 of the eye-closed state detection unit 12B detects the eye-closed time based on the image information from the image information acquisition unit 8, and outputs the detection result of the eye-closed time to the microsleep estimation unit 16B. The sleepiness level determination unit 26 of the sleepiness determination unit 14B determines the sleepiness level based on the biological information from the biological information acquisition unit 10, and outputs the sleepiness level determination result to the microsleep estimation unit 16B.

If the closed-eye time detection unit 22 detects that the closed-eye time is less than 0.5 seconds or longer than 3 seconds (NO in S202), the reliability calculation unit 24 of the closed-eye state detection unit 12B detects that the closed-eye time is A reliability, which is an index indicating the probability of the detection result of the closed-eye time detecting unit 22 to be less than 0.5 seconds or more than 3 seconds, is calculated (S203). After that, the estimation unit 18 of the microsleep estimation unit 16B estimates that the driver is not in the microsleep state (S204). The reliability calculation unit 20B of the micro-sleep estimation unit 16B determines the likelihood of the estimation result of the estimation unit 18 that the driver is not in the micro-sleep state, based on the calculation result of the reliability calculation unit 24 of the closed-eye state detection unit 12B. is calculated (S205).

Returning to step S202, when the eye-closed time detection unit 22 detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds (YES in S202), the reliability calculation unit 24 of the eye-closed state detection unit 12B , the degree of reliability, which is an index indicating the probability of the detection result of the eye closure time detection unit 22 that the eye closure time is 0.5 seconds or more and less than 3 seconds, is calculated (S206).

Thereafter, when the drowsiness level determination unit 26 determines that the drowsiness level is "4" or higher (YES in S207), the reliability calculation unit 28 of the drowsiness determination unit 14B determines that the drowsiness level is "4" or higher. A reliability, which is an index indicating the probability of the determination result of the drowsiness level determination unit 26, is calculated (S208). The estimating unit 18 of the micro-sleep estimating unit 16B estimates that the driver is in the micro-sleep state (S209), and the reliability calculating unit 20B of the micro-sleep estimating unit 16B detects the reliability calculating unit 24 of the closed-eye state detecting unit 12B. and the calculation result of the reliability calculation unit 28 of the drowsiness determination unit 14B. Calculate (S210).

Returning to step S207, when the drowsiness level determination unit 26 determines that the drowsiness level is less than "4" (NO in S207), the reliability calculation unit 28 of the drowsiness determination unit 14B determines that the drowsiness level is "4". '' is calculated (S211), and the process proceeds to step S209.

[3-3. effect]
In this embodiment, the microsleep state can be estimated with even higher accuracy.

(Embodiment 4)
[4-1. Configuration of estimation device]
A configuration of an estimation device 2C according to Embodiment 4 will be described with reference to FIG. FIG. 7 is a block diagram showing the configuration of an estimation device 2C according to Embodiment 4. As shown in FIG. In addition, in this embodiment, the same reference numerals are given to the same constituent elements as in the above-described third embodiment, and the description thereof will be omitted.

As shown in FIG. 7, the estimation device 2C according to Embodiment 4 includes an image information acquisition unit 8, a biological information acquisition unit 10, an eye-closed state detection unit 12B, a drowsiness determination unit 14B, and a microsleep estimation unit 16C. An open/close state detector 30 is provided.

The opening/closing state detection unit 30 has an opening/closing speed calculation unit 32, an opening/closing speed determination unit 34, and a reliability calculation unit 36. The opening/closing speed calculation unit 32 calculates (detects) the opening/closing speed of the driver's eyes based on the image information from the image information acquisition unit 8 . Note that the opening/closing speed calculator 32 may calculate the opening/closing speed of the driver's eyes using, for example, deep learning. The opening/closing speed determination unit 34 determines whether or not the calculated opening/closing speed is less than a threshold (an example of a second threshold). The reliability calculation unit 36 calculates a reliability (an example of a fourth reliability) that is an index indicating the likelihood of the determination result of the opening/closing speed determination unit 34 that the calculated opening/closing speed is less than the threshold. Further, the reliability calculation unit 36 calculates a reliability, which is an index indicating the likelihood of the determination result of the opening/closing speed determination unit 34 that the calculated opening/closing speed is equal to or higher than the threshold. The reliability is calculated as a numerical value between 0% and 100%, for example. The reliability calculation unit 36 outputs the reliability calculation result to the microsleep estimation unit 16C.

The estimating unit 18 of the microsleep estimating unit 16C detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds by the closed-eye state detecting unit 12B, and the drowsiness determining unit 14B detects that the drowsiness level is "4" or higher. On the condition that the opening/closing state detection unit 30 detects that the opening/closing speed is less than the threshold value, it is estimated that the driver is in the micro-sleep state.

The reliability calculation unit 20C of the microsleep estimation unit 16C uses the calculation result of the reliability calculation unit 24 of the closed-eye state detection unit 12B, the calculation result of the reliability calculation unit 28 of the drowsiness determination unit 14B, and the open/closed state detection unit 30. Based on the calculation result of the reliability calculation unit 36, the reliability, which is an index indicating the probability of the estimation result of the estimation unit 18 that the driver is in the micro-sleep state, is calculated. Further, the reliability calculation unit 20C of the micro-sleep estimation unit 16C determines the estimation result of the estimation unit 18 that the driver is not in the micro-sleep state, based on the calculation result of the reliability calculation unit 24 of the closed-eye state detection unit 12B. Reliability, which is an index indicating likelihood, is calculated. The reliability is calculated as a numerical value between 0% and 100%, for example.

In the present embodiment, the opening/closing state detection unit 30 has the opening/closing speed calculation unit 32 and the opening/closing speed determination unit 34. However, the present invention is not limited to this. A speed determination unit 34 may be provided. That is, the drowsiness determination unit 14B may include a function as the open/closed state detection unit 30. FIG.

Alternatively, the drowsiness determination unit 14B may have an eye closing time detection unit 22 in addition to the opening/closing speed calculation unit 32 and the opening/closing speed determination unit 34 . That is, the drowsiness determination unit 14B may include the functions of the closed eye state detection unit 12B and the open/closed state detection unit 30. FIG.

[4-2. Operation of estimation device]
Next, operation of the estimation device 2C according to Embodiment 4 will be described with reference to FIG. FIG. 8 is a flow chart showing the operation flow of the estimation device 2C according to the fourth embodiment. In the flowchart of FIG. 8, the same step numbers are assigned to the same processes as those of the above-described flowchart of FIG. 6, and the description thereof will be omitted.

As shown in FIG. 8, steps S201 to S208 and S211 are executed in the same manner as in the third embodiment. After step S208 or step S211, when the opening/closing speed of less than the threshold is detected by the opening/closing state detection unit 30 (that is, the opening/closing speed determination unit 34 of the opening/closing state detection unit 30 determines that the opening/closing speed is less than the threshold). (YES in S301), the reliability calculation unit 36 of the opening/closing state detection unit 30 calculates the reliability, which is an index indicating the likelihood of the determination result of the opening/closing speed determination unit 34 that the detected opening/closing speed is less than the threshold value. is calculated (S302).

After that, the estimation unit 18 of the microsleep estimation unit 16C estimates that the driver is in the microsleep state (S209), and the reliability calculation unit 20C of the microsleep estimation unit 16C calculates the reliability of the closed-eye state detection unit 12B. Based on the calculation result of the unit 24, the calculation result of the reliability calculation unit 28 of the drowsiness determination unit 14B, and the calculation result of the reliability calculation unit 36 of the open/closed state detection unit 30, it is determined that the driver is in the micro sleep state. A reliability, which is an index indicating the likelihood of the estimation result of the estimation unit 18, is calculated (S210).

Returning to step S301, when the opening/closing speed of the threshold value or more is detected by the opening/closing state detection unit 30 (that is, the opening/closing speed determination unit 34 of the opening/closing state detection unit 30 determines the opening/closing speed of the threshold value or more) (S301 NO), the reliability calculation unit 36 of the opening/closing state detection unit 30 calculates reliability, which is an index indicating the likelihood of the determination result of the opening/closing speed determination unit 34 that the detected opening/closing speed is equal to or greater than the threshold. (S303). Then, it progresses to step S209 mentioned above.

Note that the eye-closed state detection unit 12B, the drowsiness determination unit 14B, and the open/closed state detection unit 30 may not include the reliability calculation unit 24, the reliability calculation unit 28, and the reliability calculation unit 36, respectively. may omit steps S203, S206, S208, S211, S302 and S303. In this case, instead of step S210 described above, the eye-closed state detection unit 12B detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds (YES in S202), and the drowsiness determination unit 14B sets the drowsiness level to "4." ' (YES in S207), and when the opening/closing state detection unit 30 detects that the opening/closing speed is less than the threshold (YES in S301), the reliability calculation unit of the microsleep estimation unit 16C 20C may calculate a reliability of "high". On the other hand, if the drowsiness determination unit 14B determines that the drowsiness level is less than "4" (NO in S207), or if the opening/closing state detection unit 30 detects that the opening/closing speed is equal to or greater than the threshold value, (NO in S301), the reliability calculation unit 20C of the microsleep estimation unit 16C may calculate the reliability "low".

[4-3. effect]
In this embodiment, the microsleep state can be estimated with even higher accuracy.

(Embodiment 5)
[5-1. Configuration of estimation device]
A configuration of an estimation device 2D according to Embodiment 5 will be described with reference to FIG. FIG. 9 is a block diagram showing the configuration of an estimation device 2D according to Embodiment 5. As shown in FIG. In addition, in this embodiment, the same reference numerals are given to the same constituent elements as in the above-described third embodiment, and the description thereof will be omitted.

As shown in FIG. 9, the estimation device 2D according to Embodiment 5 includes an image information acquisition unit 8, a biological information acquisition unit 10, an eye-closed state detection unit 12B, a drowsiness determination unit 14B, and a microsleep estimation unit 16D. An open/close state detector 38 is provided.

The open/close state detection unit 38 has an eye closing speed calculation unit 40 , an eye closing speed determination unit 42 , an eye opening speed calculation unit 44 , an eye opening speed determination unit 46 , and a reliability calculation unit 48 . The eye closing speed calculator 40 calculates (detects) the driver's eye closing speed based on the image information from the image information acquisition unit 8 . Note that the eye closing speed calculator 40 may calculate the driver's eye closing speed using, for example, deep learning. The eye-closing speed determination unit 42 determines whether or not the calculated eye-closing speed is less than the eye-closing threshold (an example of the third threshold). The eye-opening speed calculator 44 calculates (detects) the driver's eye-opening speed based on the image information from the image information acquisition unit 8 . Note that the eye-opening speed calculator 44 may calculate the eye-opening speed of the driver using, for example, deep learning. The eye-opening speed determination unit 46 determines whether or not the calculated eye-opening speed is less than an eye-opening threshold (an example of a fourth threshold).

The reliability calculation unit 48 calculates a reliability (an example of a fifth reliability) that is an index indicating the likelihood of the detection result that the calculated eye-closing speed is less than the eye-closing threshold. The reliability calculation unit 48 also calculates a reliability (an example of a sixth reliability), which is an index indicating the probability of the detection result that the calculated eye-opening speed is less than the eye-opening threshold. The reliability is calculated as a numerical value between 0% and 100%, for example. The reliability calculation unit 48 outputs the reliability calculation result to the microsleep estimation unit 16D.

The estimating unit 18 of the microsleep estimating unit 16D detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds by the closed-eye state detecting unit 12B, and the drowsiness determining unit 14B detects that the drowsiness level is "4" or more. In addition, the open/close state detection unit 38 detects that the eye closing speed is less than the eye closing threshold, and the open/close state detection unit 38 detects that the eye opening speed is less than the eye opening threshold. , the driver is assumed to be in a micro-sleep state.

The reliability calculation unit 20D of the microsleep estimation unit 16D uses the calculation result of the reliability calculation unit 24 of the closed-eye state detection unit 12B, the calculation result of the reliability calculation unit 28 of the drowsiness determination unit 14B, and the open/closed state detection unit 38. Based on the calculation result of the reliability calculation unit 48, the reliability, which is an index indicating the likelihood of the estimation result of the estimation unit 18 that the driver is in the micro-sleep state, is calculated. Further, the reliability calculation unit 20D of the micro-sleep estimation unit 16D determines the estimation result of the estimation unit 18 that the driver is not in the micro-sleep state, based on the calculation result of the reliability calculation unit 24 of the closed-eye state detection unit 12B. Reliability, which is an index indicating likelihood, is calculated. The reliability is calculated as a numerical value between 0% and 100%, for example.

[5-2. Operation of estimation device]
Next, operation of the estimation device 2D according to Embodiment 5 will be described with reference to FIG. FIG. 10 is a flow chart showing the operation flow of the estimation device 2D according to the fifth embodiment. In the flowchart of FIG. 10, the same step numbers are given to the same processes as those of the above-described flowchart of FIG. 6, and the description thereof will be omitted.

As shown in FIG. 10, steps S201 to S208 and S211 are executed as in the third embodiment. After step S208 or step S211, if the open/close state detection unit 38 detects an eye closing speed that is less than the eye closing threshold and an eye opening speed that is less than the eye opening threshold (YES in S401), the reliability calculation unit of the open/close state detection unit 38 48 calculates reliability, which is an index indicating the probability of the detection result that the detected eye-closing speed is less than the eye-closing threshold and the detection result that the detected eye-opening speed is less than the eye-opening threshold ( S402).

Thereafter, the estimating unit 18 of the micro-sleep estimating unit 16D estimates that the driver is in the micro-sleep state (S209), and the reliability calculating unit 20D of the micro-sleep estimating unit 16D calculates the reliability of the closed-eye state detecting unit 12B. Based on the calculation result of the unit 24, the calculation result of the reliability calculation unit 28 of the drowsiness determination unit 14B, and the calculation result of the reliability calculation unit 48 of the open/closed state detection unit 38, it is determined that the driver is in the micro sleep state. A reliability, which is an index indicating the likelihood of the estimation result of the estimation unit 18, is calculated (S210).

Returning to step S401, when the open/closed state detection unit 38 detects an eye-closing speed less than the eye-closing threshold and an eye-opening speed equal to or higher than the eye-opening threshold (NO in S401, YES in S403), the reliability of the open/closed state detection unit 38 is calculated. The unit 48 calculates reliability, which is an index indicating the likelihood of the detection result that the detected eye-closing speed is less than the eye-closing threshold and the detection result that the detected eye-opening speed is equal to or greater than the eye-opening threshold. (S404). Note that the reliability calculated in step S404 is lower than the reliability calculated in step S402. Then, it progresses to step S209 mentioned above.

Returning to step S401, when the open/close state detection unit 38 detects an eye-closing speed equal to or higher than the eye-closing threshold and an eye-opening speed less than the eye-opening threshold (NO in S401, NO in S403, YES in S405), the open/close state detection unit 38 The reliability calculation unit 48 is an index indicating the probability of the detection result that the detected eye-closing speed is equal to or higher than the eye-closing threshold and the detection result that the detected eye-opening speed is less than the eye-opening threshold. degree is calculated (S406). Note that the reliability calculated in step S406 is lower than the reliability calculated in step S404. Alternatively, the reliability calculated in step S406 may be the same as the reliability calculated in step S404. Then, it progresses to step S209 mentioned above.

Returning to step S401, when the open/closed state detection unit 38 detects the eye closing speed equal to or higher than the eye closing threshold and the eye opening speed equal to or higher than the eye opening threshold (NO in S401, NO in S403, NO in S405, S407), open/closed state detection is performed. The reliability calculation unit 48 of the unit 38 is an index indicating the probability of the detection result that the detected eye-closing speed is equal to or higher than the eye-closing threshold and the detection result that the detected eye-opening speed is equal to or higher than the eye-opening threshold. A certain reliability is calculated (S408). The reliability calculated in step S408 is lower than the reliability calculated in step S406. Then, it progresses to step S209 mentioned above.

[5-3. effect]
In this embodiment, the microsleep state can be estimated with even higher accuracy.

(Embodiment 6)
[6-1. Configuration of estimation device]
The configuration of an estimation device 2E according to Embodiment 6 will be described with reference to FIG. FIG. 11 is a block diagram showing the configuration of an estimation device 2E according to Embodiment 6. As shown in FIG. In addition, in the present embodiment, the same reference numerals are given to the same constituent elements as in the above-described second embodiment, and the description thereof will be omitted.

As shown in FIG. 11, in an estimation device 2E according to Embodiment 6, each configuration of an eye-closed state detection unit 12E and a microsleep estimation unit 16E is different from that in Embodiment 2 above.

The eye-closed state detection unit 12E has an eye-closed time detection unit 22 and a both-eyes detection unit 50. The closed-eye time detection unit 22 is the same as the closed-eye time detection unit 22 described in the third embodiment. The binocular detection unit 50 detects the binoculars of the driver based on the image information from the image information acquisition unit 8 . The binocular detection unit 50 outputs the detection result to the microsleep estimation unit 16E. Note that the binocular detection unit 50 may detect the binoculars of the driver based on the biometric information from the biometric information acquisition unit 10 .

The estimating unit 18 of the microsleep estimating unit 16E detects that the closed eye time is 0.5 seconds or more and less than 3 seconds by the eye closing time detecting unit 22, and the drowsiness determining unit 14 detects that the drowsiness level is "4" or more. It is assumed that the driver is in the micro-sleep state on condition that both eyes of the driver are detected by the binocular detection unit 50 .

The reliability calculation unit 20E of the micro-sleep estimation unit 16E calculates reliability, which is an index indicating the likelihood of the estimation result of the estimation unit 18 that the driver is in the micro-sleep state. The reliability is calculated, for example, in three levels of "low", "middle" and "high".

[6-2. Operation of estimation device]
Next, operation of the estimation device 2E according to Embodiment 6 will be described with reference to FIG. FIG. 12 is a flow chart showing the operation flow of the estimation device 2E according to the sixth embodiment. In the flowchart of FIG. 12, the same step numbers are given to the same processes as those of the above-described flowchart of FIG. 4, and the description thereof will be omitted.

As shown in FIG. 12, first, after step S101 is executed in the same manner as in the second embodiment, when both eyes detection unit 50 detects both eyes of the driver (YES in S501), the above execution Steps S102 to S106 are executed in the same manner as in the second form. That is, the estimating unit 18 of the micro-sleep estimating unit 16E determines that both eyes of the driver are detected by the binocular detecting unit 50 (YES in S501), and the eye-closing time detecting unit 22 determines that the eye-closing time is 0.5 seconds or longer. second (YES in S102), and the drowsiness determination unit 14 determines that the drowsiness level is "4" or higher (YES in S104). (S106).

Returning to step S501, for example, either the left or right eye of the driver is hidden by the bangs, or the driver is wearing an eyepatch, and the binocular detection unit 50 detects only one eye of the driver. If so (NO in S501, YES in S502), the process proceeds to step S503. When the eye-closing time detecting unit 22 detects that the eye-closing time is less than 0.5 seconds or longer than 3 seconds (NO in S503), the estimating unit 18 of the microsleep estimating unit 16E determines that the driver is in microsleep. It is estimated that it is not in the state (S103). In this case, the reliability calculator 20E of the microsleep estimator 16E does not calculate the reliability.

Returning to step S503, when the eye-closed time detection unit 22 detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds (YES in S503), the drowsiness determination unit 14 sets the drowsiness level to "4". If it is determined to be above (YES in S504), the reliability calculation unit 20E of the microsleep estimation unit 16E calculates the reliability “medium” (S505), and the estimation unit 18 of the microsleep estimation unit 16E estimates that the driver is in a micro-sleep state (S106).

Returning to step S504, when the drowsiness determination unit 14 determines that the drowsiness level is less than "4" (NO in S504), the reliability calculation unit 20E of the microsleep estimation unit 16E sets the reliability to "low". (S506), and the estimation unit 18 of the microsleep estimation unit 16E estimates that the driver is in the microsleep state (S106).

[6-3. effect]
In this embodiment, the microsleep state can be estimated with even higher accuracy.

(Embodiment 7)
[7-1. Configuration of estimation device]
The configuration of an estimation device 2F according to Embodiment 7 will be described with reference to FIG. FIG. 13 is a block diagram showing the configuration of an estimation device 2F according to Embodiment 7. As shown in FIG. In addition, in the present embodiment, the same reference numerals are given to the same constituent elements as in the above-described second embodiment, and the description thereof will be omitted.

As shown in FIG. 13, the estimation device 2F according to Embodiment 7 includes an image information acquisition unit 8, a biological information acquisition unit 10, a closed eye state detection unit 12, a drowsiness determination unit 14, and a microsleep estimation unit 16F. A blink number detection unit 52 is provided. The number-of-blinks detector 52 detects the number of times the driver blinks per unit time (for example, per minute) based on the image information from the image information acquisition unit 8 . The number-of-blinks detector 52 outputs the detection result to the microsleep estimator 16F. The number-of-blinks detection unit 52 may detect the number of times the driver blinks per unit time based on the biological information from the biological information acquisition unit 10 .

The estimating unit 18 of the microsleep estimating unit 16F detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds by the closed-eye state detecting unit 12, and the drowsiness determining unit 14 detects that the drowsiness level is "4" or higher. In addition, the number of blinks detection unit 52 detects that the number of times the driver blinks per unit time has increased or decreased by a predetermined number (for example, 10 times/minute) (an example of a fifth threshold value) or more. on the condition that the driver is in a micro-sleep state. This is because when a person becomes sleepy, the number of blinks per unit time often increases or decreases.

[7-2. Operation of estimation device]
Next, the operation of the estimation device 2F according to Embodiment 7 will be described with reference to FIG. FIG. 14 is a flowchart showing the operation flow of the estimation device 2F according to the seventh embodiment. In the flowchart of FIG. 14, the same step numbers are given to the same processes as those of the above-described flowchart of FIG. 4, and the description thereof will be omitted.

As shown in FIG. 14, steps S101 to S104 are first executed in the same manner as in the second embodiment. When the closed-eye state detection unit 12 detects that the closed-eye time is 0.5 seconds or more and less than 3 seconds (YES in S102), the drowsiness determination unit 14 determines that the drowsiness level is "4" or more. If so (YES in S104), the number-of-blinks detector 52 detects the number of times the driver blinks per unit time.

When the number of blinks per unit time has increased or decreased by a predetermined number or more (YES in S601), the reliability calculation unit 20F of the microsleep estimation unit 16F calculates the reliability "high" (S105), and microsleep The estimating unit 18 of the estimating unit 16F estimates that the driver is in the micro-sleep state (S106).

Returning to step S601, if the number of blinks per unit time has not increased or decreased by a predetermined number or more (NO in S601), the reliability calculation unit 20F of the microsleep estimation unit 16F calculates the reliability as "low." Then (S107), the estimation unit 18 of the microsleep estimation unit 16F estimates that the driver is in the microsleep state (S106).

[7-3. effect]
In this embodiment, the microsleep state can be estimated with even higher accuracy.

(Embodiment 8)
[8-1. Configuration of estimation device]
The configuration of an estimation device 2G according to Embodiment 8 will be described with reference to FIG. FIG. 15 is a block diagram showing the configuration of an estimation device 2G according to the eighth embodiment. In addition, in the present embodiment, the same reference numerals are given to the same constituent elements as in the above-described second embodiment, and the description thereof will be omitted.

As shown in FIG. 15 , an estimation device 2G according to Embodiment 8 includes an image information acquisition unit 8, a biological information acquisition unit 10, a closed eye state detection unit 12, a drowsiness determination unit 14, and a microsleep estimation unit 16G. A life log information acquisition unit 54 is provided. Based on the image information from the image information acquisition unit 8, the life log information acquisition unit 54 acquires life log information related to the life of the driver. The life log information is, for example, information indicating the body type of the driver (thin, obese, etc.). Life log information acquisition unit 54 outputs the acquired life log information to microsleep estimation unit 16G. Note that the life log information acquisition unit 54 may acquire life log information based on the biometric information from the biometric information acquisition unit 10 .

The estimating unit 18 of the microsleep estimating unit 16G detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds by the closed-eye state detecting unit 12, and the drowsiness determining unit 14 detects that the drowsiness level is less than "4". It is assumed that the driver is in the micro-sleep state on the condition that it is determined that the driver is in the micro-sleep state and that the life log information acquisition unit 54 has acquired life log information that affects the micro-sleep state of the driver. Here, the lifestyle log information that affects the driver's micro-sleep state is, for example, information indicating that the driver's body type is obese. This is because obese people are prone to sleep apnea and fall asleep suddenly.

The reliability calculation unit 20G of the micro-sleep estimation unit 16G calculates reliability, which is an index indicating the likelihood of the estimation result of the estimation unit 18 that the driver is in the micro-sleep state. The reliability is calculated, for example, in three levels of "low", "middle" and "high".

In addition, in the present embodiment, the life log information acquisition unit 54 acquires the life log information based on the image information from the image information acquisition unit 8, but is not limited to this. For example, the life log information acquisition unit 54 may acquire life log information from a wearable terminal worn by the driver, or may acquire life log information from a cloud server via a network. In this case, the life log information includes, for example, (a) the driver's medical history, (b) the driver's previous day's working hours, (c) the driver's work style (night shift, etc.), and (d) the driver's exercise time. , (e) the driver's bedtime, (f) the driver's sleep quality (long sleeper, etc.), and (g) the driver's medication history.

[8-2. Operation of estimation device]
Next, operation of the estimation device 2G according to Embodiment 8 will be described with reference to FIG. FIG. 16 is a flow chart showing the operation flow of the estimation device 2G according to the eighth embodiment. In the flowchart of FIG. 16, the same step numbers are given to the same processes as those of the above-described flowchart of FIG. 4, and the description thereof will be omitted.

As shown in FIG. 16, steps S101 to S104 are first executed in the same manner as in the second embodiment. When the eye-closed state detection unit 12 detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds (YES in S102), the drowsiness determination unit 14 determines that the drowsiness level is less than "4". If so (NO in S104), the process proceeds to step S701, and the life log information acquisition unit 54 acquires life log information.

When the life log information that affects the driver's micro-sleep state is acquired by the life log information acquisition unit 54 (YES in S701), the reliability calculation unit 20G of the micro-sleep estimation unit 16G sets the reliability to "medium." (S702), and the estimation unit 18 of the microsleep estimation unit 16G estimates that the driver is in the microsleep state (S106).

That is, in the estimating unit 18 of the microsleep estimating unit 16G, the eye-closed state detection unit 12 detects that the closed-eye time is 0.5 seconds or more and less than 3 seconds (YES in S102), and the drowsiness determination unit 14 detects drowsiness. It is determined that the level is less than "4" (NO in S104), and the life log information that affects the driver's microsleep state is acquired by the life log information acquisition unit 54 (YES in S701). , the driver is assumed to be in a micro-sleep state. This is because even if the driver does not feel very drowsy, there is a possibility that the driver will close his/her eyes because of his/her tendency to fall asleep suddenly.

Returning to step S701, when the life log information acquisition unit 54 does not acquire the life log information that affects the microsleep state of the driver (NO in S701), the reliability calculation unit 20G of the microsleep estimation unit 16G The degree "low" is calculated (S107), and the estimation unit 18 of the microsleep estimation unit 16G estimates that the driver is in the microsleep state (S106).

[8-3. effect]
In this embodiment, the microsleep state can be estimated with even higher accuracy.

(Embodiment 9)
[9-1. Configuration of estimation device]
The configuration of an estimation device 2H according to Embodiment 9 will be described with reference to FIG. FIG. 17 is a block diagram showing the configuration of an estimation device 2H according to Embodiment 9. As shown in FIG. In addition, in the present embodiment, the same reference numerals are given to the same constituent elements as in the above-described second embodiment, and the description thereof will be omitted.

As shown in FIG. 17, the estimation device 2H according to the ninth embodiment includes an image information acquisition unit 8, a biological information acquisition unit 10, a closed eye state detection unit 12, a drowsiness determination unit 14, and a microsleep estimation unit 16H. A facial feature information acquisition unit 56 is provided. Based on the image information from the image information acquisition unit 8, the facial feature information acquisition unit 56 acquires facial feature information indicating the facial features of the driver in time series. Facial features refer to parts of the face such as eyes and mouth. The facial feature information acquiring section 56 outputs the acquired facial feature information to the microsleep estimating section 16H. Note that the facial feature information acquisition section 56 may acquire facial feature information based on the biometric information from the biometric information acquisition section 10 .

The estimating unit 18 of the microsleep estimating unit 16H detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds by the closed-eye state detecting unit 12, and the drowsiness determining unit 14 detects that the drowsiness level is less than "4". It is assumed that the driver is in the micro-sleep state on the condition that it is determined that the driver is in the micro-sleep state and that the facial characteristic information acquiring unit 56 acquires facial feature information that affects the micro-sleep state of the driver. Here, the facial feature information that affects the driver's micro-sleep state means, for example, that the driver's mouth is open due to drowsiness, or that eyebrows are not moving due to drowsiness and the muscles around the eyes are relaxed. This is the information shown.

The reliability calculation unit 20H of the micro-sleep estimation unit 16H calculates reliability, which is an index indicating the likelihood of the estimation result of the estimation unit 18 that the driver is in the micro-sleep state. The reliability is calculated, for example, in three levels of "low", "middle" and "high".

[9-2. Operation of estimation device]
Next, the operation of the estimating device 2H according to Embodiment 9 will be described with reference to FIG. FIG. 18 is a flow chart showing the operation flow of the estimating device 2H according to the ninth embodiment. In the flowchart of FIG. 18, the same step numbers are given to the same processes as those of the above-described flowchart of FIG. 4, and the description thereof will be omitted.

As shown in FIG. 18, steps S101 to S104 are first executed in the same manner as in the second embodiment. When the eye-closed state detection unit 12 detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds (YES in S102), the drowsiness determination unit 14 determines that the drowsiness level is less than "4". If so (NO in S104), the process proceeds to step S801, and the facial feature information acquiring unit 56 acquires facial feature information.

When facial feature information that affects the driver's microsleep state is acquired by the facial feature information acquisition unit 56 (YES in S801), the reliability calculation unit 20H of the microsleep estimation unit 16H sets the reliability to "medium." (S802), and the estimation unit 18 of the microsleep estimation unit 16H estimates that the driver is in the microsleep state (S106).

That is, in the estimating unit 18 of the microsleep estimating unit 16H, the eye-closed state detection unit 12 detects that the closed-eye time is 0.5 seconds or more and less than 3 seconds (YES in S102), and the drowsiness determination unit 14 It is determined that the level is less than "4" (NO in S104), and facial feature information that affects the driver's micro-sleep state is acquired by the facial feature information acquisition unit 56 (YES in S801). , the driver is assumed to be in a micro-sleep state. This is because the driver may close his/her eyes due to fatigue or the like even if the driver does not feel drowsy.

Returning to step S801, when facial feature information that affects the driver's microsleep state is not acquired by the facial feature information acquisition unit 56 (NO in S801), the reliability calculation unit 20H of the microsleep estimation unit 16H The degree "low" is calculated (S107), and the estimation unit 18 of the microsleep estimation unit 16G estimates that the driver is in the microsleep state (S106).

[9-3. effect]
In this embodiment, the microsleep state can be estimated with even higher accuracy.

(Embodiment 10)
[10-1. Configuration of estimation device]
A configuration of an estimation device 2J according to the tenth embodiment will be described with reference to FIG. FIG. 19 is a block diagram showing the configuration of an estimation device 2J according to the tenth embodiment. In addition, in the present embodiment, the same reference numerals are given to the same constituent elements as in the above-described second embodiment, and the description thereof will be omitted.

As shown in FIG. 19, the estimation device 2J according to Embodiment 10 includes an image information acquisition unit 8, a biological information acquisition unit 10, an eye-closed state detection unit 12, a drowsiness determination unit 14, and a microsleep estimation unit 16J. A head motion detector 58 is provided. Based on the image information from the image information acquisition unit 8, the head movement detection unit 58 detects movements of the driver's head in time series. The head motion detector 58 outputs the detection result to the microsleep estimator 16J. Note that the head motion detection unit 58 may detect head motion based on the biometric information from the biometric information acquisition unit 10 .

The estimating unit 18 of the microsleep estimating unit 16J detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds by the closed-eye state detecting unit 12, and the drowsiness determining unit 14 detects that the drowsiness level is less than "4". It is assumed that the driver is in the micro-sleep state on the condition that it is determined that there is a micro-sleep state and that the head motion detector 58 acquires a head motion that affects the micro-sleep state of the driver. Here, the motion of the head that affects the driver's micro-sleep state is, for example, a so-called motion such as nodding, in which the driver's head shakes up and down due to drowsiness.

The reliability calculation unit 20J of the microsleep estimation unit 16J calculates the reliability, which is an index indicating the likelihood of the estimation result of the estimation unit 18 that the driver is in the microsleep state. The reliability is calculated, for example, in three levels of "low", "middle" and "high".

[10-2. Operation of estimation device]
Next, operation of the estimation device 2J according to the tenth embodiment will be described with reference to FIG. FIG. 20 is a flowchart showing the operation flow of the estimation device 2J according to the tenth embodiment. In the flowchart of FIG. 20, the same step numbers are assigned to the same processes as those of the above-described flowchart of FIG. 4, and the description thereof will be omitted.

As shown in FIG. 20, steps S101 to S104 are first executed in the same manner as in the second embodiment. When the eye-closed state detection unit 12 detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds (YES in S102), the drowsiness determination unit 14 determines that the drowsiness level is less than "4". If so (NO in S104), the process proceeds to step S901, and the head movement detection unit 58 detects the movement of the driver's head.

When the head motion detector 58 detects a head motion that affects the driver's microsleep state (YES in S901), the reliability calculator 20J of the microsleep estimator 16J determines that the reliability is medium. (S902), and the estimation unit 18 of the microsleep estimation unit 16J estimates that the driver is in the microsleep state (S106).

That is, in the estimating unit 18 of the microsleep estimating unit 16J, the eye-closed state detection unit 12 detects that the closed-eye time is 0.5 seconds or more and less than 3 seconds (YES in S102), and the drowsiness determination unit 14 It is determined that the level is less than "4" (NO in S104), and the head motion detection unit 58 detects a head motion that affects the driver's microsleep state (YES in S901). As a condition, it is assumed that the driver is in a microsleep state. This is because the driver may close his/her eyes due to fatigue or the like even if the driver does not feel drowsy.

Returning to step S901, when the head motion detection unit 58 does not acquire a head motion that affects the driver's microsleep state (NO in S901), the reliability calculation unit 20J of the microsleep estimation unit 16J The reliability "low" is calculated (S107), and the estimation unit 18 of the microsleep estimation unit 16J estimates that the driver is in the microsleep state (S106).

[10-3. effect]
In this embodiment, the microsleep state can be estimated with even higher accuracy.

(Embodiment 11)
[11-1. Configuration of estimation device]
A configuration of an estimation device 2K according to Embodiment 11 will be described with reference to FIG. FIG. 21 is a block diagram showing the configuration of an estimating device 2K according to Embodiment 11. As shown in FIG. In addition, in the present embodiment, the same reference numerals are given to the same constituent elements as in the above-described second embodiment, and the description thereof will be omitted.

As shown in FIG. 21, the estimation device 2K according to Embodiment 11 includes an image information acquisition unit 8, a biological information acquisition unit 10, an eye-closed state detection unit 12, a drowsiness determination unit 14, and a microsleep estimation unit 16K. An erroneous estimation situation detection unit 60 is provided. Based on the image information from the image information acquisition unit 8, the erroneous estimation situation detection unit 60 detects a situation that affects the estimation of the driver's micro-sleep state by the estimation unit 18 (hereinafter referred to as "erroneous estimation situation"). do. The erroneously estimated situation is, for example, a solar situation where the driver's face is exposed to the afternoon sun, or a road situation where the driver's face is in fine shadows such as roadside trees. The erroneous estimation condition detection unit 60 outputs the detection result to the microsleep estimation unit 16K. The erroneous estimation situation detection unit 60 may detect an erroneous estimation situation based on the biometric information from the biometric information acquisition unit 10 .

The reliability calculation unit 20K of the microsleep estimating unit 16K is an index indicating the likelihood of the estimation result of the estimating unit 18 that the driver is in the microsleep state, taking into consideration the detection result of the erroneous estimation state detecting unit 60. Calculate a certain reliability. The reliability is calculated, for example, in three levels of "low", "middle" and "high".

[11-2. Operation of estimation device]
Next, the operation of the estimation device 2K according to Embodiment 11 will be described with reference to FIG. FIG. 22 is a flow chart showing the operation flow of the estimating device 2K according to the eleventh embodiment. In the flowchart of FIG. 22, the same step numbers are given to the same processes as those of the above-described flowchart of FIG. 4, and the description thereof will be omitted.

As shown in FIG. 22, steps S101 to S104 are first executed in the same manner as in the second embodiment. When the eye-closed state detection unit 12 detects that the eye-closed time is 0.5 seconds or more and less than 3 seconds (YES in S102), the drowsiness determination unit 14 determines that the drowsiness level is less than "4". If so (NO in S104), the process proceeds to step S1001, and the erroneous estimation situation detection unit 60 detects the presence or absence of an erroneous estimation situation.

When the erroneous estimation state detection unit 60 detects an erroneous estimation state (YES in S1001), the reliability calculation unit 20K of the microsleep estimating unit 16K calculates the reliability “medium” (S1002), and the microsleep state is detected. The estimating unit 18 of the estimating unit 16K estimates that the driver is in the micro-sleep state (S106).

Returning to step S1001, if the erroneous estimation condition detection unit 60 does not detect an erroneous estimation condition (NO in S1001), the reliability calculation unit 20K of the microsleep estimation unit 16K calculates the reliability “low” (S107 ), the estimator 18 of the microsleep estimator 16K estimates that the driver is in the microsleep state (S106).

[11-3. effect]
In this embodiment, the microsleep state can be estimated with even higher accuracy.

(Embodiment 12)
[12-1. Configuration of estimation device]
A configuration of an estimation device 2L according to Embodiment 12 will be described with reference to FIG. FIG. 23 is a block diagram showing the configuration of an estimation device 2L according to the twelfth embodiment. In addition, in the present embodiment, the same reference numerals are given to the same constituent elements as in the above-described second embodiment, and the description thereof will be omitted.

As shown in FIG. 23, the estimation device 2L according to the twelfth embodiment includes an image information acquisition unit 8, a biological information acquisition unit 10, an eye-closed state detection unit 12, a drowsiness determination unit 14, and a microsleep estimation unit 16L. An erroneous estimation situation detection unit 62 is provided. The erroneous estimation condition detection unit 62 detects an erroneous estimation condition based on the image information from the image information acquisition unit 8 or the biometric information from the biometric information acquisition unit 10 . The erroneously estimated situation is, for example, a solar situation where the driver's face is exposed to the afternoon sun, or a road situation where the driver's face is in fine shadows such as roadside trees. The erroneous estimation situation detection unit 62 outputs the detection result to the microsleep estimation unit 16L via the closed eye state detection unit 12 and the drowsiness determination unit 14 .

The estimating unit 18 of the microsleep estimating unit 16L does not estimate the driver's microsleep state when the erroneous estimation state detection unit 62 detects an erroneous estimation state.

[12-2. Operation of estimation device]
Next, the operation of the estimation device 2L according to Embodiment 12 will be described with reference to FIG. FIG. 24 is a flowchart showing the operation flow of the estimation device 2L according to the twelfth embodiment. In addition, in the flowchart of FIG. 24, the same step numbers are assigned to the same processes as those of the flowchart of FIG. 4, and the description thereof is omitted.

As shown in FIG. 24, first, similarly to the second embodiment, after step S101 is executed, the erroneous estimation situation detection unit 62 detects the presence or absence of an erroneous estimation situation. If the erroneous estimation situation detection unit 62 does not detect an erroneous estimation situation (NO in S1101), the process proceeds to step S102, and steps S102 to S107 are executed in the same manner as in the second embodiment. On the other hand, if the erroneous estimation state detection unit 62 detects an erroneous estimation state (YES in S1101), the estimating unit 18 of the microsleep estimating unit 16L does not estimate the microsleep state of the driver (S1102). .

[12-3. effect]
In this embodiment, erroneous estimation of the microsleep state can be avoided.

The erroneous estimation situation is not limited to the situations described above. is conversing with a fellow passenger. In such a situation, it is not necessary to estimate the micro-sleep state. Therefore, by not estimating the micro-sleep state, erroneous estimation of the micro-sleep state can be effectively avoided.

In addition, the image information from the image information acquisition unit 8 (the driver's image information indicating the face of the person) contains noise, processing for correcting the image information may be performed. This can prevent erroneous estimation of the microsleep state.

(Embodiment 13)
[13-1. Configuration of estimation device]
A configuration of an estimation device 2M according to Embodiment 13 will be described with reference to FIG. FIG. 25 is a block diagram showing the configuration of an estimation device 2M according to the thirteenth embodiment. In addition, in the present embodiment, the same reference numerals are given to the same constituent elements as in the above-described second embodiment, and the description thereof will be omitted.

As shown in FIG. 25, the estimation device 2M according to the thirteenth embodiment includes an image information acquisition unit 8, a biological information acquisition unit 10, an eye-closed state detection unit 12, a drowsiness determination unit 14, and a microsleep estimation unit 16A. A driving situation determination unit 64 is provided. The driving situation determination unit 64 has a driving situation detection unit 66 and an eye closure time change unit 68 . The biological information from the biological information acquiring section 10 is output to the drowsiness determining section 14 via the driving situation determining section 64 .

The driving condition detection unit 66 detects the driving condition of the vehicle by the driver based on the image information from the image information acquisition unit 8 or the biometric information from the biometric information acquisition unit 10 . Here, the driving situation is, for example, the driving time zone of the vehicle, the driving place of the vehicle, or the work pattern of the driver on the previous day. The driving situation detection unit 66 outputs the detection result to the closed-eye state detection unit 12 .

The eye-closed time changer 68 changes the first time and/or the second time used to detect the eye-closed time in the eye-closed state detector 12 based on the driving situation detected by the driving situation detector 66.

[13-2. Operation of estimation device]
Next, a first operation of the estimation device 2M according to Embodiment 13 will be described with reference to FIG. FIG. 26 is a flow chart showing the flow of the first operation of the estimating device 2M according to the thirteenth embodiment. In addition, in the flowchart of FIG. 26, the same step numbers are assigned to the same processes as those of the flowchart of FIG. 4, and the description thereof will be omitted.

As shown in FIG. 26, first, similarly to the second embodiment, after step S101 is executed, the driving situation detection unit 66 detects the driving situation. If the driving condition detection unit 66 does not detect any driving condition that affects the micro-sleep state (NO in S1201), the eye closing time change unit 68 sets the first time and the second time to the default value of 0.5, respectively. Seconds and 3 seconds are set (S1202). Driving conditions that affect the micro-sleep state are, for example, conditions in which the driver is driving at night or working the night shift the previous day. Thereafter, as in the second embodiment, the process proceeds to step S102, and steps S102 to S107 are executed in the same manner as in the second embodiment.

Returning to step S1201, if the driving condition detection unit 66 detects a driving condition that affects the microsleep state (YES in S1201), the eye closing time changing unit 68 changes the second time from the default value of 3 seconds to 2 seconds. Seconds are shortened (S1203). After that, when the eye-closed state detection unit 12 detects that the eye-closed time is less than 0.5 seconds or longer than 2 seconds (NO in S1204), the process proceeds to step S103. On the other hand, if the eye-closed state detection unit 12 detects that the eye-closed time is 0.5 seconds or more and less than 2 seconds (YES in S1204), the process proceeds to step S104.

As described above, in the first operation, for example, the driver is driving at night, or the driver was on the night shift the day before, and the driver is likely to feel drowsy. First, by shortening the closed-eye time detected by the closed-eye state detection unit 12, the micro-sleep state of the driver is estimated rather harshly.

Next, the second operation of the estimation device 2M according to Embodiment 13 will be described with reference to FIG. FIG. 27 is a flow chart showing the flow of the second operation of the estimating device 2M according to the thirteenth embodiment. In the flowchart of FIG. 27, the same step numbers are assigned to the same processes as those of the above-described flowchart of FIG. 4, and the description thereof will be omitted.

As shown in FIG. 27, first, as in the second embodiment, after step S101 is executed, the driving situation detection unit 66 detects the driving situation. If the driving condition detection unit 66 detects a driving condition that affects the microsleep state (YES in S1301), the eye closing time changing unit 68 sets the first time and the second time to the default value of 0.00. 5 seconds and 3 seconds are set (S1302). Thereafter, as in the second embodiment, the process proceeds to step S102, and steps S102 to S107 are executed in the same manner as in the second embodiment.

Returning to step S1301, if the driving condition detection unit 66 does not detect any driving condition that affects the microsleep state (NO in S1301), the eye closing time change unit 68 changes the first time from the default value of 0.5 seconds. It is lengthened to 1 second, and the second time is lengthened from the default value of 3 seconds to 4 seconds (S1303). After that, when the eye-closed state detection unit 12 detects that the eye-closed time is less than 1 second or 4 seconds or longer (NO in S1304), the process proceeds to step S103. On the other hand, if the eye-closed state detection unit 12 detects that the eye-closed time is 1 second or more and less than 4 seconds (YES in S1304), the process proceeds to step S104.

As described above, in the second operation, for example, when the driver is in a situation where it is difficult to feel drowsy, such as immediately after a nap or after ingesting caffeine. First, the driver's micro-sleep state is loosely estimated by lengthening the closed-eye time detected by the closed-eye state detection unit 12 .

[13-3. effect]
In this embodiment, erroneous estimation of the microsleep state can be avoided.

(Other modifications)
Although the estimation apparatus according to one or more aspects has been described based on the above embodiments, the present disclosure is not limited to the above embodiments. As long as it does not deviate from the spirit of the present disclosure, various modifications that can be conceived by those skilled in the art may be applied to the above-described embodiments, and a configuration constructed by combining the components of different embodiments may also be one or more aspects. may be included within the scope.

In each of the above embodiments, the reliability calculation unit 20 (20B, 20C, 20D, 20E, 20F, 20G, 20H, 20J, 20K, 24, 28, 36, 48) calculates the reliability. etc. may be used to calculate the reliability.

It should be noted that in each of the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor.

Also, some or all of the functions of the estimation device according to each of the above embodiments may be implemented by a processor such as a CPU executing a program.

A part or all of the components that make up each device described above may be configured from an IC card or a single module that can be attached to and removed from each device. The IC card or module is a computer system composed of a microprocessor, ROM, RAM and the like. The IC card or the module may include the super multifunctional LSI. The IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may have tamper resistance.

The present disclosure may be the method shown above. Moreover, it may be a computer program for realizing these methods by a computer, or it may be a digital signal composed of the computer program. In addition, the present disclosure provides a computer-readable non-temporary recording medium for the computer program or the digital signal, such as a flexible disk, hard disk, CD-ROM, MO, DVD, DVD-ROM, DVD-RAM, BD (Blu -ray (registered trademark) Disc), semiconductor memory or the like. Alternatively, the digital signal recorded on these recording media may be used. Further, according to the present disclosure, the computer program or the digital signal may be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, data broadcasting, or the like. The present disclosure may also be a computer system comprising a microprocessor and memory, the memory storing the computer program, and the microprocessor operating according to the computer program. Also, by recording the program or the digital signal on the recording medium and transferring it, or by transferring the program or the digital signal via the network or the like, it is implemented by another independent computer system It is good as

The present disclosure is applicable to, for example, an estimation device for estimating a micro-sleep state of a vehicle driver.

2, 2A, 2B, 2C, 2D, 2E, 2F, 2G, 2H, 2J, 2K, 2L, 2M estimation device 3 sensor group 4 imaging unit 5 vehicle state sensor 6 biological sensor 7 sensor information acquisition unit 8 image information acquisition unit 10 biological

information acquisition units

12, 12B, 12E closed-eye

state detection units

14, 14B

drowsiness determination units

16, 16A, 16B, 16C, 16D, 16E, 16F, 16G, 16H, 16J, 16K, 16L microsleep estimation unit 18

estimation unit

20, 20B, 20C, 20D, 20E, 20F, 20G, 20H, 20J, 20K, 24, 28, 36, 48 Reliability calculation unit 22 Closed eye time detection unit 26 Sleepiness

level determination units

30, 38 Open/close state detection unit 32 Open/close Speed calculation unit 34 Opening/closing speed determination unit 40 Eye closing speed calculation unit 42 Eye closing speed determination unit 44 Eye opening speed calculation unit 46 Eye opening speed determination unit 50 Binocular detection unit 52 Blink number detection unit 54 Life log information acquisition unit 56 Facial feature information acquisition unit 58 head

movement detection units

60 and 62 erroneous estimation situation detection unit 64 driving situation determination unit 66 driving situation detection unit 68 eye closure time change unit

Claims

An estimation device for estimating that a vehicle driver is in a micro-sleep state,
an eye-closed state detection unit that detects an eye-closed time of the driver;
a drowsiness determination unit that determines the drowsiness level of the driver;
The eye-closed state detection unit detects that the eye-closed time is greater than or equal to a first time period and less than a second time period, and the drowsiness determination unit determines that the drowsiness level is greater than or equal to a first threshold. a micro-sleep estimator that estimates that the driver is in a micro-sleep state on condition that:
The estimating device according to claim 1, wherein the first time is 0.5 seconds and the second time is 3 seconds.
The estimating device according to claim 1, wherein the microsleep estimator further calculates a first reliability that is an index indicating the probability of the estimation result that the driver is in the microsleep state.
The eye-closed state detection unit further calculates a second reliability that is an index indicating the likelihood of the detection result that the eye-closed time is greater than or equal to the first time and less than the second time,
The drowsiness determination unit further calculates a third reliability that is an index indicating the likelihood of the determination result that the drowsiness level is equal to or greater than the first threshold,
The estimation device according to claim 3, wherein the microsleep estimator calculates the first reliability based on the second reliability and the third reliability.
The estimating device further comprises an open/closed state detection unit that detects an eye opening/closing speed of the driver,
The micro-sleep estimating unit detects that the eye-closed time is longer than or equal to the first time and less than the second time by the eye-closed state detecting unit, and the drowsiness determining unit determines that the drowsiness level is equal to or greater than the first time. and the opening/closing state detection unit detects that the opening/closing speed is less than the second threshold, presuming that the driver is in the micro sleep state The estimating device according to claim 3.
The eye-closed state detection unit further calculates a second reliability that is an index indicating the likelihood of the detection result that the eye-closed time is greater than or equal to the first time and less than the second time,
The drowsiness determination unit further calculates a third reliability that is an index indicating the likelihood of the determination result that the drowsiness level is equal to or greater than the first threshold,
The open/closed state detection unit further calculates a fourth reliability, which is an index indicating the likelihood of the detection result that the open/close speed is less than the second threshold,
The estimation device according to claim 5, wherein the microsleep estimator calculates the first reliability based on the second reliability, the third reliability, and the fourth reliability.
The estimation device according to claim 5, wherein the drowsiness determination unit includes a function as the open/closed state detection unit.
The estimation device according to claim 7, wherein the drowsiness determination unit further includes a function as the closed eye state detection unit.
The estimating device further comprises an open/closed state detection unit that detects the eye closing speed and the eye opening speed of the driver,
The micro-sleep estimating unit detects that the eye-closed time is longer than or equal to the first time and less than the second time by the eye-closed state detecting unit, and the drowsiness determining unit determines that the drowsiness level is equal to or greater than the first time. and the open/closed state detection unit detects that the eye-closing speed is less than a third threshold, and the open/closed state detection unit detects that the eye-opening speed is less than a fourth threshold 4. The estimating device according to claim 3, wherein the driver is estimated to be in a micro-sleep state on the condition that it is detected that .
The eye-closed state detection unit further calculates a second reliability that is an index indicating the likelihood of the detection result that the eye-closed time is greater than or equal to the first time and less than the second time,
The drowsiness determination unit further calculates a third reliability that is an index indicating the likelihood of the determination result that the drowsiness level is equal to or greater than the first threshold,
The open/closed state detection unit further includes a fifth reliability, which is an index indicating the likelihood of the detection result that the eye closing speed is less than the third threshold, and the eye opening speed is less than the fourth threshold. Calculate the sixth reliability, which is an index indicating the likelihood of the detection result that it is less than
10. The microsleep estimator calculates the first reliability based on the second reliability, the third reliability, the fifth reliability, and the sixth reliability. The estimating device described in .
The estimating device according to claim 3, wherein the microsleep estimating unit calculates the first reliability according to the drowsiness level determined by the drowsiness determining unit.
The estimating device further comprises a binocular detection unit that detects the binoculars of the driver,
The micro-sleep estimating unit detects that the eye-closed time is longer than or equal to the first time and less than the second time by the eye-closed state detecting unit, and the drowsiness determining unit determines that the drowsiness level is equal to or greater than the first time. The driver is estimated to be in a micro-sleep state on the condition that it is determined to be equal to or greater than the threshold of and the both eyes of the driver are detected by the binocular detection unit. estimation device.
The estimating device further comprises a blink number detection unit that detects the number of blinks of the driver,
The micro-sleep estimating unit detects that the eye-closed time is longer than or equal to the first time and less than the second time by the eye-closed state detecting unit, and the drowsiness determining unit determines that the drowsiness level is equal to or greater than the first time. and that the number of blinks of the driver per unit time has increased or decreased by a fifth threshold or more, and the driver is in a micro-sleep state.
The estimating device further comprises a life log information acquisition unit that acquires life log information relating to the life of the driver,
The micro-sleep estimating unit detects that the eye-closed time is longer than or equal to the first time and less than the second time by the eye-closed state detecting unit, and the drowsiness determining unit determines that the drowsiness level is equal to or greater than the first time. and that the life log information that affects the micro sleep state of the driver is acquired by the life log information acquiring unit, the driver is in the micro sleep state. The estimating device according to claim 1, which is estimated as .
The estimating device further comprises a facial feature information acquisition unit that acquires facial feature information indicating facial features of the driver,
The micro-sleep estimating unit detects that the eye-closed time is longer than or equal to the first time and less than the second time by the eye-closed state detecting unit, and the drowsiness determining unit determines that the drowsiness level is equal to or greater than the first time. and that the facial feature information that affects the driver's micro-sleep state is acquired by the facial feature information acquiring unit, and the driver is in the micro-sleep state. The estimating device according to claim 1, which is estimated as .
The estimating device further includes a head motion detection unit that detects a motion of the driver's head,
The micro-sleep estimating unit detects that the eye-closed time is longer than or equal to the first time and less than the second time by the eye-closed state detecting unit, and the drowsiness determining unit determines that the drowsiness level is equal to or greater than the first time. is determined to be less than the threshold of, and the head movement detection unit detects the head movement that affects the micro sleep state of the driver, on the condition that the driver is in the micro sleep state The estimating device according to claim 1, which estimates that there is.
The estimating device further includes an erroneous estimation situation detection unit that detects a situation that affects the estimation of the driver's microsleep state by the microsleep estimating unit,
The estimation device according to claim 3, wherein the microsleep estimator changes the first reliability in consideration of the detection result of the erroneous estimation situation detector.
The estimating device further comprises an erroneous estimation situation detection unit that detects an erroneous estimation situation, which is a situation that affects the estimation of the driver's microsleep state by the microsleep estimating unit,
The estimation device according to claim 1, wherein the micro-sleep estimating unit does not estimate the driver's micro-sleep state when the erroneous estimation condition detection unit detects the erroneous estimation condition.
The estimation device further
a driving situation detection unit that detects a driving situation of the vehicle by the driver;
An eye-closing time changing unit that changes the first time and/or the second time used for detecting the eye-closing time in the eye-closing state detecting unit based on the driving situation detected by the driving situation detecting unit. The estimating device according to claim 1, comprising:
The eye-closed state detection unit detects the eye-closed time based on the image information of the driver captured by the imaging unit,
The estimation device according to any one of claims 1 to 19, wherein the drowsiness determination unit determines the drowsiness level based on the driver's biological information detected by a biological sensor.
The eye-closed state detection unit detects the eye-closed time based on the image information of the driver captured by the imaging unit,
The estimation device according to any one of claims 1 to 19, wherein the drowsiness determination unit determines the drowsiness level based on the image information.
An estimation method for estimating that a vehicle driver is in a micro-sleep state, comprising:
(a) detecting the closing time of the driver's eyes;
(b) determining a drowsiness level of the driver;
(c) detecting that the closed-eye time is longer than or equal to the first time and shorter than the second time in (a), and determining that the drowsiness level is greater than or equal to the first threshold in (b); and estimating that the driver is in a micro-sleep state if the driver is in a micro-sleep state.
A program that causes a computer to execute the estimation method according to claim 22.