JP7231035B2 - Sleepiness estimation information correction device, sleepiness estimation device, sleepiness estimation information correction method, sleepiness estimation method and program - Google Patents

Description

The present invention relates to a drowsiness estimation information correction device, a drowsiness estimation device, a drowsiness estimation information correction method, a drowsiness estimation method, and a program .

In relation to estimation of a person's (subject's) state, for example, Patent Document 1 describes a mood estimation system for reducing the computational load associated with mood estimation and improving the accuracy of mood estimation. . This mood estimation system simply estimates the subject's mood as a first mood value based only on the heartbeat information. The mood estimation system then estimates a second mood value based on heart rate information and other information only when the first mood value deviates from a predetermined set range.

Japanese Patent Application Laid-Open No. 2018-88966

One method of estimating a subject's condition is estimating the drowsiness of the subject. It is preferable to be able to reduce the decrease in estimation accuracy when the subject is drowsy.

An example of an object of the present invention is to provide a drowsiness estimation information correction device, a drowsiness estimation device, a drowsiness estimation information correction method, a drowsiness estimation method, and a program that can solve the above problems.

According to the first aspect of the present invention, the drowsiness estimation information correcting apparatus obtains, from time-series data obtained by repeatedly measuring the degree of eye opening of a subject at a predetermined frequency, the percentage of closed eyes within each predetermined time period. Drowsiness estimation information acquiring means for acquiring data obtained by obtaining Drowsiness estimation information as drowsiness estimation information ; and Drowsiness estimation information calculating post-correction drowsiness estimation information obtained by correcting the drowsiness estimation information with humidity information indicating the humidity in the surrounding environment of the subject. and a correction means.

According to the second aspect of the present invention, the drowsiness estimation device includes eyelid movement information acquisition means for acquiring eyelid movement information of a subject, the eyelid movement information, and humidity in the surrounding environment of the subject. Based on the humidity information, as the drowsiness estimation information of the subject, from the time-series data obtained by repeatedly measuring the degree of eye opening of the subject at a predetermined frequency, the rate of eye closure within the time period is calculated for each predetermined time period. a drowsiness estimation means for calculating the obtained data .

According to the third aspect of the present invention, a method for correcting drowsiness estimation information is performed by a computer, from time-series data obtained by repeatedly measuring the degree of eye opening of a subject at a predetermined frequency, for each predetermined time period, within that time period. data obtained by calculating the closed-eye ratio of the subject as drowsiness estimation information , and calculating post-correction drowsiness estimation information by correcting the drowsiness estimation information with humidity information indicating the humidity in the surrounding environment of the subject.

According to a fourth aspect of the present invention, a method for estimating drowsiness comprises acquiring eyelid movement information of a subject by a computer, and generating the eyelid movement information and humidity information indicating the humidity in the surrounding environment of the subject. Based on, as the drowsiness estimation information of the subject, from the time-series data obtained by repeatedly measuring the degree of eye opening of the subject at a predetermined frequency, data obtained by calculating the percentage of eye closure within the time period for each predetermined time period including calculating

According to the fifth aspect of the present invention, the program causes the computer to calculate, for each predetermined time period, the percentage of closed eyes within that time period from the time-series data obtained by repeatedly measuring the degree of eye opening of the subject at a predetermined frequency. A program for acquiring the obtained data as drowsiness estimation information , and calculating post-correction drowsiness estimation information by correcting the drowsiness estimation information with humidity information indicating the humidity in the surrounding environment of the subject. .

According to a sixth aspect of the present invention, a program acquires eyelid movement information of a subject in a computer, based on the eyelid movement information and humidity information indicating the humidity in the surrounding environment of the subject. Then, as the drowsiness estimation information of the subject, from the time-series data obtained by repeatedly measuring the degree of eye opening of the subject at a predetermined frequency, data obtained by obtaining the percentage of closed eyes within each predetermined time period is calculated. It is a program that makes you do things.

According to the embodiment of the present invention, when estimating the drowsiness of the subject, it is possible to reduce the decrease in estimation accuracy.

1 is a schematic block diagram showing an example of a functional configuration of an awakening level control system according to a first embodiment; FIG. FIG. 4 is a diagram showing an example of data input/output in the drowsiness estimation device and the drowsiness estimation information correction device according to the first embodiment; 1 is a schematic block diagram showing an example of a functional configuration of a drowsiness estimation information correcting device according to a first embodiment; FIG. FIG. 7 is a diagram showing an example of correction when the drowsiness estimation information correction unit according to the first embodiment outputs corrected drowsiness estimation information of continuous values. FIG. 7 is a diagram showing an example of correction when the drowsiness estimation information correction unit according to the first embodiment outputs corrected drowsiness estimation information of discrete values. It is a figure which shows the example of the operation control system which concerns on 1st Embodiment. It is a figure showing an example of an alarm system concerning a 1st embodiment. It is a figure which shows the example of the drowsiness estimation apparatus which concerns on 2nd Embodiment. FIG. 10 is a diagram showing an example of data input/output in the drowsiness estimation device according to the second embodiment; FIG. 10 is a functional block diagram showing an example of the functional configuration of a drowsiness estimation device according to a second embodiment; It is a figure which shows the example of the operation control system which concerns on 2nd Embodiment. It is a figure which shows the example of the alarm system which concerns on 2nd Embodiment. It is a figure which shows the example of a structure of the awakening degree control apparatus which concerns on embodiment. It is a figure which shows the example of a structure of the wakefulness characteristic display apparatus which concerns on embodiment. It is a figure which shows the example of the processing procedure in the wakefulness control method which concerns on embodiment. It is a figure which shows the example of the processing procedure in the wakefulness characteristic display method which concerns on embodiment. 1 is a schematic block diagram showing a configuration of a computer according to at least one embodiment; FIG.

Embodiments of the present invention will be described below, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution of the invention.

<First Embodiment>
FIG. 1 is a schematic block diagram showing an example of the functional configuration of the wakefulness control system according to the first embodiment. With the configuration shown in FIG. 1, the wakefulness control system 1 includes a camera 11, a humidity sensor 12, a sleepiness estimation device 13, a sleepiness estimation information correction device 14, an alertness control device 15, and an environment control device 16. Prepare.

The wakefulness control system 1 estimates the drowsiness of a person (subject) and controls the subject's wakefulness based on the estimation result. The target person here is a person who is the subject of drowsiness estimation. In addition, the degree of wakefulness referred to here is a degree indicating weakness of drowsiness. That is, a high degree of wakefulness means weak sleepiness (small sleepiness), and a low wakefulness degree means strong sleepiness (great sleepiness).

In particular, the wakefulness control system 1 calculates post-correction sleepiness estimation information, which is sleepiness estimation information reflecting the humidity in the surrounding environment of the subject. Then, the awakening level control system 1 controls the subject's awakening level based on the obtained post-correction sleepiness estimation information. The target person's surrounding environment here is the environment of the space in which the target person is located. The environment referred to here is the state of space, and the environment referred to here includes humidity.

The drowsiness estimation information here is information indicating the degree of drowsiness estimated in the drowsiness estimation. A magnitude relationship is defined for values that the drowsiness estimation information can take. The degree of drowsiness is indicated by the magnitude of the drowsiness estimation information value.
For example, the value of the drowsiness estimation information may indicate one of the classes for classifying the degree of drowsiness. When the value of the drowsiness estimation information indicates the class of degree of drowsiness in this way, it corresponds to an example in which the drowsiness estimation information takes discrete values.
Alternatively, the drowsiness estimation information may indicate the degree of drowsiness numerically. As an example of the case where the drowsiness estimation information takes a continuous value, there is a case where the drowsiness estimation information takes a real value in a predetermined interval such as [0, 1]. [0,1] indicates a value between 0 and 1.

For example, if the subject is a driver or operator of a vehicle such as an automobile, railway vehicle, ship or aircraft, the subject's surrounding environment may It can be the environment. Alternatively, the subject's ambient environment may be the environment of some space within the vehicle, such as the environment of the driver's seat or cockpit.
Further, when the target person is a worker who works indoors, the surrounding environment of the target person may be the indoor environment in which the target person is in the room. Alternatively, the surrounding environment of the subject may be the environment of a part of the room, such as the environment of the seat of the subject.

The camera 11 captures an image including an image of the subject's eyes.
For example, when the target person is a driver or operator of a vehicle such as an automobile, a railroad vehicle, a ship, or an aircraft, the camera 11 is installed in front of the driver's seat or the cockpit facing the driver's seat or the cockpit, The face of the person or the operator may be imaged.

Alternatively, when the subject is a worker who works while looking at a display (display device) indoors, the camera 11 is provided on the display used by the subject and captures the face of the subject positioned in front of the display. You may make it image.
The camera 11 may capture moving images. Alternatively, the camera 11 may take still images periodically, for example, every 5 seconds.

The camera 11 may output image data representing the captured image. In this case, the image data output by the camera 11 corresponds to an example of movement information of the subject's eyelids. The eyelid movement information is information indicating eyelid movement. Eyelid movement information is also referred to as eyelid movement information or simply movement information.

Alternatively, the camera 11 may have an image processing function, extract the feature amount of the eyelid movement from the captured image, and output time-series information of the extracted feature amount. In this case, the feature amount output by the camera 11 corresponds to an example of movement information of the subject's eyelids. Information indicating the extent to which the eyes are open for a certain period of time, information indicating the number of blinks, or information indicating the length of time the eyes are closed, or Various information can be used, such as combinations of these. Thus, the feature amount extracted by the camera 11 is not limited to a specific one. Also, the process of extracting motion information may be implemented using a device different from the camera 11 .
For example, the camera 11 may extract a feature quantity, which will be described later in explaining the processing of the drowsiness estimation device 13 . That is, either the camera 11 or the drowsiness estimation device 13 may extract the feature amount.

The drowsiness estimation device 13 estimates the drowsiness of the subject based on the eyelid movement information of the subject output by the camera 11 . Specifically, the drowsiness estimation device 13 calculates and outputs drowsiness estimation information of the subject. As described above, the drowsiness estimation information is information indicating the degree of drowsiness by the magnitude of the value. It may be defined such that the value of the drowsiness estimation information increases as the drowsiness increases. Alternatively, it may be defined such that the value of the drowsiness estimation information decreases as the drowsiness increases.

An example of processing performed by the drowsiness estimation device 13 will be described later. However, the drowsiness estimation device 13 is not limited to a specific device as long as it calculates the subject's drowsiness estimation information based on the movement of the subject's eyelids. An existing drowsiness estimation device may be used as the drowsiness estimation device 13 .
The drowsiness estimation device 13 corresponds to an example of a drowsiness estimation information acquisition unit (drowsiness estimation information acquisition means).

The humidity sensor 12 is a sensor that measures humidity and outputs humidity information. The humidity information here is information indicating humidity (information indicating the level of humidity).
The humidity sensor 12 may output the most recent humidity measurement itself as the humidity information. Alternatively, the humidity sensor 12 may output, as the humidity measurement value, a value obtained by processing the humidity measurement value, such as outputting a value obtained by averaging the humidity measurement values over a predetermined time window _TH as the humidity information. .

The humidity information output by the humidity sensor 12 is used as information indicating the humidity in the surrounding environment of the subject.
The installation position of the humidity sensor 12 is not limited to a specific position as long as it is a position where the humidity in the space where the subject is located can be measured. It is more preferable if the installation position of the humidity sensor 12 is close to the subject's eyes, but the embodiment is not limited to such an example.

For example, if the target person is a driver or operator of a vehicle such as an automobile, a railway vehicle, a ship, or an aircraft, the humidity sensor 12 is installed in the vehicle, such as an automobile, a railway vehicle, a ship, or an aircraft. may be In particular, the humidity sensor 12 may be installed in the driver's or pilot's seat.
Moreover, when the subject is a worker who works indoors, the humidity sensor 12 may be installed in the room where the subject is present. In particular, the humidity sensor 12 may be installed at the subject's seat.

A humidity sensor 12 may be placed at eye level of the subject. Alternatively, the humidity sensor 12 may be placed at the subject's chest level.
The humidity sensor 12 may measure the relative humidity [%]. Alternatively, the humidity sensor 12 may measure absolute humidity [g/m ³ ].

The humidity sensor 12 periodically measures humidity and outputs humidity information. The time interval at which humidity sensor 12 measures humidity is not limited to a specific one. For example, the humidity sensor 12 may measure the humidity at regular intervals of several minutes to several tens of minutes and output humidity information each time the measurement is made.

The drowsiness estimation information correction device 14 corrects the drowsiness estimation information calculated by the drowsiness estimation device 13 so as to reflect the humidity in the surrounding environment of the subject.
Here, as a result of conducting an experiment to estimate drowsiness, it was found that there is a relationship between the movement of a person's eyelids and the humidity in the surrounding environment of the person. Based on this relationship, it has been found that the accuracy of drowsiness estimation decreases depending on the state of the surrounding environment.

In particular, when the humidity in the surrounding environment is low, eyelids move more than usual due to the dryness of the eyes, resulting in a more intense drowsiness than is actually the case.
Therefore, the drowsiness estimation information correction device 14 corrects the drowsiness estimation information calculated by the drowsiness estimation device 13 so as to reflect the humidity in the surrounding environment of the subject. As a result, in the wakefulness control system 1, when calculating the subject's drowsiness estimation information, it is possible to reduce deterioration in estimation accuracy due to the influence of the surrounding environment. Specifically, in the wakefulness control system 1, it is possible to reduce the decrease in accuracy of drowsiness estimation due to the influence of the humidity in the surrounding environment of the subject.

It should be noted that increased eyelid movement means an increase in the number of eyelid movements, an increase in the extent to which the eyelids open (i.e., opening the eyes more), an increase in the speed of eyelid movement, and an increase in eyelid movement per blink. or a combination thereof.

FIG. 2 is a diagram showing an example of data input/output in the drowsiness estimation device 13 and the drowsiness estimation information correction device 14. As shown in FIG.
In the example of FIG. 2, the drowsiness estimation device 13 acquires eyelid movement information. As described above, the drowsiness estimation device 13 may acquire image data including an eyelid image as the eyelid movement information.
Alternatively, the drowsiness estimation device 13 may acquire the feature amount of the eyelid movement as the eyelid movement information.

The drowsiness estimation device 13 applies the acquired eyelid movement information to the drowsiness estimation model to calculate drowsiness estimation information. The drowsiness estimation model is a model that outputs drowsiness estimation information in response to input of eyelid movement information.
The expression format of the drowsiness estimation model provided in the drowsiness estimation device 13 is not limited to a specific format.
For example, the drowsiness estimation device 13 may be provided with a drowsiness estimation model expressed in the form of a formula, but the embodiment is not limited to such an example.

The sleepiness estimation information correcting device 14 acquires the sleepiness estimation information calculated by the sleepiness estimation device 13 and the humidity information. The drowsiness estimation information correcting device 14 may acquire the measured humidity value itself as the humidity information. Alternatively, the drowsiness estimation information correcting device 14 may acquire a value obtained by processing the humidity measurement value as the humidity information.
The drowsiness estimation information correcting device 14 applies the acquired drowsiness estimation information and humidity information to a correction model to correct the drowsiness estimation information by adding the influence of humidity, and outputs the drowsiness estimation information after correction. do. The correction model is a model that outputs corrected drowsiness estimation information in response to input of drowsiness estimation information and humidity information.

The representation format of the correction model provided in the drowsiness estimation information correction device 14 is not limited to a specific format. For example, the drowsiness estimation information correcting device 14 may be provided with a correction model expressed in the form of a formula, but the embodiment is not limited to such an example.
The corrected drowsiness estimation information output by the drowsiness estimation information correction device 14 is referred to as post-correction drowsiness estimation information.

FIG. 3 is a schematic block diagram showing an example of the functional configuration of the drowsiness estimation information correction device 14. As shown in FIG. With the configuration shown in FIG. 3 , the drowsiness estimation information correction device 14 includes a first communication section 110 , a first storage section 180 and a first control section 190 . The first storage unit 180 has a correction model storage unit 181 . The first control unit 190 includes a drowsiness estimation information correction unit 191 .
The first communication unit 110 communicates with other devices. In particular, the first communication unit 110 receives drowsiness estimation information from the drowsiness estimation device 13 . Also, the first communication unit 110 receives humidity information from the humidity sensor 12 . The first communication unit 110 also transmits the post-correction drowsiness estimation information to the wakefulness control device 15 .

The first storage unit 180 stores various information. The functions of the first storage unit 180 are executed using a storage device included in the awakening level control system 1 .
The correction model storage unit 181 stores correction models. As described above, the correction model is a model that outputs corrected drowsiness estimation information in response to inputs of drowsiness estimation information and humidity information.

The first control unit 190 controls each unit of the drowsiness estimation information correction device 14 to execute various processes. The functions of the first control unit 190 are executed by a CPU (Central Processing Unit) of the drowsiness estimation information correcting device 14 reading out a program from the first storage unit 180 and executing it.
The drowsiness estimation information correction unit 191 corrects the drowsiness estimation information so as to take into account the influence of humidity, and calculates post-correction drowsiness estimation information. Specifically, the drowsiness estimation information correction unit 191 reads the correction model from the correction model storage unit 181 . Then, the drowsiness estimation information correcting unit 191 applies the drowsiness estimation information from the drowsiness estimation device 13 and the humidity information from the humidity sensor 12 to the correction model to obtain post-correction drowsiness estimation information.

The post-correction sleepiness estimation information output by the sleepiness estimation information correction unit 191 may be a continuous value or a discrete value.
FIG. 4 is a diagram showing an example of correction when the drowsiness estimation information correction unit 191 outputs corrected drowsiness estimation information of continuous values. The horizontal axis of the graph in FIG. 4 indicates relative humidity. The vertical axis indicates correction coefficients.

FIG. 4 shows an example in which the drowsiness estimation information has a larger value as the drowsiness is stronger. Further, in the example of FIG. 4, the thresholds C11 and C12 of the correction coefficient values are set as "0<C11<C12<1".
The drowsiness estimation information correction unit 191 acquires a correction coefficient value associated with the humidity indicated by the humidity information in the graph of FIG. 4 . Then, the drowsiness estimation information correction unit 191 multiplies the drowsiness estimation information (the value before correction) by the acquired correction coefficient value to calculate post-correction drowsiness estimation information.
The graph in FIG. 4 corresponds to an example of the correction model portion. The combination of the graph in FIG. 4 and the formula for multiplying the drowsiness estimation information by the correction coefficient value corresponds to an example of the correction model.

Here, when the humidity is low, the target person repeatedly blinks or closes their eyes in response to dryness of the eyes, so that the drowsiness estimation information is greater than the actual drowsiness (for example, the drowsiness experienced by the target person). tend to be valuable.
Therefore, the drowsiness estimation information correction unit 191 calculates a smaller value of the correction coefficient as the humidity is lower, and makes the drowsiness estimation information smaller (it is estimated that the drowsiness is weaker). The amount of correction by the drowsiness estimation information correction unit 191 (the amount of decrease from the original drowsiness estimation information) increases as the humidity decreases.

On the other hand, in the graph of FIG. 4, when the humidity is H11 or less, the correction coefficient is a constant value (C11). This is because when the humidity drops below a certain level, even if the humidity drops further, the influence of the humidity on the movement of the eyelids is considered to be the same.
Further, in the graph of FIG. 4, when the humidity is H12 or higher, the correction coefficient is a constant value (C12). This is because when the humidity rises above a certain level, even if the humidity increases further, the influence of the humidity on the movement of the eyelids is considered to be the same.

However, the correction method shown in FIG. 4 is an example, and the embodiment is not limited to such an example. For example, although FIG. 4 shows an example in which the drowsiness estimation information correction unit 191 performs correction using a piecewise linear function, correction may be performed using a nonlinear function. Here, E is the drowsiness estimation information before correction, E′ is the drowsiness estimation information after correction, H is humidity, f is a nonlinear function, and the correction of the drowsiness estimation information by the drowsiness estimation information correction unit 191 is “E′ =f(E,H)”. Various non-linear functions f, in which E' becomes smaller than E and E-E' becomes larger as H becomes smaller, can be used for correcting the drowsiness estimation information correction unit 191. FIG.

Also in this case, as in the example of FIG. 4, the value of EE' may not change (the correction amount may not change) when the humidity is equal to or lower than the lower limit threshold. In the example of FIG. 4, H11 corresponds to an example of the lower limit threshold.
Further, as in the example of FIG. 4, the value of EE' may not change (the correction amount may not change) when the humidity is equal to or higher than the upper limit threshold. In the example of FIG. 4, H12 corresponds to an example of the upper limit threshold.

FIG. 5 is a diagram showing an example of correction when the drowsiness estimation information correction unit 191 outputs corrected drowsiness estimation information of discrete values. The data table shown in FIG. 5 corresponds to an example of a correction model.
FIG. 5 shows an example of a correspondence relationship between the humidity (relative humidity) indicated by the humidity information and the threshold used for classifying the sleepiness estimation information. In the example of FIG. 5, four thresholds TH ₁ ¹ , TH ₂ ¹ , TH ₃ ¹ , and TH ₄ ¹ are used when the relative humidity H is "0%≦H<40%". When the relative humidity H is "40%≦H<60%", four thresholds TH ₁ ² , TH ₂ ² , TH ₃ ² , and TH ₄ ² are used as thresholds. When the relative humidity H is "60%≦H≦100%", four thresholds TH ₁ ³ , TH ₂ ³ , TH ₃ ³ , and TH ₄ ³ are used as thresholds.
Here, i is an integer that satisfies "1 ≤ i ≤ 3" and j is an integer that satisfies "1 ≤ j ≤ 3", and the relationship between TH _i ^j and TH _i+1 ^j is as shown in equation (1). shown in

By applying this threshold to the drowsiness estimation information, the subject's drowsiness is estimated to be one of five levels.
Further, when the drowsiness estimation information is calculated to be larger as the drowsiness is stronger, i is an integer satisfying “1≦i≦3” and j is an integer satisfying “1≦j≦3”, and TH _i ^j and TH The relationship with _i ^j+1 is shown as Equation (2).

Thus, if the level of drowsiness indicated by the pre-correction drowsiness estimation information is the same, the lower the humidity, the weaker the drowsiness level indicated by the post-correction drowsiness estimation information.
Alternatively, if the sleepiness estimation information is calculated to be smaller as the sleepiness increases, the threshold is set so that the relationship of Equation (3) holds instead of Equation (2).

Thus, if the level of drowsiness indicated by the pre-correction drowsiness estimation information is the same, the lower the humidity, the weaker the drowsiness level indicated by the post-correction drowsiness estimation information.

The environment control device 16 is a device that changes physical quantities in the subject's surrounding environment. Here, the physical quantity in the subject's surrounding environment may be any physical quantity that affects the subject's drowsiness. For example, the environment control device 16 may be an air conditioner, which adjusts the temperature (air temperature) in the surrounding environment of the subject. Alternatively, the environment control device 16 may be a lighting device that adjusts the brightness of the subject's surrounding environment.

The arousal level control device 15 controls the arousal level of the subject by controlling the environment control device 16 . The arousal level here is a concept opposite to drowsiness, and the higher the arousal level, the weaker the drowsiness. For example, when the subject's arousal level is low (drowsiness is strong), the arousal level control device 15 may control the environment control device 16 to increase the subject's arousal level (drowsiness is weakened). .

FIG. 6 is a schematic block diagram showing an example of the functional configuration of the operation control system according to the first embodiment. With the configuration shown in FIG. 6 , the operation control system 2 includes a camera 11 , a humidity sensor 12 , a drowsiness estimation device 13 , a drowsiness estimation information correction device 14 , an operation control device 21 , and a driving device 22 .
6, the same reference numerals (11, 12, 13, 14) are given to the parts having the same functions as those in FIG. 1, and the description thereof is omitted.

The operation control system 2 controls the operation of a vehicle such as an automobile, railroad vehicle, ship, or aircraft. Specifically, the driving control system 2 estimates the drowsiness of the target person, who is the driver or operator of the vehicle to be controlled. When the driving control system 2 determines that the subject's drowsiness exceeds a predetermined condition, the driving control system 2 controls the vehicle to be controlled so as to cope with the case where the subject cannot drive or steer properly due to the drowsiness. . For example, the driving control system 2 may limit the speed of the controlled vehicle to shorten the braking distance, or switch the driving of the controlled vehicle to automatic driving. Control of the target vehicle is not limited to these.

The driving device 22 includes any one or a combination of a driving device of the vehicle to be controlled, such as an engine or a motor, a braking device (brake) of the vehicle to be controlled, or a steering device of the vehicle to be controlled, and is used for driving control. It becomes the control object of the device 21 .

Based on the post-correction drowsiness estimation information from the drowsiness estimation information correction device 14, the operation control device 21 determines whether or not the subject's drowsiness exceeds a predetermined condition. The operation control device 21 may compare the post-correction drowsiness estimation information with a predetermined threshold to determine whether or not the subject's drowsiness is greater than or equal to the threshold. is not limited to
When it is determined that the subject's drowsiness exceeds a predetermined condition, the operation control device 21 controls the driving device 22 so as to cope with the case where the subject cannot drive or operate appropriately due to drowsiness.

For example, the driving device 22 may include at least one of a driving device and/or a braking device, and the driving control device 21 may control the driving device 22 to limit the speed of the controlled vehicle. For example, the operating device 22 may control the operating device 22 so that the speed of the controlled vehicle is at or below a certain speed. Alternatively, the driving control device 21 may control the driving device 22 so as to decelerate the controlled vehicle.

Alternatively, the operating device 22 may include a braking device, and the operating control device 21 may control the operating device 22 to prefill the braking device. Prefilling the braking device here means shortening the play of the braking device to accelerate the start of the braking operation.
Alternatively, the driving device 22 may include a driving device, a braking device, and a steering device of the vehicle to be controlled, and the driving control device 21 may control the driving device 22 to automatically drive the vehicle to be controlled. .

The operation control device 21 may be configured as a brake system such as an antilock brake system, an automatic brake prefill system, a brake assist system, or a collision mitigation brake system. Alternatively, the operation control device 21 may be configured as a cruise control system. The cruise control system referred to here is a system that keeps the speed of a controlled vehicle constant. Alternatively, the operation control device 21 may be configured as an automatic operation system. Alternatively, the operation control device 21 may be configured as a system separate from these systems.

FIG. 7 is a schematic block diagram showing an example of the functional configuration of the alarm system according to the first embodiment; With the configuration shown in FIG. 7 , the alarm system 3 includes a camera 11 , a humidity sensor 12 , a drowsiness estimation device 13 , a drowsiness estimation information correction device 14 , and an alarm device 31 .
7, the same reference numerals (11, 12, 13, 14) are given to the parts having the same functions as those in FIG. 1, and the description thereof is omitted.

The alarm system 3 is a system that estimates the drowsiness of the subject and outputs an alarm when it is determined that the drowsiness exceeds a predetermined condition.
For example, the subject may be a driver or operator of a vehicle such as an automobile, rail vehicle, ship or aircraft. In this case, the alarm system 3 outputs an alarm to at least temporarily reduce the drowsiness of the subject (that is, wake him up), and reduce the possibility that the subject cannot drive appropriately due to drowsiness. .

Based on the corrected sleepiness estimation information from the sleepiness estimation information correcting device 14, the alarm device 31 determines whether or not the subject's sleepiness exceeds a predetermined condition. The alarm device 31 may compare the post-correction drowsiness estimation information with a predetermined threshold to determine whether or not the subject's drowsiness is greater than or equal to the threshold. Not limited.
When it is determined that the subject's drowsiness exceeds a predetermined condition, the alarm device 31 outputs an alarm.

A method for outputting an alarm by the alarm device 31 is not limited to a specific method. For example, the alarm device 31 may include an audio output device such as a speaker or buzzer to output an alarm message or alarm sound. Alternatively, the alarm device 31 may visually output the alarm, such as displaying an alarm message on a display or lighting a lamp. The alarm device 31 may output both an audible alarm and a visual alarm.
The alarm system 3 may output a message urging the vehicle to move to a safe location and sleep.

According to the configuration in which the drowsiness estimation information correction device 14 is provided as in the configuration example of FIG. Information combined with arousal during movement can be used as learning data.
In particular, according to the configuration in which the drowsiness estimation information correction device 14 is provided, the machine learning of the drowsiness estimation information calculation processing by the drowsiness estimation device 13 and the machine learning of the correction processing by the drowsiness estimation information correction device 14 are performed separately. It can be a machine learning process. Therefore, humidity information is unnecessary in the machine learning process of calculating drowsiness estimation information by the drowsiness estimation device 13 .

Here, there are no research reports on the relationship between subject's drowsiness and humidity in addition to subject's eyelid movement. Therefore, no data are found that show the relationship between subject's eyelid movement and humidity and subject's drowsiness.

According to the configuration in which the drowsiness estimation information correction device 14 is provided, machine learning for the processing of calculating drowsiness estimation information by the drowsiness estimation device 13 can use a model that has already been created, and there is no need to acquire new data. .
Alternatively, as the drowsiness estimation device 13, a device that calculates drowsiness estimation information based on the movement of the subject's eyelids can be used. That is, a model generally known as a drowsiness estimation model can be used. In this case, there is no need to newly perform machine learning for the process of calculating drowsiness estimation information by the drowsiness estimation device 13 .

As described above, the drowsiness estimation device 13 acquires drowsiness estimation information based on the movement of the subject's eyelids. The drowsiness estimation information correction unit 191 calculates post-correction drowsiness estimation information by correcting the drowsiness estimation information with humidity information indicating the humidity in the surrounding environment of the subject.
As a result, the drowsiness estimation information correcting device 14 can reduce the decrease in estimation accuracy when estimating the drowsiness of the subject. In particular, the drowsiness estimation information correcting device 14 can avoid or reduce the decrease in drowsiness estimation accuracy due to the influence of the humidity in the surrounding environment of the subject.

In addition, the drowsiness estimation information correction unit 191 performs correction so that the post-correction drowsiness estimation information becomes smaller as the humidity is lower.
As a result, in the drowsiness estimation information correcting device 14, the subject feels dry eyes and blinks. Alternatively, it is possible to reduce drowsiness being determined more strongly than it actually is due to an action such as closing one's eyes.

Further, when the humidity is greater than the upper limit threshold, the drowsiness estimation information correction unit 191 corrects the same amount of correction as when the humidity is the upper limit threshold.
As a result, the drowsiness estimation information correcting device 14 can more accurately estimate drowsiness by reflecting the fact that when the humidity is greater than the upper limit threshold, the influence on the movement of the eyelids is the same.

Further, when the humidity is smaller than the lower limit threshold, the drowsiness estimation information correction unit 191 corrects the same amount of correction as when the humidity is the lower limit threshold.
As a result, the drowsiness estimation information correcting device 14 can more accurately estimate drowsiness by reflecting the fact that when the humidity is greater than the lower limit threshold, the influence on eyelid movement is similar.

<Second embodiment>
Instead of the drowsiness estimating device 13 and the drowsiness estimation information correcting device 14, a drowsiness estimating device that outputs drowsiness estimation information that takes into account the influence of humidity may be provided.
FIG. 8 is a diagram showing an example of an awakening level control system according to the second embodiment. With the configuration shown in FIG. 8 , the wakefulness control system 4 includes a camera 11 , a humidity sensor 12 , a drowsiness estimation device 41 , a wakefulness control device 15 , and an environment control device 16 .
8, the same reference numerals (11, 12, 15, 16) are given to the parts having the same functions as those in FIG. 1, and the description thereof is omitted.

The wakefulness control system 4 of FIG. 8 is different from the wakefulness control system 1 of FIG.
Otherwise, the wakefulness control system 4 is the same as the wakefulness control system 1 . In particular, the wakefulness control system 4 calculates post-correction sleepiness estimation information that reflects the humidity in the surrounding environment of the subject, as in the case of the wakefulness control system 1, as the sleepiness estimation information. Then, similarly to the wakefulness control system 1, the wakefulness control system 4 controls the subject's wakefulness based on the calculated post-correction sleepiness estimation information.

The drowsiness estimation device 41 calculates post-correction drowsiness estimation information. Specifically, the drowsiness estimation device 41 calculates post-correction drowsiness estimation information using a drowsiness estimation model that outputs post-correction drowsiness estimation information in response to input of eyelid movement information and humidity information.
FIG. 9 is a diagram showing an example of data input/output in the drowsiness estimation device 41. As shown in FIG.
In the example of FIG. 9, the drowsiness estimation device 41 acquires eyelid movement information and humidity information. The drowsiness estimation device 41 applies the obtained eyelid movement information and humidity information to the drowsiness estimation model to calculate drowsiness estimation information. The drowsiness estimation device 41 calculates drowsiness estimation information that reflects the influence of humidity by using a model that reflects the influence of humidity.

FIG. 10 is a functional block diagram showing an example of the functional configuration of the drowsiness estimation device 41. As shown in FIG. With the configuration shown in FIG. 10 , the drowsiness estimation device 41 includes a second communication section 210 , a second storage section 280 and a second control section 290 . The second storage unit 280 includes a drowsiness estimation model storage unit 281 . The second control unit 290 has a drowsiness estimation unit 291 .

The second communication unit 210 communicates with other devices. In particular, the second communication unit 210 receives eyelid movement information from the camera 11 . The second communication unit 210 corresponds to an example of an eyelid movement information acquisition unit (eyelid movement information acquisition means). Second communication unit 210 also receives humidity information from humidity sensor 12 . The second communication unit 210 also transmits the post-correction drowsiness estimation information to the wakefulness control device 15 .

The second storage unit 280 stores various information. The function of the second storage unit 280 is executed using a storage device included in the awakening level control system 1 .
The drowsiness estimation model storage unit 281 stores a drowsiness estimation model that outputs post-correction drowsiness estimation information in response to inputs of eyelid movement information and humidity information. This drowsiness estimation model corresponds to a combination of the drowsiness estimation model included in the drowsiness estimation device 13 of the first embodiment and the correction model included in the drowsiness estimation information correction device 14 .

A model in which the drowsiness estimation model stored in the drowsiness estimation model storage unit 281 outputs post-correction drowsiness estimation information directly (without calculating pre-correction drowsiness estimation information) in response to inputs of eyelid movement information and humidity information. may be Alternatively, the drowsiness estimation model stored in the drowsiness estimation model storage unit 281 calculates drowsiness estimation information before correction based on the eyelid movement information, corrects the drowsiness estimation information based on the humidity information, and calculates the drowsiness estimation information after correction. It may be a model that calculates

The second control unit 290 controls each unit of the drowsiness estimation device 41 to perform various processes. The functions of the second control unit 290 are executed by the CPU of the drowsiness estimation device 41 reading a program from the second storage unit 280 and executing it.
The drowsiness estimation unit 291 calculates post-correction drowsiness estimation information based on the eyelid movement information and the humidity information. Specifically, the drowsiness estimation unit 291 reads the drowsiness estimation model from the drowsiness estimation model storage unit 281 . Then, the drowsiness estimation unit 291 applies the eyelid movement information from the camera 11 and the humidity information from the humidity sensor 12 to the drowsiness estimation model to obtain post-correction drowsiness estimation information.

The corrected sleepiness estimation information output by the sleepiness estimation unit 291 is the same as the corrected sleepiness estimation information output by the sleepiness estimation information correction unit 191 of the first embodiment. In particular, the drowsiness estimation unit 291 reduces the drowsiness estimation information for the same eyelid movement information as the humidity is lower (estimates that the drowsiness is weaker). The corrected drowsiness estimation information output by the drowsiness estimation unit 291 may be a continuous value or a discrete value, like the corrected drowsiness estimation information output by the drowsiness estimation information correction unit 191 of the first embodiment. good too.

FIG. 11 is a diagram showing an example of an operation control system according to the second embodiment. With the configuration shown in FIG. 11 , the driving control system 5 includes a camera 11 , a humidity sensor 12 , a drowsiness estimation device 41 , a driving control device 21 and a driving device 22 .
11, the same reference numerals (11, 12, 21, 22) are given to the parts having the same functions as those in FIG. 6, and the description thereof will be omitted.

The operation control system 5 of FIG. 11 differs from the operation control system 2 of FIG. 6 in that a drowsiness estimation device 41 is provided instead of the drowsiness estimation device 13 and the drowsiness estimation information correction device 14 . Operation control system 5 is otherwise similar to operation control system 2 .
Also, the drowsiness estimation device 41 in FIG. 11 is the same as the drowsiness estimation device 41 in FIG.
In the example of FIG. 11, the drowsiness estimation device 41 calculates post-correction drowsiness estimation information similar to the drowsiness estimation information correction device 14 of the first embodiment, and the operation control system 5 is similar to the operation control system 2 of FIG. function.

FIG. 12 is a diagram showing an example of an alarm system according to the second embodiment. With the configuration shown in FIG. 12 , the alarm system 6 includes a camera 11 , a humidity sensor 12 , a drowsiness estimation device 41 and an alarm device 31 .
12, the same reference numerals (11, 12, 31) are given to the parts having the same functions as those in FIG. 7, and the description thereof is omitted.

The alarm system 6 of FIG. 12 differs from the alarm system 3 of FIG. 7 in that a drowsiness estimation device 41 is provided instead of the drowsiness estimation device 13 and the drowsiness estimation information correction device 14 . Otherwise, alarm system 6 of FIG. 12 is similar to alarm system 3 of FIG.
12 is the same as the drowsiness estimation device 41 shown in FIG. 8, so the same reference numerals are given and description thereof is omitted.
In the example of FIG. 12, the drowsiness estimation device 41 calculates post-correction drowsiness estimation information similar to the drowsiness estimation information correction device 14 of the first embodiment, and the alarm system 6 functions similarly to the alarm system 3 of FIG. do.

As described above, the second communication unit 210 acquires the eyelid movement information of the subject. In addition, the drowsiness estimation unit 291 calculates drowsiness estimation information of the subject based on the eyelid movement information and the humidity information indicating the humidity in the surrounding environment of the subject.
As a result, the drowsiness estimation device 41 can reduce deterioration in estimation accuracy due to the influence of the surrounding environment when calculating the subject's drowsiness estimation information.

<Example of drowsiness estimation>
An example of calculation of drowsiness estimation information performed by the drowsiness estimation device 13 described in the first embodiment will be described below. The drowsiness estimation device 41 described in the second embodiment also applies the same processing as the processing described below as part of the processing for calculating post-correction drowsiness estimation information based on the eyelid movement information and the humidity information. be able to.

[First example]
The drowsiness estimation device 13 performs image processing on the moving image signal, detects the subject's degree of eye openness, and acquires time-series information of the subject's degree of eye openness. The degree of eye opening is information representing the extent to which the subject's eyes are open. The degree of eye openness may be, for example, the proportion of time the eyes are open in a unit time, or may be a value obtained by accumulating the area of the open portion of the eyes for each predetermined period. However, the degree of eye opening is not limited to the information shown in these examples.
The drowsiness estimation device 13 may detect the degree of openness of the left eye and the degree of openness of the right eye, and acquire time-series information of the degree of left eye openness and time-series information of the degree of right eye openness, respectively. . In this case, the drowsiness estimation device 13 may calculate time-series information of the subject's degree of eye openness by averaging the time-series information of the degree of eye openness of the left eye and the time-series information of the degree of eye openness of the right eye. good.

The drowsiness estimation device 13 obtains an average value by adding and averaging the time-series information X _L (t) on the degree of openness of the left eye and the time-series information X _R (t) on the degree of openness of the right eye. It is output as time-series information X(t) of the degree of eye opening of the subject. Therefore, time-series information X(t) of the degree of eye opening of the subject is expressed as in Equation (4).

The drowsiness estimation device 13 generates time-series information X _L (t) on the degree of openness of the left eye, time-series information X _R (t) on the degree of openness of the right eye, and time-series information X(t) on the degree of eye openness of the subject. A value normalized to a value in the range of 0 to 1 may be used as each of .

The drowsiness estimation device 13 performs filtering to remove changes due to blinking of the subject 20 from the time-series information X(t) of the degree of eye openness, and outputs the time-series information _XF (t) of the degree of eye openness after filtering. can be The time-series information X _F (t) of the degree of eye openness after this filtering is expressed, for example, by Equation (5).

Here, F[] is an operator indicating filtering processing. N indicates the number of frames for filtering.
The drowsiness estimation device 13 performs the filtering process in units of filtering calculation window width T _N [seconds]. The number of frames N, the filtering calculation window width T _N [seconds], and the frame rate fs [frames/second] have the relationship shown in Equation (6).

For example, when the number of frames N is 3 and the frame rate fs is equal to 30 [frames/second], the filtering calculation window width T _N [seconds] is "3/30=0.1 [seconds]".
In this example, the drowsiness estimation device 13 replaces the time-series information X(t) of the degree of eye openness with a predetermined value by filtering in units of the filtering calculation window width T _N [seconds], and filters the replaced value. It is output as time-series information X _F (t) of the degree of eye opening after that.

In addition to or instead of the time-series information X(t) of the degree of eye openness, the drowsiness estimation device 13 obtains the time-series information X _L (t) of the degree of openness of the left eye and the time-series information X _R of the degree of right eye openness. (t) may be used.
For example, the drowsiness estimation device 13 replaces the time-series information X(t) of the degree of eye openness with a predetermined main value by filtering in units of filtering calculation window width T _N [seconds], and replaces the replaced value with is output as time-series information X _F (t) of the degree of eye opening.

In addition, the drowsiness estimation device 13 replaces the time-series information X _L (t) of the degree of openness of the left eye with a predetermined first subvalue by filtering in units of the filtering calculation window width T _N [seconds], The replaced value is output as time-series information X _FL (t) of the degree of openness of the left eye after filtering.
Similarly, the drowsiness estimation device 13 replaces the time-series information X _R (t) of the degree of openness of the right eye with a predetermined second value by filtering in units of filtering calculation window width T _N [seconds], The replaced value is output as time-series information X _FR (t) of the degree of openness of the right eye after filtering.
The drowsiness estimation device 13 may perform only one of the above three filterings. Various methods can be adopted as filtering processing performed by the drowsiness estimation device 13 .

Next, a specific example of filtering processing performed by the drowsiness estimation device 13 will be described.
Filtering by the drowsiness estimation device 13 is applicable when the filtering calculation window width T _N [seconds] is at least 0.1 [seconds] (T _N ≧0.1). The reason is that the blinking time is 0.1 to 0.15 [seconds]. For example, when the frame rate fs is 30 [frames/second], filtering by the drowsiness estimation device 13 can be applied when the number of frames N is 3 or more. A case where the number of frames N is equal to 30 and the filtering calculation window width T _N [seconds] is 1 [seconds] will be described below, but the embodiment is not limited to such an example.

The drowsiness estimation device 13 performs filtering in units of a filtering calculation window width T _N [seconds], acquires the maximum value of the time-series information X(t) of the degree of eye openness, and uses the acquired maximum value as the degree of eye openness after filtering. Output as time-series information X _F (t).
The drowsiness estimation device 13 takes the maximum value of the time-series information X(t) of the degree of eye openness for the filtering calculation window width T _N [seconds], as shown in Equation (7).

max[] is an operator that takes the maximum value of the elements. By this filtering, the drowsiness estimation device 13 removes changes due to blinking from the time-series information on the degree of eye openness.
In addition, when the subject's eyes are clear, the degree of eye opening can be kept constant. On the other hand, when the subject is sleepy, the degree of eye opening cannot be kept constant. In particular, when the subject is sleepy, the degree of eye openness is always smaller than when the subject is alert, and the maximum value of the degree of eye openness obtained by filtering is also considered to be smaller. It is considered that the more drowsy the subject is, the smaller the value of the time-series information X _F (t) of the degree of eye openness after filtering.

In the first example, the time-series information of the degree of eye opening output by the drowsiness estimation device 13 corresponds to an example of drowsiness estimation information. The drowsiness estimation device 13 may output any one of X(t), _XL (t), and _XR (t) without filtering as drowsiness estimation information, or a combination thereof. Alternatively, the drowsiness estimation device 13 may output any one of X _F (t), X _FL (t), and X _FR (t) after filtering as the drowsiness estimation information, or a combination thereof. .
The drowsiness estimation information correction device 14 performs the above-described correction on the drowsiness estimation information output by the drowsiness estimation device 13, and outputs post-correction drowsiness estimation information.

Alternatively, the drowsiness estimation device 13 may calculate the degree of variation in the time-series information X _F (t) of the degree of eye openness after filtering, and output the calculated degree of variation as the first feature amount F1(T). good. In this case, the first feature amount F1(T) corresponds to an example of drowsiness estimation information.
The first feature quantity F1(T) is expressed as in Equation (8).

Here, T is a time index and indicates a window number (0, 1, 2, . . . ) of the feature amount calculation window.
V[] is an operator that calculates the degree of variability. Various things such as variance, standard deviation, difference between the maximum value and the minimum value, or entropy can be used as the degree of variation here.
M indicates the number of frames for feature quantity extraction processing. Using the feature quantity calculation window width T _M [seconds] and the frame rate fs [frames/second], it is expressed as “M=T _M ×fs”.

Note that the data used by the drowsiness estimation device 13 is not limited to the time-series information X _F (t) of the degree of eye openness after filtering shown in Equation (8). The drowsiness estimation device 13 replaces the filtered time-series information X _F (t) of the degree of eye openness in Equation (8) with the filtered time-series information X _FL (t) of the degree of eye openness of the left eye, or the filtered The subsequent time-series information X _FR (t) of the degree of openness of the right eye may be used.

For example, the drowsiness estimation device 13 calculates the degree of variation in the time-series information X _FL (t) of the degree of openness of the left eye after filtering within the feature amount calculation window width T _M [seconds], and calculates the degree of variation as You may make it output as 1st feature-value F1 (T). In addition, the drowsiness estimation device 13 calculates the degree of variation of the time-series information X _FR (t) of the degree of openness of the right eye after filtering within the feature amount calculation window width T _M [seconds], and calculates the degree of variation as You may make it output as 1st feature-value F1 (T).
In this case, the first feature amount F1(T) corresponds to an example of drowsiness estimation information.

[Second example]
The second example is the same as the first example, except that the feature amount calculation processing is different as described later.
In the second example, the drowsiness estimation device 13 calculates the absolute value of the inter-frame difference (hereinafter referred to as the “inter-frame difference”) of the eye openness time series information X(t), and calculates the feature value calculation window width (T _M ) Obtain the maximum value of the inter-frame difference within [seconds], and output the maximum value as the second feature quantity F2(T).
Therefore, the drowsiness estimation device 13 calculates and outputs the second feature amount F2(T) from the time-series information X(t) of the degree of eye openness. The maximum absolute value of the difference between frames is useful for estimating drowsiness because the value is small when the subject is sleepy and increases when the subject is alert.
The maximum value of the inter-frame difference, that is, the second feature amount F2(T) is expressed as in Equation (9).

The reason why "1-max[]" is set here is that the less drowsiness is, the smaller the value is, as with the other feature amounts.
Data used by the drowsiness estimation device 13 is not limited to the time-series information X(t) of the degree of eye openness shown in Equation (9). The drowsiness estimation device 13 replaces the time-series information X(t) of the degree of eye openness in Equation (9) with the time-series information X L (t) of the degree of openness of the left eye or the time-series information X _L (t) of the degree of openness of the right eye. Information X _R (t) may be used.

For example, the drowsiness estimation device 13 calculates the maximum value of the difference between the frames of the time-series information X _L (t) of the degree of openness of the left eye within the feature value calculation window width T _M [seconds]. The value may be output as the second feature quantity F2(T). In addition, the drowsiness estimation device 13 calculates the maximum value of the difference between the frames of the time-series information X _R (t) of the degree of openness of the right eye within the feature amount calculation window width T _M [seconds], and The value may be output as the second feature quantity F2(T).
The second feature quantity F2(T) output by the drowsiness estimation device 13 in the second example corresponds to an example of drowsiness estimation information. The drowsiness estimation information correcting device 14 performs the above-described correction on the drowsiness estimation information (second feature value F2(T)) output by the drowsiness estimation device 13, and outputs post-correction drowsiness estimation information.

[Third example]
The third example is the same as the first example, except that the process of calculating the feature amount is different.
In the third example, the drowsiness estimation device 13 detects closed eyes from the time-series information X(t) of the degree of eye openness, and obtains an eye closure rate (eye closure rate) within the feature amount calculation window width (T _M ) [seconds]. , and outputs the closed-eye ratio as the third feature quantity F3(T). Therefore, the drowsiness estimation device 13 calculates and outputs the third feature amount F3(T) from the time-series information X(t) of the degree of eye openness. This eye closure rate is useful for estimating drowsiness because the value of the eye closure ratio is large when the subject is sleepy and is small when the subject is alert.
The closed-eye ratio, that is, the third feature amount F3(T) is expressed as in Equation (10).

However, C is the element of time-series information [X(T× _M +t), . Among them, the number of elements below the closed-eye determination threshold value (for example, 0.5).
Note that the drowsiness estimation device 13 calculates the third feature value F3(T), instead of the time-series information X(t) of the degree of eye openness, the time-series information X _L (t) of the degree of openness of the left eye, Alternatively, the time-series information X _R (t) of the degree of openness of the right eye may be used. In that case, the drowsiness estimation device 13 calculates the eye closure ratio of the time-series information X _L (t) of the degree of openness of the left eye within the feature amount calculation window width T _M [seconds], and sets the eye closure ratio to the third is output as a feature amount F3(T) of . In addition, the drowsiness estimation device 13 calculates the eye closure ratio of the time-series information X _R (t) of the degree of openness of the right eye within the feature value calculation window width T _M [seconds], and calculates the eye closure ratio as the third Output as feature quantity F3(T).

In addition, the drowsiness estimation device 13 calculates the proportion of both the time-series information X _L (t) of the degree of openness of the left eye and the time-series information X _R (t) of the degree of right eye openness at the same time. , and the ratio of closed eyes may be output as the third feature amount F3(T). In addition, the drowsiness estimation device 13 calculates the eye closure rate of the time-series information X _L (t) of the eye openness of the left eye and the eye closure rate of the time-series information X _R (t) of the eye openness of the right eye. The third feature value F3(T) as an element may be output as [the closed-eye ratio of X _L (t), the closed-eye ratio of X _R (t)]. In addition, the drowsiness estimation device 13 uses a plurality of closed-eye determination thresholds (for example, 0.5 and 0.8) to calculate a plurality of closed-eye ratios, and a third feature amount F3 (T )=[eye-closed ratio when eye-closed determination threshold is 0.5, eye-closed ratio when eye-closed determination threshold is 0.8] may be output.
The third feature quantity F3(T) output by the drowsiness estimation device 13 in the third example corresponds to an example of drowsiness estimation information. The drowsiness estimation information correction device 14 performs the above-described correction on the drowsiness estimation information (third feature amount F3(T)) output by the drowsiness estimation device 13, and outputs post-correction drowsiness estimation information.

[4th example]
The fourth example is the same as the first example, except that the process of calculating the feature amount is different.
In the fourth example, the drowsiness _estimation device 13 _calculates the absolute value of the motion difference ( hereinafter referred to as a "motion difference") is calculated, the average value of the motion differences within the feature amount calculation window width (T _M ) [seconds] is calculated, and the average value of the motion differences is used as a fourth feature amount F4 ( T).

Therefore, the drowsiness estimation device 13 calculates and outputs the fourth feature amount F4(T) from the time-series information X(t) of the degree of eye openness. This motion difference average value is useful for estimating drowsiness because the value increases when the subject is sleepy and decreases when the subject is alert.
The average value of motion differences, that is, the fourth feature amount F4(T) is expressed as in Equation (11).

The fourth feature quantity F4(T) output by the drowsiness estimation device 13 in the fourth example corresponds to an example of drowsiness estimation information. The drowsiness estimation information correction device 14 performs the above-described correction on the drowsiness estimation information (fourth feature value F4(T)) output by the drowsiness estimation device 13, and outputs post-correction drowsiness estimation information.

[Example 5]
The fifth example is the same as the first example, except that the processing for feature amount calculation is different.
In the fifth example, the drowsiness estimation device 13 calculates the first feature amount F1(T) calculated with a certain feature amount calculation window width (T _M ) [seconds] and the feature amount calculation window width (T _M ) adjacent thereto. The absolute value of the difference from the first feature value F1 (T−1) calculated in [seconds] (hereinafter referred to as the “difference between adjacent windows”) is calculated, and the difference between the adjacent windows is calculated as the second 5 as a feature amount F5(T).

Therefore, the drowsiness estimation device 13 calculates and outputs the difference between adjacent windows in the first feature amount F1(T) as the fifth feature amount F5(T). The difference between adjacent windows is useful for estimating drowsiness because the value of the difference between adjacent windows is large when the user is sleepy and small when the user is alert.
The difference between adjacent windows, that is, the fifth feature amount F5(T) is expressed by Equation (12).

Note that the drowsiness estimation device 13 uses the difference between adjacent windows in the second feature amount F2 (T), the third feature amount F3 ( T) or the difference between adjacent windows in the fourth feature amount F4(T) may be calculated and output as the fifth feature amount F5(T).
The fifth feature value F5(T) output by the drowsiness estimation device 13 in the fifth example corresponds to an example of drowsiness estimation information. The drowsiness estimation information correction device 14 performs the above-described correction on the drowsiness estimation information (fifth feature amount F5(T)) output by the drowsiness estimation device 13, and outputs post-correction drowsiness estimation information.

[6th example]
The sixth example is the same as the first example, except that the process of calculating the feature amount is different.
In the sixth example, the drowsiness estimation device 13 calculates the logarithm of the first feature amount F1(T) and outputs the logarithm as the sixth feature amount F6(T). Therefore, the drowsiness estimation device 13 calculates and outputs the logarithm of the first feature amount F1(T) as the sixth feature amount F6(T). Here, it is generally said that human senses follow logarithms. By taking the logarithm of the first feature amount F1(T), the drowsiness estimating device 13 can output drowsiness estimation information close to human sensation.
The logarithm of the first feature quantity, that is, the sixth feature quantity F6(T) is expressed as in Equation (13).

where α is a small positive value, eg, 10 ⁻⁶ . The reason why the small positive value α is added to the first feature quantity F1(T) is to prevent the content of the logarithm from becoming zero.
Note that the drowsiness estimation device 13 replaces the logarithm of the first feature amount F1(T) with the logarithm of the second feature amount F2(T), the logarithm of the third feature amount F3(T), the logarithm of the fourth The logarithm of the feature quantity F4(T) or the logarithm of the fifth feature quantity F5(T) may be calculated as the sixth feature quantity F6(T) and output.
The sixth feature quantity F6(T) output by the drowsiness estimation device 13 in the sixth example corresponds to an example of drowsiness estimation information. The drowsiness estimation information correction device 14 performs the above-described correction on the drowsiness estimation information (the sixth feature F6(T)) output by the drowsiness estimation device 13, and outputs post-correction drowsiness estimation information.

The sixth feature quantity F6(T) output by the drowsiness estimation device 13 in the sixth example corresponds to an example of drowsiness estimation information. The drowsiness estimation information correction device 14 performs the above-described correction on the drowsiness estimation information (the sixth feature F6(T)) output by the drowsiness estimation device 13, and outputs post-correction drowsiness estimation information.

[Example 7]
The seventh example is the same as the first example, except that the processing for calculating the feature amount is different.
In the seventh to ninth examples, the drowsiness estimation device 13 calculates the above-described feature quantity (eg, the first feature quantity F1(T)) for each statistic calculation window width T _K [seconds]. Statistics F7(S) to F9(S) are calculated and used as feature amounts. Here, S is a time index and indicates a window number (0, 1, 2, . . . ) of a statistic calculation window. For example, assume that the number of first feature quantities T1(T) within the statistic calculation window width T _K [seconds] is K.

In the seventh example, the drowsiness estimation device 13 uses the first feature amount F1(T) as the feature amount described above, and uses the average value of the first feature amount F1(T) as the statistic amount. The drowsiness estimation device 13 calculates the average value of the first feature amount F1(T) within the statistical amount calculation window width T _K [seconds], and uses the average value as the seventh feature amount F7(S). Output. Therefore, the drowsiness estimation device 13 calculates the average value of the first feature amount F1(T) as the seventh feature amount F7(S) for each statistic calculation window width T _K [seconds], and outputs do.
The average value of the first feature quantity, that is, the seventh feature quantity F7(S) is expressed as in Equation (14).

average[] is an operator that calculates the average value of elements.
The value of T in equation (14) expresses the relationship between the window number of the feature amount calculation window when obtaining the first feature amount F1 and the window number of the feature amount calculation window when obtaining the seventh feature amount F7. Acts as an offset to indicate.

Instead of the average value of the first feature amount F1(T), the drowsiness estimation device 13 calculates the average value of the second feature amount F2(T), the average value of the third feature amount F3(T), the average value of the fourth feature amount F3(T), Calculate the seventh feature amount F7(S) using the average value of the feature amount F4(T), the average value of the fifth feature amount F5(T), and the average value of the sixth feature amount F6(T) and output.
The seventh feature quantity F7(S) output by the drowsiness estimation device 13 in the seventh example corresponds to an example of drowsiness estimation information. The drowsiness estimation information correction device 14 performs the above-described correction on the drowsiness estimation information (seventh feature F7(S)) output by the drowsiness estimation device 13, and outputs post-correction drowsiness estimation information.

[Example 8]
The eighth example is the same as the first example, except that the process of calculating the feature amount is different.
In the eighth example, the drowsiness estimation device 13 calculates the standard deviation of the first feature quantity F1(T) within the statistic calculation window width T _K [seconds], and uses the standard deviation as the eighth feature quantity Output as F8(S).
Therefore, the drowsiness estimation device 13 calculates the standard deviation of the first feature amount F1(T) as the eighth feature amount F8(S) for each statistic calculation window width T _K [seconds], and outputs do.
The standard deviation of the first feature quantity, that is, the eighth feature quantity F8(S) is expressed as in Equation (15).

standard_dev[] is an operator that calculates the standard deviation of the elements.
The value of T in equation (15) expresses the relationship between the window number of the feature amount calculation window when obtaining the first feature amount F1 and the window number of the feature amount calculation window when obtaining the eighth feature amount F8. Acts as an offset to indicate.

The drowsiness estimation device 13 replaces the first feature amount F1(T) with a second feature amount F2(T), a third feature amount F3(T), a fourth feature amount F4(T), a 5 feature amount F5(T) or the standard deviation of the sixth feature amount F6(T) may be calculated as the eighth feature amount F8(S) and output.
The eighth feature quantity F8(S) output by the drowsiness estimation device 13 in the eighth example corresponds to an example of drowsiness estimation information. The drowsiness estimation information correction device 14 performs the above-described correction on the drowsiness estimation information (eighth feature amount F8(S)) output by the drowsiness estimation device 13, and outputs post-correction drowsiness estimation information.

[9th example]
The ninth example is the same as the first example, except that the process of calculating the feature amount is different.
In the ninth example, the drowsiness estimation device 13 calculates the variance of the first feature quantity F1(T) within the statistical quantity calculation window width T _K [seconds], and converts the variance to the ninth feature quantity F9 ( S). Therefore, the drowsiness estimation device 13 calculates the variance of the first feature quantity F1(T) as the ninth feature quantity F9(S) for each statistic calculation window width T _K [seconds], and outputs .
The variance of the first feature quantity, that is, the ninth feature quantity F9(S) is expressed as in Equation (16).

variance[] is an operator that calculates the variance value of an element.
The value of T in Equation (16) expresses the relationship between the window number of the feature amount calculation window when obtaining the first feature amount F1 and the window number of the feature amount calculation window when obtaining the ninth feature amount F9. Acts as an offset to indicate.

The drowsiness estimation device 13 calculates the variance of the second feature quantity F2(T), the variance of the third feature quantity F3(T), and the fourth feature quantity instead of the variance of the first feature quantity F1(T). The variance of F4(T), the variance of the fifth feature quantity F5(T), or the variance of the sixth feature quantity F6(T) is calculated as the ninth feature quantity F9(T) and output. can be
The ninth feature quantity F9(S) output by the drowsiness estimation device 13 in the ninth example corresponds to an example of drowsiness estimation information. The drowsiness estimation information correction device 14 performs the above-described correction on the drowsiness estimation information (the ninth feature value F9(S)) output by the drowsiness estimation device 13, and outputs post-correction drowsiness estimation information.

The drowsiness estimation device 13 may output only one of the feature quantities F1(T) to F9(S) described above as the drowsiness estimation information. Alternatively, the drowsiness estimation device 13 may output a plurality of the feature quantities F1(T) to F9(S) as drowsiness estimation information.

<Example of Configuration of Embodiment>
Next, an example of the configuration of the embodiment will be described with reference to FIGS. 13 to 16. FIG.
FIG. 13 is a diagram showing an example of the configuration of the drowsiness estimation information correction device according to the embodiment. A drowsiness estimation information correcting device 610 shown in FIG.

With such a configuration, the drowsiness estimation information acquisition unit 611 acquires drowsiness estimation information based on the movement of the subject's eyelids. The drowsiness estimation information correction unit 612 calculates post-correction drowsiness estimation information by correcting the drowsiness estimation information acquired by the drowsiness estimation information acquisition unit 611 with humidity information indicating the humidity in the surrounding environment of the subject.
As a result, the drowsiness estimation information correcting device 610 can reduce the decrease in estimation accuracy due to the influence of the surrounding environment when calculating the subject's drowsiness estimation information (corrected drowsiness estimation information).

FIG. 14 is a diagram illustrating an example of the configuration of a drowsiness estimation device according to the embodiment; A drowsiness estimation device 620 shown in FIG.
With such a configuration, the eyelid movement information acquisition unit 621 acquires eyelid movement information of the subject. The drowsiness estimation unit 622 calculates the drowsiness estimation information of the subject based on the eyelid movement information acquired by the eyelid movement information acquisition unit 621 and the humidity information indicating the humidity in the surrounding environment of the subject.
As a result, the drowsiness estimation device 620 can reduce the decrease in estimation accuracy when estimating the drowsiness of the subject. In particular, the drowsiness estimation device 620 can avoid or reduce the deterioration of the drowsiness estimation accuracy due to the influence of the humidity in the surrounding environment of the subject.

FIG. 15 is a diagram showing an example of a processing procedure in the drowsiness estimation information correcting method according to the embodiment.
In the process of FIG. 15, drowsiness estimation information based on the movement of the subject's eyelids is acquired (step S11), and corrected drowsiness estimation is performed by correcting the acquired drowsiness estimation information with humidity information indicating the humidity in the surrounding environment of the subject. Information is calculated (step S12).

According to this drowsiness estimation information correcting method, when estimating a subject's drowsiness, it is possible to reduce a decrease in estimation accuracy. In particular, in the drowsiness estimation correction information related to the processing of FIG. 15, when calculating the drowsiness estimation information of the subject (post-correction drowsiness estimation information), the drowsiness estimation accuracy is reduced due to the influence of the humidity in the surrounding environment of the subject. avoid or reduce clutter.

FIG. 16 is a diagram illustrating an example of a processing procedure in the drowsiness estimation method according to the embodiment;
In the process of FIG. 16, eyelid movement information of the subject is acquired (step S21), and drowsiness estimation information of the subject is calculated based on the acquired movement information and humidity information indicating humidity in the surrounding environment of the subject. Calculate (step S22).

According to this drowsiness estimation method, when estimating the drowsiness of the subject, it is possible to reduce the decrease in estimation accuracy. In particular, in the drowsiness estimation method according to the processing of FIG. 16, it is possible to avoid or reduce the decrease in drowsiness estimation accuracy due to the influence of the humidity in the surrounding environment of the subject.

FIG. 17 is a schematic block diagram showing the configuration of a computer according to at least one embodiment;
With the configuration shown in FIG. 17 , computer 700 includes CPU 710 , main storage device 720 , auxiliary storage device 730 , and interface 740 .
Any one or more of the drowsiness estimation information correction device 14 , drowsiness estimation device 41 , drowsiness estimation information correction device 610 , and drowsiness estimation device 620 described above may be implemented in the computer 700 . In that case, the operation of each processing unit described above is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads out the program from the auxiliary storage device 730, develops it in the main storage device 720, and executes the above processing according to the program. In addition, the CPU 710 secures storage areas corresponding to the storage units described above in the main storage device 720 according to the program. Communication between each device and another device is performed by the interface 740 having a communication function and performing communication under the control of the CPU 710 .

When the drowsiness estimation information correcting apparatus 14 is implemented in the computer 700, the operations of the first control section 190 and its respective sections are stored in the auxiliary storage device 730 in the form of programs. The CPU 710 reads out the program from the auxiliary storage device 730, develops it in the main storage device 720, and executes the above processing according to the program.
In addition, CPU 710 secures a storage area corresponding to first storage unit 180 in main storage device 720 according to a program. Communication performed by the first communication unit 110 is performed by the interface 740 having a communication function and performing communication under the control of the CPU 710 .

When the drowsiness estimation device 41 is implemented in the computer 700, the operations of the second control unit 290 and its respective units are stored in the auxiliary storage device 730 in the form of programs. The CPU 710 reads out the program from the auxiliary storage device 730, develops it in the main storage device 720, and executes the above processing according to the program.
In addition, CPU 710 secures a storage area corresponding to second storage unit 280 in main storage device 720 according to a program. Communication performed by the second communication unit 210 is performed by the interface 740 having a communication function and performing communication under the control of the CPU 710 .

When the estimated drowsiness information correction device 610 is implemented in the computer 700, the operation of the estimated drowsiness information correction unit 612 is stored in the auxiliary storage device 730 in the form of a program. The CPU 710 reads out the program from the auxiliary storage device 730, develops it in the main storage device 720, and executes the above processing according to the program.
Further, the operation of the drowsiness estimation information acquisition unit 611 is executed when the interface 740 has a communication function and performs communication under the control of the CPU 710 .

When drowsiness estimation device 620 is implemented in computer 700, the operation of drowsiness estimation unit 622 is stored in auxiliary storage device 730 in the form of a program. The CPU 710 reads out the program from the auxiliary storage device 730, develops it in the main storage device 720, and executes the above processing according to the program.
Further, the operation of the eyelid movement information acquisition section 621 is executed by the interface 740 having a communication function and performing communication under the control of the CPU 710 .

A program for realizing all or part of the functions of the drowsiness estimation information correction device 14, drowsiness estimation device 41, drowsiness estimation information correction device 610, or drowsiness estimation device 620 is recorded on a computer-readable recording medium. Then, the program recorded on the recording medium may be loaded into the computer system and executed to perform the processing of each section. The "computer system" here includes an OS (operating system) and hardware such as peripheral devices.
“Computer-readable recording media” means portable media such as flexible discs, magneto-optical discs, ROM (Read Only Memory), CD-ROM (Compact Disc Read Only Memory), hard disks built into computer systems, etc. Refers to a storage device. Further, the program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system.

Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and design changes and the like are also included within the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2019-134835 filed on July 22, 2019, and the entire disclosure thereof is incorporated herein.

The present invention may be applied to a drowsiness estimation information correction device, a drowsiness estimation device, a drowsiness estimation information correction method, a drowsiness estimation method, and a recording medium.

Reference Signs List 1, 4 wakefulness control system 2, 5 driving control system 3, 6 alarm system 11 camera 12 humidity sensor 13 sleepiness estimation device 14 sleepiness estimation information correction device 15 wakefulness control device 16 environment control device 21 driving control device 22 driving device 31 Alarm device 41 Drowsiness estimation device 110 First communication unit (first communication means)
180 first storage unit (first storage means)
181 correction model storage unit (correction model storage means)
190 first control unit (first control means)
191 sleepiness estimation information correction unit (drowsiness estimation information correction means)

Claims (12)

drowsiness estimation information acquisition means for acquiring, as drowsiness estimation information, data obtained by calculating an eye closure ratio within a predetermined time period from time-series data obtained by repeatedly measuring the subject's degree of eye openness at a predetermined frequency;
drowsiness estimation information correcting means for calculating post-correction drowsiness estimation information obtained by correcting the drowsiness estimation information with humidity information indicating the humidity in the surrounding environment of the subject;
A drowsiness estimation information correction device comprising: The drowsiness estimation information correcting means corrects the post-correction drowsiness estimation information so that the lower the humidity, the smaller the drowsiness estimation information.
The drowsiness estimated information correcting device according to claim 1. The drowsiness estimation information correcting means corrects the same amount of correction as when the humidity is the upper threshold when the humidity is greater than the upper threshold.
The drowsiness estimation information correcting device according to claim 1 or 2. The drowsiness estimation information correcting means corrects the same amount of correction as when the humidity is the lower threshold when the humidity is lower than the lower threshold.
The drowsiness estimation information correcting device according to claim 1 or 2. The drowsiness estimation information acquiring means divides, as the drowsiness estimation information, time-series data obtained by repeatedly measuring the degree of eye opening of the subject at a predetermined frequency into a predetermined number of continuous measurement data, Acquire the data that takes the maximum value of the degree of eye opening for each measurement data of the number of
The drowsiness estimation information correction device according to any one of claims 1 to 4. The drowsiness estimation information obtaining means obtains, as the drowsiness estimation information, a difference in degree of eye openness for each two consecutive measurement data from time-series data obtained by repeatedly measuring the degree of eye openness of the subject at a predetermined frequency. , for each predetermined time width, obtain data that takes the maximum value of the magnitude of the difference in the degree of eye opening included in that time width;
The drowsiness estimation information correction device according to any one of claims 1 to 5. The drowsiness estimation information acquiring means obtains the drowsiness estimation information from the time-series data obtained by repeatedly measuring the degree of eye openness of each of the left eye and the right eye of the subject at a predetermined frequency, and obtains the degree of eye openness for each two consecutive measurement data. Further, the absolute value of the difference between the difference in the left eye and the difference in the right eye is obtained, and the data obtained by averaging the absolute values of the differences included in the time width for each predetermined time width is obtained. do,
The drowsiness estimated information correction device according to any one of claims 1 to 6 . eyelid movement information acquisition means for acquiring eyelid movement information of a subject;
Based on the eyelid movement information and the humidity information indicating the humidity in the surrounding environment of the subject , time-series data obtained by repeatedly measuring the eye opening degree of the subject at a predetermined frequency as the drowsiness estimation information of the subject. Drowsiness estimating means for calculating , for each predetermined time span, data obtained by obtaining a percentage of closed eyes within that time span from
A drowsiness estimation device comprising: by computer,
Acquiring, as drowsiness estimation information, data obtained by obtaining the percentage of eyes closed for each predetermined time period from time-series data obtained by repeatedly measuring the subject's degree of eye openness at a predetermined frequency, and
calculating post-correction drowsiness estimation information obtained by correcting the drowsiness estimation information with humidity information indicating the humidity in the surrounding environment of the subject;
A drowsiness estimation information correction method comprising: by computer,
Acquire eyelid movement information of the subject,
Based on the eyelid movement information and the humidity information indicating the humidity in the surrounding environment of the subject , time-series data obtained by repeatedly measuring the eye opening degree of the subject at a predetermined frequency as the drowsiness estimation information of the subject. From, for each predetermined time width, calculate the data for the eye closure ratio within that time width ,
A sleepiness estimation method comprising: to the computer,
Acquiring, as drowsiness estimation information, data obtained by obtaining the percentage of eyes closed for each predetermined time period from time-series data obtained by repeatedly measuring the subject's degree of eye openness at a predetermined frequency, and
calculating post-correction drowsiness estimation information obtained by correcting the drowsiness estimation information with humidity information indicating the humidity in the surrounding environment of the subject;
A program that does something. to the computer,
Acquire eyelid movement information of the subject,
Based on the eyelid movement information and the humidity information indicating the humidity in the surrounding environment of the subject , time-series data obtained by repeatedly measuring the eye opening degree of the subject at a predetermined frequency as the drowsiness estimation information of the subject. From, for each predetermined time width, calculate the data for the eye closure ratio within that time width ,
A program that does something.

Images