JP2020057198A - Sleepiness prediction device - Google Patents

Abstract

To provide a device for predicting sleepiness of a driver of a vehicle and the like that allows the prediction of the sleepiness with higher accuracy before sleepiness level actually increases considering that a physiological state of the driver is affected by changes in the driving operation of the driver.SOLUTION: A device according to the present invention predicts that sleepiness occurs after a lapse of a predetermined time period from when a feature amount state determined by a physiological state feature amount calculated sequentially from time-series data of a driver's physiological state in a feature amount collection time range, and a vehicle motion feature amount that is sequentially calculated from a motion state of a vehicle or the like that changes according to a driving operation of the driver, amounts to a feature amount state where the driver's sleepiness level reaches a predetermined level within a predicted time range after a predetermined time has passed from a feature amount collection time range. Means for predicting the occurrence of sleepiness is prepared by machine learning using previously-prepared time-series data of the physiological state feature amount and the vehicle motion feature amount in which sleepiness level is labeled, as supervised learning data.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an apparatus for predicting the occurrence of drowsiness of a human, and more particularly, to the drowsiness of a driver during driving using an index value obtained from the opening / closing operation of eyelids of a driver of a vehicle or a moving body. It relates to an apparatus for predicting occurrence.

  As a device for detecting human drowsiness, an image of a human face or eyelids is taken, and in the captured image, changes in eyelid movements and other facial expressions are captured to determine whether or not the person feels drowsy. Various determination devices have been proposed. For example, in Patent Literature 1, in a photographed image of the subject's eyes and its surroundings, the time during which the eyelids are continuously opened (hereinafter, eye opening time) is observed, and the variation in the eye opening time ( (Standard deviation) is calculated, and when the standard deviation of the eye opening time decreases with an increase in drowsiness and falls below a predetermined threshold, it is proposed to determine that “snapping has occurred”. Further, in Patent Document 2, the occurrence of four types of facial expressions of sighing, yawning, eye opening, and mouth movements when frowning are measured from an image of the face of the subject, and different types of facial expressions are measured. A configuration has been proposed in which an arousal level defined corresponding to the frequency of occurrence of two combinations of facial expressions is output as an estimation result. In Patent Documents 3 to 5, when drowsiness increases, the opening and closing movement of the eyelids does not slow down, but intentionally intensifies blinks, opens eyes, or does not blink, and returns. In the case of a subject whose opening and closing movements become intense, even in the case of a subject in which opening and closing movements become intense, an amount correlated with the presence or absence or degree of drowsiness of the subject so that the occurrence of drowsiness of the subject can be detected more accurately than before. When the opening / closing feature value deviates from either the upper side or the lower side from the value range defined by the upper and lower thresholds when the subject does not feel drowsy, the subject becomes drowsy. There has been proposed a configuration of a drowsiness detection device that determines that the user feels sleep. In Non-Patent Document 1, a video of a chest image is taken from the head of a subject performing a driving operation with a vehicle driving simulator, and the expression of the subject in the video image is given to the evaluator. The sleepiness expression value, which indicates the degree of sleepiness of the subject obtained by observing and evaluating the degree of sleepiness of the subject as a numerical value, is an objective value of the subject's brain waves, blinks, heart rate, etc. It has a correlation with physiological information that is typically measured, and it is suggested that it is possible to detect the degree of drowsiness or predict the occurrence of drowsiness using such physiological information.

JP 2011-167398A JP 2007-236488 JP-A-2006-146978 JP-A-2006-219419 JP 2017-209262A

Transactions of the Japan Society of Mechanical Engineers (C) 63, 613 (1997-9) 96-1780 3059-3066

  Meanwhile, a drowsiness detection device for detecting the presence or absence or degree of drowsiness of a subject using a feature amount obtained from eyelid opening / closing motion or blinking motion of the subject as described in the above-mentioned patent document is used in an automobile or the like. It is found that when used for detecting the drowsiness of a driver while driving a vehicle or a moving body (vehicle or the like), the detection accuracy of drowsiness may be reduced depending on the running state of the vehicle or the like. Was. According to a study by the inventors of the present invention, during driving of a vehicle or the like, for example, when a driving operation such as steering or acceleration / deceleration operation of a vehicle is performed during traveling on a curved road or changing lanes. When the driver gazes at a specific point or is nervous, the driver's eyelid opening / closing movement or blinking movement changes from a normal state, that is, a state during steady running. It turned out that there is. That is, the state of the driver's eyelid opening / closing motion or blinking motion may change due to a change in the driver's driving operation regardless of the degree of drowsiness, and the eyelid opening / closing motion or blinking motion or In a configuration in which the drowsiness is detected by assuming that the feature amount obtained from the change only due to the occurrence of drowsiness, performing the driving operation by the driver may reduce the drowsiness detection accuracy. Can be one of the factors. The same applies to the occurrence of drowsiness during driving of a driver such as a vehicle using other physiological conditions correlated with the occurrence of drowsiness of a person, for example, a respiratory state, a heartbeat state, a pulse state, and an electroencephalogram. It can also occur when detecting. Therefore, in the configuration for detecting the occurrence of drowsiness using the characteristic amount obtained from the driver's eyelid opening / closing movement or blinking movement or other physiological state, the driver's driving operation also includes the driver's eyelid opening / closing. It is expected that the detection accuracy of the occurrence of drowsiness can be improved by considering the influence on the exercise or the blink movement.

  Further, in the case of using the device for detecting the occurrence of driver drowsiness as described above in order to ensure the safety of driving a vehicle or the like, before the drowsiness of the driver actually occurs, or Preferably, the degree can be predicted in advance and notified or communicated to the driver or other occupants before the degree actually increases. If the occurrence of strong drowsiness can be predicted before the occurrence, the occurrence of such drowsiness can be prevented, or the vehicle can be stopped or the parking area can be entered before the strong drowsiness occurs. And the like can be safely retracted from the traveling path, which is very advantageous. Therefore, for the safety of driving a vehicle or the like, a device for detecting drowsiness is not a device for detecting the actual occurrence of drowsiness of a driver of a vehicle or the like, but rather a device of the driver of a vehicle or the like in the near future. It is preferable to use a device that can detect the possibility of drowsiness, that is, a device that predicts drowsiness.

  Thus, one object of the present invention is an apparatus that predicts the occurrence of drowsiness of a driver by using an index value obtained from the opening and closing movement of eyelids or other physiological states of a driver while driving a vehicle or the like. In consideration of the fact that the opening and closing movement of the driver's eyelids or other physiological conditions are affected by the execution of the driving operation by the driver, the drowsiness occurs more accurately and before the degree of drowsiness actually increases. Another object of the present invention is to provide a device capable of predicting that the degree of drowsiness will increase.

According to the present invention, the above-described object is an apparatus (hereinafter, referred to as a “sleepiness occurrence prediction device”) that predicts the occurrence of drowsiness of a driver (ie, a subject) while driving a vehicle or a moving object. So,
Means for detecting a physiological state of the driver;
Means for sequentially calculating the physiological state feature quantity from the time series data of the physiological state,
Means for detecting the motion state of the vehicle or the moving body that changes according to the driving operation of the driver,
Means for sequentially calculating a vehicle motion feature amount from time-series data of the motion state of the vehicle or the moving body,
Using the physiological state feature amount and the vehicle motion feature amount, the drowsiness occurrence prediction means for predicting in advance the occurrence of drowsiness of the driver, the first time width in the feature amount collection time range The feature amount state determined by the physiological state feature amount and the vehicle motion feature amount is the third time width after the elapse of the second time width margin time period from the last time point of the feature amount collection time range. When the drowsiness level of the driver reaches a predetermined level within the prediction time range, the prediction time after the lapse of the margin time range from the last time of the feature value collection time range when the feature amount state is reached in which the driver's drowsiness level reaches a predetermined level. Means for predicting drowsiness occurrence when the drowsiness level of the driver reaches a predetermined level within a range,
Sleepiness occurrence prediction result output means for outputting a sleepiness occurrence prediction result by the sleepiness occurrence prediction means,
The drowsiness occurrence prediction means performs the first learning by machine learning using a plurality of previously prepared time-series data of the physiological state feature amount and the vehicle motion feature amount labeled with a drowsiness level as supervised learning data. The drowsiness level in the first group of the learning data defined in the feature space defined by the physiological state feature and the vehicle motion feature in the time width of A second group of the learning data is placed in a distribution range of the feature amount state in a time range that is traced back by the third time width from a time point before the second time width before reaching the level. In the time range of the third time width after the elapse of the second time width from the point in time when the feature amount state of the learning data enters, the drowsiness level in the second group of the learning data is Reach the predetermined level Point in time is achieved by a device which is prepared to enter.

In the above configuration, the “physiological state of the driver” may be any physiological state having a correlation with the occurrence or degree of drowsiness of a person, for example, a respiratory state, a heartbeat state, a pulse state, an electroencephalogram, or the like. However, typically, the driver's "eyelid open / closed state" may be used. Therefore, the “means for detecting the physiological condition of the driver” may be a means for sequentially detecting a respiratory state, a heartbeat state, a pulse state, an electroencephalogram, and the like. Any device that can sequentially determine whether the eyelid of the subject (driver) is open or closed may be used. As means for sequentially detecting the open / closed state of the eyelid, more specifically, a device for determining the open / closed state of the eyelid based on an image of the eyelid of the subject taken by a camera, or an eye electrogram of the subject A device that determines the open / closed state of the eyelids based on the number is adopted. The “physiological state feature amount” calculated from the time series data of the physiological state is a state quantity representing an arbitrary physiological state that changes according to the degree of sleepiness of the person in the above-described sequentially detected physiological state. It may be. When the detected physiological state is the “open / closed state of the eyelids”, the “time interval of transition between the open state and the closed state of the eyelids” (every hour by the means for detecting the open / closed state of the eyelids) In the obtained data of the judgment result of whether the eyelid is open or closed, the interval at which the transition of the eyelid from the open state to the closed state or from the closed state to the open state occurs) Any eyelid opening / closing feature amount calculated by the above method may be used. It is possible to experimentally select what feature amount can be used as the “physiological state feature amount” or the “lid opening / closing feature amount” in the apparatus of the present invention. In the embodiment, specifically, the following three quantities of time-series data are advantageously used as the physiological condition feature quantity (eyelid opening / closing feature quantity).
(1) The ratio of the closed eye time (the length of time the eyelids are in the closed state) in the calculation unit time [eye closed time ratio],
(2) the number of occurrences of blinks (transition from the open state of the eyelids to the closed state) [number of blinks] in the calculation unit time,
(3) Blink interval in calculation unit time, that is, dispersion of eye opening time (length of time the eyelid is in the open state) [variation of blink interval]
Furthermore, as the "physiological state feature value", statistics (standard deviation value, variance, average value, median value, etc.) such as eyelid opening / closing state transition time interval, breathing state, heartbeat state, pulse state, and brain wave can be used. It is.

Further, in the above configuration, the “motion state of the vehicle or the moving body” refers to the motion state of the vehicle or the moving body that changes in response to the driving operation by the physical movement of the driver while driving the vehicle or the moving body. The value may be represented by a steering wheel operation corresponding to the movement of the driver's upper body (arm), an accelerator pedal operation corresponding to the movement of the driver's lower body (foot, leg), a brake pedal operation, or the like. It may be a value representing a changing motion state of the vehicle or the moving body. Therefore, the means for detecting the motion state of the vehicle or the moving body that changes according to the driving operation of the driver may be a steering angle sensor, an accelerator opening (throttle opening) sensor, a vehicle speed sensor (wheel speed sensor), or the like. The detected motion state of the vehicle or the moving body may be a steering angle, an accelerator opening, a vehicle speed, an acceleration / deceleration, or the like. As the vehicle motion feature amount sequentially calculated from the time series data of the motion state of the vehicle or the moving body, the driving operation that affects the physiological state of the driver or the open / closed state of the eyelid is reflected on the vehicle or the moving body. Any state quantity representing a change in the running state may be used. Which feature can be used as the "vehicle motion feature" in the apparatus of the present invention can be experimentally selected. In the embodiment, the following three quantities of time-series data are advantageously used as the vehicle motion feature quantity.
(1) Variance of steering angle differential value per unit time of calculation [variance of change in steering wheel turning angle (change from zero position)]
(2) Dispersion of differential value of accelerator opening in operation unit time [variance of change in accelerator depression amount] (3) Dispersion of differential value of vehicle speed in operation unit time [variance of acceleration]
Further, as the “vehicle motion feature amount”, other amounts (standard deviation value, variance, average value, median value, differential value, integral value, etc.) calculated from each vehicle motion index value can also be used.

  In the above configuration, the “drowsiness occurrence predicting means” is, in a nutshell, using a physiological condition feature value and a vehicle motion feature value, and after an elapse of a time arbitrarily set from the current time, If the drowsiness level of the driver of the vehicle or the moving object is likely to reach a predetermined level, it is configured to detect this in advance, that is, before the degree of drowsiness reaches the predetermined level. In other words, the drowsiness occurrence predicting means refers to the physiological state feature value and the vehicle motion feature value and determines whether the drowsiness level of the driver may reach the predetermined level after a lapse of a predetermined time from the current time. Monitor whether. The “drowsiness level” is an index value indicating the degree of drowsiness of a person, and the degree of drowsiness from a state in which a person does not feel drowsiness to a state in which a person feels strong drowsiness is determined in a plurality of stages (for example, two levels). To 4 stages), and can be determined by observation of the expression of the subject (by the evaluator), subjective evaluation by the subject himself, brain wave measurement, etc. (for example, Non-Patent Document 1) See “Sleepiness expression value”). The “predetermined level” for the “drowsiness level” can be appropriately set by the designer or user of the device in consideration of the driving safety of the vehicle or the moving object.

  In the drowsiness occurrence predicting means, more specifically, first, the feature amount state (the current state) determined by the physiological state feature amount and the vehicle motion feature amount in the feature amount collection time range of the first time width. State). Here, the “feature amount state” is a physiological state at each time point when the feature amount in the “feature amount collection time zone” having the “first time width” of the driver driving the vehicle or the moving body is calculated. This is a feature amount vector using the feature amount and the vehicle motion feature amount as variables (that is, the number of dimensions of the feature amount vector is (number of calculation times of the feature amount within the feature amount collection time range) × (physiological state feature amount and The "first time width" is a width including at least one feature value calculation time point, and is determined to an appropriate value through machine learning described later (the first time width). There may be a case where the number of calculation points within the time width is 1. In such a case, a feature quantity vector is determined by the physiological state feature quantity and the vehicle motion feature quantity at each calculation time point.) Then, the current feature amount state is changed to the “third time period” after the lapse of the “margin time zone” having the “second time width” from the current time point (that is, from the last time point of the feature amount collection time area). When the drowsiness level of the driver is predicted to reach a predetermined level within the "predicted time range" having the "time width", the "predicted time range" Is predicted that the drowsiness level of the driver may reach a predetermined level, that is, prediction of drowsiness occurrence is made.

  In the above configuration of the drowsiness occurrence predicting means, the “second time width” of the “margin time zone” indicates a time point at which a drowsiness occurrence prediction is obtained (a time point at which the drowsiness occurrence prediction can be notified) and a drowsiness. This is the time width between the first time point in the future time range where the occurrence is predicted and is appropriately set in consideration of an appropriate time margin when notifying the driver or the like in advance of the prediction of drowsiness. You. In other words, if the second time width is too short, even if the driver or the like is informed of the prediction of the occurrence of drowsiness, evacuation to a place where the vehicle or the like can be safely stopped (a parking area or the like) to cope with the drowsiness. Less time to get them to sleep, and less time to take measures to reduce drowsiness (drinking beverages, chewing gum, playing music, etc.) It will be. On the other hand, if the second time width is too long, that is, if the notification of the prediction is too early, the reliability of the prediction may be reduced, or the driver may be too early to take measures against drowsiness. Therefore, the second time width is set to an appropriate time width by the designer or the user to notify the prediction of drowsiness occurrence before the time point at which drowsiness is predicted. The “third time width” of the “prediction time range” is the width of the (future) time range in which drowsiness is predicted, and is appropriately set in consideration of the drowsiness prediction performance of the present apparatus. Is done. In the embodiment, the time at which the prediction of drowsiness is obtained is made 60 to 900 seconds before the time at which drowsiness is predicted, that is, the second time width is set to about 60 seconds, The third time width may be set to about 840 seconds.

  The drowsiness occurrence prediction means for executing the drowsiness occurrence prediction as described above, as described above, converts the time series data of the physiological state feature amount and the vehicle motion feature amount, which are prepared in advance and labeled with the drowsiness level. It is configured through machine learning used as supervised learning data.

  In such machine learning, first, using a first group of data (teacher data) of learning data, a feature space (e.g., a physiological space feature of a first time span) and a vehicle motion feature are spanned. Within the space in which the feature vector is distributed), a time period that has been moved back by a third time width from a time point that is a second time width earlier than a time point when the drowsiness level reaches a predetermined level (a drowsiness occurrence time point). A process of defining an area (identification area) in which the feature amount state is distributed using a function (identification function) for determining a boundary surface of the identification area is executed. As a method of determining the identification region, for example, any method that can define a distribution region of a specific feature amount group as an identification region in a feature amount space, such as a support vector machine method and a k-NN method, is used. May be. Next, the point in time when the feature amount state in the data of the second group (verification data) of the learning data enters the above-described identification area (the drowsiness prediction point) is set to be a second time later than the drowsiness occurrence time. Whether it is within the time range that has been traced back by the third time width from the time point before the time width of, in other words, the drowsiness occurrence time in the data is the time elapsed from the sleepiness prediction time by the second time width. A process of checking whether or not the process is started before the third time width elapses is executed. Then, while changing the first time width and the internal parameters that determine the characteristics of the discriminant function, the drowsiness prediction time at which the feature amount state obtained from the verification data enters the discrimination area defined using the teacher data. Is repeatedly executed, and the number of data in which the drowsiness occurrence point falls within the time zone of the third time width after the elapse of the second time width from the drowsiness prediction time point (in other words, the drowsiness prediction time is Also, the first time width and the internal parameters of the discriminant function that maximize the number of data that fall within the range from the time before the second time width to the time before the third time width) are searched and determined, An identification function is generated.

  The first time interval and the internal parameters of the discriminant function determined as described above are, if necessary, the sleepiness level within the third time interval after the passage of the second time interval at a certain point in time. When the possibility of reaching the predetermined level is high, the feature amount state (determined using the determined first time width) at that time is determined using the determined internal parameter in the feature amount space. This is a parameter for determining the feature amount state and the discrimination function so as to enter the discrimination area defined by the discrimination function. That is, the feature state at the current time point determined using the first time width determined as described above is the identification area defined by the identification function determined using the internal parameters determined as described above. When entering, the drowsiness level may reach the predetermined level within the third time span (predicted time zone) after the lapse of the second time span (the extra time zone) from the current time point Is high. Thus, the first time width and the internal parameters of the discriminant function determined as described above are respectively used for determining the current feature amount state and for the discrimination area (the second time width from the last time point of the feature amount collection time region). Used to generate a discriminant function that defines a feature state in which the driver's drowsiness level reaches a predetermined level within a predicted time range of the third time width after the allowance time range has elapsed. Will be done.

  In the search for the internal parameters of the first time interval and the discriminant function, when a plurality of available sets of the first time interval and the internal parameters of the discriminant function are found, the notification of the prediction of the drowsiness occurrence is as described above. Since it is advantageous to be able to perform as early as possible (as long as the condition is satisfied), the combination of the first time width and the internal parameter of the discriminant function, in which the time width between the drowsiness prediction time and the drowsiness occurrence time is as long as possible, is used. May be adopted as a set used in the above.

  In the above-described apparatus of the present invention, when the drowsiness occurrence predicting means predicts the occurrence of drowsiness as described above, the result is output by the drowsiness occurrence prediction result output means, and the driver of the vehicle or the moving body and / or the like. May be notified to the occupant. The output of the prediction result may be made by voice, mechanical vibration, light stimulation, or the like. In addition, the prediction result is recorded together with the detected physiological condition of the driver and / or the motion condition of the vehicle or the moving body, or the calculated physiological condition feature value and / or time-series data of the vehicle motion feature value. It may be possible to use it for the purpose.

  As described above, in the device of the present invention, not only the physiological state feature amount obtained from the driver's physiological state, but also the vehicle motion feature amount obtained from the motion state of the vehicle or the like that is changed by the driving operation of the driver. Is also used as a variable to predict drowsiness. Therefore, when there is a change in the driving operation of the driver according to the change in the traveling environment of the vehicle or the moving body, the change is reflected in the drowsiness prediction process through the vehicle motion feature amount. Therefore, there is a change in the physiological state of the driver due to the driving operation of the vehicle or the moving body, and even if the influence is included in the physiological state feature amount, the presence or absence of the possibility of drowsiness is accurately determined. It is expected that judgment will be possible. Further, in the apparatus of the present invention, the configuration for predicting the drowsiness of the drowsiness occurrence predicting means is prepared by machine learning. Therefore, by repeating the learning in a timely manner, the prediction accuracy of the drowsiness occurrence can be improved. It is also possible to improve, for example, prepare learning data according to the driver's age, driving experience period, etc., execute machine learning for drowsiness prediction, and predict drowsiness generation more suitable for driver conditions. You may be able to.

  Still further, it should be noted that in the device of the present invention, the drowsiness level reaches the predetermined level before significant or strong drowsiness occurs, that is, before the drowsiness level reaches the predetermined level. With reference to the physiological condition feature value and the vehicle motion feature value, the signs that increase due to the danger can be caught, and the driver or the occupant can be notified or transmitted of the possibility of drowsiness with sufficient time. Be composed. Therefore, it is possible to take measures such as evacuating the vehicle or the like from the traveling road before the driver becomes drowsy while driving the vehicle or the moving body, and thus improvement in driving safety of the vehicle or the like is expected. .

  In the drowsiness occurrence predicting means of the device of the present invention, the drowsiness level of the driver temporarily or transiently reaches the predetermined level even if the drowsiness occurrence is not actually predicted. It can happen that you enter a state where you are going to. Therefore, after the feature amount state reaches a state where the drowsiness level reaches the predetermined level over a predetermined time, or the feature amount state reaches the predetermined level during a predetermined time. It may be so arranged that the number of times that the user enters the state to be sleepy exceeds a predetermined number of times, and then a prediction that drowsiness occurs will be made.

  Thus, according to the configuration of the present invention described above, an apparatus that predicts the occurrence of drowsiness of a driver by using an index value or the like obtained from the eyelid opening / closing movement or other physiological states of a driver while driving a vehicle or the like is provided. In this case, the occurrence of drowsiness is monitored using the vehicle motion characteristic amount obtained from the motion state of the vehicle or the moving body together with the physiological state characteristic amount obtained from the driver's physiological state. As a result, in the device of the present invention, erroneous detection or oversight of the occurrence of drowsiness due to the influence of the driver's opening / closing movement of the driver such as a vehicle or a change in driving operation in other physiological conditions is eliminated, and more accurate drowsiness is obtained. It is expected that the occurrence can be detected or monitored. Further, the device of the present invention uses the drowsiness occurrence predicting means to detect the drowsiness sign in the physiological condition in consideration of the driver's driving operation from the physiological condition feature value and the vehicle motion feature value through machine learning. Since it is configured to be able to catch and inform in advance, it becomes possible for the driver of the vehicle etc. to take appropriate measures or driving actions before actually feeling strong drowsiness, and the driving safety of the vehicle etc. Is expected to be further enhanced.

  Other objects and advantages of the present invention will become apparent from the following description of preferred embodiments of the present invention.

FIG. 1A is a diagram schematically illustrating a configuration of a sleepiness occurrence prediction device according to the present embodiment. FIG. 1B is a diagram showing the operation of the drowsiness occurrence prediction device of the present invention in the form of a flowchart. FIG. 2A is a schematic diagram of a subject's face when an image of eyes and eyelids is captured using a camera image in the drowsiness occurrence prediction device of the present embodiment. FIG. 2B shows an example of a temporal change of the eyelid opening obtained from the eyelid image and an example of a temporal change of the eyelid opening / closing state index value obtained by binarizing the eyelid opening. FIG. 2C schematically shows an arrangement of electrodes attached to the face of the subject when the degree of eyelid opening is detected by an electrooculogram signal in the drowsiness occurrence prediction device of the present embodiment. FIG. 2D is a diagram illustrating a method of calculating a feature value (eyelid opening / closing feature value) from a measured value (eye opening time or eye closing time). 3A and 3B are examples of time series data of the eye opening time and the eye closing time measured in the time series data of the eyelid opening / closing state index value, respectively. FIGS. 4A to 4C are examples of time-series data of the ratio of the eye closing time, the number of blinks, and the variance of the blink interval, which are preferably used as the physiological state features in the drowsiness occurrence prediction device of the present embodiment. is there. FIGS. 5A to 5C are time-series data of a steering angle, a throttle opening (accelerator opening), and a vehicle speed referred to as a motion state of a vehicle or a moving body in the drowsiness occurrence prediction device of the present embodiment. This is an example. FIGS. 6A to 6C are time series of variance of steering wheel turning angle, variance of accelerator depression amount change, and variance of acceleration, which are preferably used as vehicle motion feature amounts in the drowsiness occurrence prediction device of the present embodiment. It is an example of data. FIG. 7 is a diagram schematically illustrating a temporal change of the drowsiness level. In the drowsiness occurrence prediction device of the present embodiment, the time T at which the drowsiness occurrence is predicted and the time T at which the drowsiness is predicted to occur (the drowsiness level). Is expected to reach the predetermined level A) S. FIG. 8 is a flowchart illustrating a process of learning a classifier parameter group and a discriminant function for use in a drowsiness occurrence prediction process in the drowsiness occurrence prediction processing unit of the drowsiness occurrence prediction device of the present embodiment. FIG. FIG. 9A is a diagram schematically illustrating a group of supervised learning data used for machine learning of the drowsiness occurrence prediction means in the drowsiness occurrence prediction device of the present embodiment. FIG. 9B is a diagram illustrating a process of preparing a feature amount vector from a physiological state feature amount and a vehicle motion feature amount. FIG. 9C is a diagram for explaining the relationship between the time of calculation of the feature amount vector and the time of drowsiness occurrence on the time axis. FIG. 9D is a diagram schematically showing the distribution state of the feature vector of the teacher data in the feature space, and the distribution of the feature vector in the time domain L of FIG. 9C. An identification function that defines an identification area あ る, which is an area, is described. FIG. 10A is a diagram schematically illustrating a time-series path in which the feature amount vector of the verification data in the feature amount space of FIG. 9D is included. The sleepiness prediction time Ti, which is the time at which the sleepiness is predicted, is explained. FIGS. 10B to 10E show examples of the relationship between the drowsiness occurrence time point S and the drowsiness prediction time point Ti in the verification data on the time axis. (B) shows the case where the drowsiness occurrence time S enters the time range L after the elapse of the time range K from the drowsiness prediction time Ti, and (C) and (D) show the time range K from the drowsiness prediction time Ti. (E) is a case where the drowsiness predicted time Ti does not appear before the drowsiness occurrence time S.

DESCRIPTION OF SYMBOLS 1 ... Subject 2 ... Eye of a subject 3 ... Camera 4 ... Signal processing apparatus

  The present invention will be described in detail with respect to some preferred embodiments with reference to the accompanying drawings. In the drawings, the same reference numerals indicate the same parts.

In the drowsiness occurrence prediction device of the present embodiment, any physiological state correlated with the degree of drowsiness of a person is measured and used for predicting the occurrence of drowsiness. In this case, an example will be described in which the occurrence of drowsiness is predicted using the "eyelid opening / closing state" as the "physiological state" and the "eyelid opening / closing characteristic amount" as the physiological state feature amount. Those skilled in the art, even when predicting the occurrence of drowsiness using physiological states other than the "open / closed state of the eyelids", for example, a respiratory state, a heartbeat state, a pulse state, an electroencephalogram, etc. It should be understood that the calculation method of the state feature amount is appropriately selected, and the characteristic configuration of the present invention can be similarly applied and belongs to the scope of the present invention.

  In short, the drowsiness occurrence prediction device according to the present embodiment is a device mounted on a vehicle or the like in order to predict occurrence of drowsiness of a driver (test subject) of a vehicle or a moving object (vehicle or the like), When using the eyelid opening / closing feature amount as the physiological state feature amount, similarly to the devices described in Patent Documents 1 and 3 to 5, the driver's eyelid opening / closing motion is measured, and the measured eyelid opening / closing motion is measured. Along with extracting eyelid opening / closing features correlated with the drowsiness of the subject from the time-series data, the driving state of the vehicle or the like in which the driver's driving operation is reflected is measured, and the measured driving state of the vehicle or the like is measured. The vehicle motion feature amount correlated with the subject's eyelid opening / closing feature amount is extracted from the time series data of the subject, and the feature amount state determined from the eyelid opening / closing feature amount and the vehicle movement feature amount is arbitrarily changed from the current time. Drowsiness of the driver within the predicted time range after the set allowance time has elapsed When the bell becomes the feature amount state is expected to reach a predetermined level is a device for predicting in advance the estimated time drowsiness regional after a margin time domain is generated.

  Referring to FIG. 1 (A), in the basic configuration of the drowsiness occurrence prediction device of the present embodiment, first, the eyes and eyelids 2 of a driver 1 as a subject are imaged by a camera 3 or the like. An imaging device is arranged so as to be included in the field of view of the device, and an image signal obtained by the imaging device is taken into an image processing unit of the signal processing device 4. As will be described later, the opening and closing movements of the driver 1's eyes and eyelids 2 may be detected by an electro-oculogram signal. In this case, a signal from an electrode attached to the driver's face is converted to a signal processing device. Given to 4. Then, the image processing unit constructs an image including the image of the driver's eyes and the eyelids 2 from the image signal transmitted from the imaging device, and the eyelid data processing unit Using the image of the eyelid 2, the detection of the eyelid opening / closing state and the calculation of the eyelid opening / closing feature amount are executed. Further, in the device of the present embodiment, as described above, the vehicle motion feature amount that changes according to the driver's driving operation is used for prediction of drowsiness. As the quantity representing the state, a steering angle δ detected by a steering angle sensor, an accelerator opening detected by an accelerator opening sensor (accelerator pedal sensor), an accelerator pedal depression amount or a throttle opening θa, a vehicle speed sensor or a wheel speed sensor Is input to a vehicle data processing unit provided in the signal processing device 4, and the vehicle data processing unit is configured to output the data such as the vehicle speed V detected by the acceleration sensor or the acceleration a detected by the acceleration sensor. The vehicle motion feature amount described in detail is calculated. The eyelid opening / closing feature amount and the vehicle motion feature amount thus calculated are given to the drowsiness occurrence prediction processing unit, and the drowsiness occurrence prediction processing unit refers to the eyelid opening / closing feature amount and the vehicle motion feature amount, and will be described later. Using the identification parameters determined by machine learning and stored in the data memory, whether or not the driver experiences drowsiness within a predetermined time range (predicted time range) after a predetermined time (allowance time range) has elapsed Monitor whether it is. Then, when the drowsiness prediction processing unit predicts that the driver will be drowsy, the result is transmitted to the result output unit. For example, the sound (buzzer, warning sound, etc.) of the speaker 5 or the like is output. The driver or the occupant is notified or notified that the drowsiness has been predicted through a vibrator that gives a mechanical stimulus to the driver, a light source that gives a light stimulus (warning lamp, display, or the like). The signal processing device 4 including the image processing unit, the eyelid data processing unit, the vehicle data processing unit, the drowsiness generation processing unit, the result output unit, the data memory, and the like may be typically a computer device, and may be a normal mode. A CPU, a storage device, and an input / output device (I / O) interconnected by a bidirectional common bus (not shown) are provided, and each unit of the device operates by executing a program in the CPU. Will be achieved.

Overview of Operation of Apparatus Referring to FIG. 1 (B), in the drowsiness occurrence prediction processing by the apparatus of the present embodiment, during driving of the vehicle, first, according to a program, the eyelid of the subject (driver) is placed on the eyelid. Acquisition of an index value representing the open / closed state is sequentially executed (step 1), and calculation of “eyelid opening / closing feature amount” correlated with the degree of drowsiness is sequentially executed from the time-series data of the index value (step 1). Step 2). At the same time, the acquisition of index values representing the motion state of the vehicle and the like related to the driving operation of the driver is sequentially executed (step 3), and the time series data of the index values is used to calculate the movement of the driver's eyelids. Calculation of correlated "vehicle motion feature values" is sequentially performed (step 4). Thereafter, it is determined whether or not the feature amount vector composed of the eyelid opening / closing feature amount and the vehicle movement feature amount is within the identification area defined through machine learning described later in the feature amount space ( Step 5) Predicting that the drowsiness level reaches the predetermined level in the prediction time range after the lapse of the predetermined allowance time range, ie, that drowsiness occurs when the feature amount vector enters the identification area. A determination is made (step 6), and notification or transmission of prediction of sleepiness occurrence by the result output unit is executed (step 7). Hereinafter, each of the series of processes will be described in detail. In the following, a “subject” is a driver of a vehicle or the like unless otherwise specified.

Acquisition of eyelid open / closed state index value (Step 1)
In the acquisition of an index value (eyelid opening / closing state index value) representing the eyelid opening / closing state of the subject in the apparatus of the present embodiment, as one aspect, as described above, an imaging device such as a camera is used. When the eyelid opening / closing motion is detected in the sequentially photographed eye and eyelid images of the subject, and the detected data of the eyelid opening / closing motion indicates that the eyelid is open and closed. Sequence data is prepared. In that case, first, an imaging device such as a camera may be installed at an arbitrary location such that the field of view includes the eyes and eyelids of the subject, as schematically illustrated in FIG. . For example, when determining the drowsiness of the driver of the vehicle as the subject, the installation position of the imaging device such as a camera is determined by the eyes of the subject, such as on the dashboard of the vehicle, the steering wheel, and the ceiling. Any location can be used as long as it can capture the eyelids and eyelids. In addition, an imaging device such as a camera may be attached to a wearable object such as glasses or a hat of the subject. Then, an imaging device such as a camera sequentially images the eyes and eyelids of the subject, and images of the eyes and eyelids are sequentially formed from the image signals.

  When a sequential image or a continuous image of the subject's eyes and eyelids is obtained, the positions of the upper and lower eyelids are sequentially detected in the image, and the distance between them is measured, and FIG. 2 (B) Time series data of the eyelid opening (the distance between the upper eyelid and the lower eyelid) as shown in the upper row is prepared. The detection of the positions of the upper and lower eyelids in the image may be achieved by an arbitrary image processing method based on the brightness and hue characteristics of the upper and lower eyelid images and the eyeball image. Then, whether or not the obtained time series data of the eyelid opening exceeds the threshold value set at the boundary between the open state and the closed state of the eyelid between the maximum value and the minimum value of the eyelid opening degree Is binarized into an open state and a closed state, and an eyelid open / closed state index value which is an index value indicating whether the eyelid is open or closed as illustrated in the lower part of FIG. Time series data is prepared.

  It should be understood that the eyelid opening described above may be measured in any manner, and such cases also fall within the scope of the present invention. As another aspect, for example, as already mentioned, the eyelid opening can also be measured by the level of an electro-oculogram signal (voltage change due to rotation of the eyeball due to blinking) of the subject. In this case, as schematically illustrated in FIG. 2C, for example, electrodes are attached to a plurality of regions such as the upper and lower sides of the subject's eyes, and the voltage between the electrodes is a signal (an electro-optical signal). ) And transmitted to the signal processing device. In addition, the electrode may be attached to a headwear such as glasses, goggles, and a helmet, or may be incorporated in a decorative article attached to the skin such as a tattoo. In the case of an electro-oculography signal, time series data similar to that shown in FIG. 2B is obtained for eyelid movement. By binarizing the data, the time series data of the eyelid opening / closing state index value is obtained. Prepared.

  In the processing of the apparatus according to the present embodiment, the eyelid opening / closing feature amount is calculated by further performing a statistical amount calculation process related to the eyelid opening / closing state from the time-series data of the eyelid opening / closing state index value. The time series data of the index value is stored in a data memory in the signal processing device 4.

Calculation of eyelid opening / closing feature (Step 2)
When the time series data of the eyelid opening / closing state index value is obtained, an “eyelid opening / closing feature amount” that is an amount correlated with the degree of sleepiness of the subject is extracted from the time series data of the eyelid opening / closing state index value. It is possible to find out what amount can be used as the “eyelid opening / closing feature amount” by an experiment or the like. In one aspect of the present embodiment, the “eyelid opening / closing feature amount” sequentially changes the transition time interval of the eyelid opening / closing state in the time series data of the eyelid opening / closing state index value over a predetermined period. It may be measured and calculated by a statistical calculation process of the measured transition time interval. Hereinafter, an example of the process of calculating the “eyelid opening / closing feature amount” will be described.

(I) Measurement of Transition Time Interval of Eyelid Opening / Closing State In the process of calculating the eyelid opening / closing feature amount, first, the eyelid opening / closing state is stored on the time series data of the eyelid opening / closing state index value stored in the data memory. The time interval of the transition between the state and the closed state is measured. The time interval of the transition between the open state and the closed state of the eyelid (hereinafter, referred to as “eyelid open / closed state transition time interval”) is, as described in the “Summary of the Invention”, the state in which the eyelid is in the open state. It is the interval at the time when the transition from the closed state to the closed state or from the closed state to the open state occurs. Specifically, for example, (1) eye opening time (the length of time the eyelids are continuously open), The time interval from the transition from the closed state of the eyelids to the open state to the transition from the open state of the eyelids to the closed state (FIG. 2B, lower row, to), (2) eye closing time (the eyelids are continuously closed Length of time), that is, a time interval from the transition from the open state of the eyelids to the closed state to the transition from the closed state of the eyelids to the open state (FIG. 2B, lower stage, tc). These eyelid opening / closing state transition time intervals are provided with time values at transition points after each time interval, and are configured as time-series data. FIGS. 3A and 3B show the eye opening time and the eye closing time measured based on the eyelid opening / closing state index value obtained from the temporal change of the eyelid opening obtained from the eye and eyelid images, respectively. 4 shows an example of series data.

(Ii) Calculation of Eyelid Opening / Closing Feature Amount As described above, when the eyelid opening / closing state transition time interval is measured, an arbitrarily set unit time (computation unit time) retroactively along the time axis from the current time point or the judgment time point The eyelid opening / closing feature amount is calculated as the physiological state feature amount using the data of the eyelid opening / closing state transition time interval in the inside. As shown in FIG. 2 (D), the eyelid opening / closing feature amount is typically included in the eyelid opening / closing included in the calculation unit time every measurement time of the eyelid opening / closing state index value, that is, every measurement unit time. Calculated using the value of the state transition time interval, and time-series data of the eyelid opening / closing feature amount may be prepared (that is, in this case, as shown in FIG. The calculation unit time used shifts momentarily by the measurement unit time.) Note that the eyelid opening / closing feature amount may be calculated for each calculation unit time (in that case, the calculation unit times do not overlap).

Specifically, it has been found that the following amounts are advantageously used as the eyelid opening / closing feature amount.
(1) The ratio of the eye closing time tc in the calculation unit time [eye closing time ratio] (FIG. 4A),
(2) Number of occurrences of blinking (number of times of occurrence of tc) [number of times of blinking] (FIG. 4B) in operation unit time
(3) Blink interval in calculation unit time, that is, dispersion of eye opening time to [variation of blink interval] (FIG. 4C)
Here, the operation unit time may be arbitrarily set according to adaptation, for example, 1 minute (the measurement unit time is, for example, one frame length of video image (eg, 33 ms) to several seconds, depending on adaptation). It may be set arbitrarily.). Further, as the eyelid opening / closing feature quantity, statistics of other eyelid opening / closing state transition time intervals (standard deviation value, variance, and the like) such as eye closing time, its standard deviation value, its variance, eye opening time, its standard deviation value, its variance, etc. Average, median, etc.).

Acquisition of vehicle motion index value (Step 3)
As described above, in the apparatus according to the present embodiment, an index value (vehicle motion index value) representing the motion state of the vehicle or the like caused by the driver's driving operation during driving of the vehicle or the like at the same time as the measurement of the eyelid opening / closing state. Is also measured. In this regard, since the driving action by the driver is mainly performed on the upper body (arms) and the lower body (legs), preferably, at least one type of operation performed by the arm and one operation performed by the legs are measured. . For example, for an operation performed with the arm, the steering angle associated with the steering wheel operation may be measured, and for the operation performed with the legs, the accelerator opening or the throttle opening associated with the accelerator operation may be measured. Further, the vehicle motion can be classified into three types of “run, turn, stop”. Since there is a correlation between the respective physical quantities, preferably, any one type of data regarding the above three types, for example, the vehicle speed is It may be measured. Thus, the vehicle motion index value includes, for example, steering angle data (FIG. 5A), throttle opening (or accelerator opening) data (FIG. 5B), vehicle speed data (FIG. 5C), and the like. It may be. Further, as the vehicle motion index value, a value representing a motion state of the vehicle or the like that changes according to the driving behavior of the driver, such as an acceleration, a steering angular velocity, a yaw rate, and a lateral acceleration, may be selected.

  Thus, the obtained time series data of the vehicle motion index value is used for calculating the vehicle motion feature amount by statistical calculation processing as described below, and thus is the same as the time series data of the eyelid opening / closing state index value. Is stored in the data memory in the signal processing device 4.

Calculation of vehicle motion features (Step 4)
When the time-series data of the vehicle motion index value is obtained as described above, a “vehicle motion feature amount” that is an amount correlated with the eyelid opening / closing state of the subject is calculated from the data. It is possible to find out what amount can be used as the “vehicle motion feature amount” by an experiment or the like. Specifically, the amount used as the vehicle motion feature amount may be, for example, the following amount.
(1) Dispersion of steering angular velocity in calculation unit time [Dispersion of change in steering wheel turning angle (change amount from zero position)] (FIG. 6 (A))
(2) Dispersion of time derivative of accelerator opening in unit time of operation [variance of change in accelerator depression amount] (FIG. 6 (B))
(3) Dispersion of time derivative of vehicle speed in calculation unit time [Dispersion of acceleration] (FIG. 6 (C))
Note that, similarly to the case of the eyelid opening / closing feature amount, as illustrated in FIG. 2D, typically, the vehicle movement feature amount is obtained at each measurement time of the vehicle movement index value, that is, the measurement unit time. Alternatively, the calculation may be performed using the vehicle motion index value included in the calculation unit time and prepared as time-series data (that is, in this case, as shown in FIG. The calculation unit time used shifts momentarily by the measurement unit time.) Note that the vehicle motion feature may also be calculated for each calculation unit time (in that case, the calculation unit times do not overlap). The calculation unit time may be arbitrarily set by adaptation, for example, 1 minute (the measurement unit time may be, for example, one frame length of a video image (eg, 33 ms) according to the eyelid opening / closing feature amount). It may be set arbitrarily by adaptation, such as several seconds.) Further, as the vehicle motion feature amount, other amounts calculated from each vehicle motion index value, for example, the accelerator opening, its differential value, its second differential value, its third differential value, its steering angle, its angular velocity, The angular acceleration, the vehicle speed, the acceleration, the yaw rate, the yaw acceleration, the position of the own vehicle in the traveling lane, its differential value, and the like may be used.

Sleepiness occurrence prediction processing (steps 5 and 6)
When the eyelid opening / closing feature amount and the vehicle motion feature amount are calculated as described above, a process of predicting the occurrence of drowsiness of the driver is executed using the feature amounts. Regarding the prediction of the occurrence of drowsiness, referring to FIG. 7, generally, when a strong drowsiness occurs in a human, the degree of drowsiness is expressed as a drowsiness level indicating the strength of drowsiness, as shown in FIG. The level gradually increases as time elapses from the level 0, which is not in the state (not shown). The mode of increase in the level shown is an example for explanation, and is not necessarily the same as the example shown in the figure. At some point S, it reaches a certain strong level (A). Therefore, in the present embodiment, the eyelid opening / closing feature in a time range (F) before a time (T) some time before the time S when the drowsiness of the driver reaches the predetermined level (A). With reference to the amount and the vehicle motion feature amount, a sign that the drowsiness level increases to the predetermined level (A) is captured, and a time point before the drowsiness occurrence time point S at which drowsiness is predicted to reach the predetermined level (A) ( At prediction time T, an attempt is made to predict that drowsiness will occur at a future time point (the drowsiness reaches a predetermined level (A)). In this regard, at a certain time point (current time point), it is predicted that drowsiness will occur at a future time point S at which the remaining time G has elapsed, that is, the drowsiness time G before the drowsiness occurrence time point S It is somewhat difficult to pinpoint the drowsiness occurrence time S at the time point T. Therefore, in the drowsiness occurrence prediction process in the apparatus of the present embodiment, the eyelid opening / closing feature amount and the vehicle motion feature in a time range (feature amount collection time range) of a predetermined time width F (first time width). The amount is referred to from time to time, and a state determined by the eyelid opening / closing characteristic amount and the vehicle motion characteristic amount within the characteristic amount collection time range (feature amount state. The state is a vector represented by a plurality of numerical values. ) Is a predetermined level of drowsiness during a predicted time range L (third time width) after a lapse of a spare time range K (second time width) from the time point (end point of the feature amount collection time range). When the vehicle is in the state to reach A, a process of predicting that drowsiness will occur in the prediction time range L after the allowance time range K has elapsed is executed.

  The time width K of the spare time range and the time width L of the prediction time range may be appropriately set by a designer or a user of the apparatus. The time width K of the allowance time range should be determined in consideration of the time allowance up to the time when sleepiness is predicted to actually occur when notifying the driver or the like of the occurrence of sleepiness. If the time width K is too short, even if the driver is informed of the prediction of drowsiness, there is enough time to cope with it (until the vehicle or the moving body can be safely stopped (e.g., to a parking area). The time to take measures to resolve drowsiness (drinking beverages, chewing gum, playing music, etc.) is reduced, and the effect of informing in advance is reduced. On the other hand, if the time width K is too long, the reliability of the prediction will be reduced, or it will be too early for the driver to take measures against drowsiness. On the other hand, the time width L of the prediction time range is appropriately set in consideration of the drowsiness prediction performance of the device. If the time width L is too short, the prediction accuracy relatively decreases, and the reliability of the prediction by the driver decreases. Even if the time width L is too long, the drowsiness occurrence prediction range is too wide. , The expectation or confidence in the prediction by the driver may be reduced. In the embodiment, the predicted time T of the drowsiness occurrence, that is, the time when the prediction is obtained, is 60 to 900 seconds before the drowsiness occurrence time S, that is, the future time when the drowsiness is predicted to occur. Thus, the time width K may be set to about 60 seconds and the time width L may be set to about 840 seconds.

  In order to execute the drowsiness occurrence prediction process in the above-described manner, in the present embodiment, as described in more detail later, the eyelid opening / closing feature amount and the vehicle The time width F of the feature amount collection time range referring to the motion feature amount and the predetermined level A between the sleepiness level and the predicted time range L after a lapse of the allowance time range K from a certain time point (predicted time point) T. When it is predicted that the arrival will be reached, an identification function that defines a range (identification region) to be taken by the feature amount state at the prediction time T is determined in advance. In the drowsiness occurrence prediction process, when the eyelid opening / closing feature amount and the vehicle movement feature amount are calculated in Steps 2 and 4, respectively, the feature amount state (feature amount vector) determined from those feature amounts is calculated. ) Is included in the identification area (step 5), and when the feature amount state is in the identification area, drowsiness occurs during the estimated time range L after the lapse of the allowance time range K. Then, a prediction determination is made (step 6). In the above-described determination, the entry of the feature amount state into the identification area may be transient or temporary, or may be caused by a calculation error. Over a cycle (when the number of ingresses n exceeds a predetermined number of times Nt), or when the period T during which the feature amount state enters the identification area exceeds a predetermined period ΔTt (at that time, Alternatively, the prediction determination that drowsiness occurs during the prediction time range L after the allowance time range K has elapsed (from the time when the feature amount state first enters the identification area). (Step 6).

Sleepiness occurrence prediction notification process (step 7)
When the drowsiness occurrence prediction processing unit makes a prediction determination of drowsiness occurrence, the result of the prediction is transmitted or reported from the result output unit to a driver such as a vehicle and / or other occupants. As described above, specifically, the predicted result is a sound (buzzer, warning sound, etc.) by the speaker 5, a vibrator (vibrator, etc.) for giving a mechanical stimulus to the driver, a light source (warning light) for giving a light stimulus. , Display, etc.) or the like. The prediction result is an arbitrary data storage together with the detected physiological state of the driver and / or the motion state of the vehicle or the moving object, or the calculated physiological state feature amount and / or time series data of the vehicle movement feature amount. It may be recorded on a medium and used for various purposes.

Preparation by machine learning of the drowsiness occurrence prediction processing unit As described above, in the apparatus of the present embodiment, the time width F and the characteristic of the collection time range of the feature amount used in the drowsiness occurrence prediction processing of the drowsiness occurrence prediction processing unit The discriminant function す る that defines the discrimination area in the quantity space is determined by machine learning, and is calculated every moment using the determined time width F of the feature amount collection time range and the discrimination function Ψ. The drowsiness occurrence prediction processing unit is prepared as a “discriminator” for discriminating whether or not the feature amount state formed by the feature amount is within the identification area. Referring to FIG. 8, machine learning for the preparation of the discriminator may be performed as follows.

Preparation of supervised learning data (Step 10)
In machine learning for determining parameters set in the classifier (the time width F of the feature amount collection time range and the parameter for determining the classification function Ψ), first, at least two, preferably many teachers are used. The attached learning data is prepared (FIG. 9A). In preparing the learning data, first, as described above with reference to FIG. 1B, the driver's eyelid opening / closing state and vehicle movement state during driving of the vehicle or the like are first described. Measurement (acquisition of the eyelid opening / closing state index value and the vehicle motion index value) and creation of time-series data of the eyelid opening / closing feature amount and the vehicle movement feature amount are performed. Then, labeling of the driver's drowsiness level at the time of acquiring the time series data is performed on the time series data of the eyelid opening / closing feature amount and the vehicle movement feature amount (or the eyelid opening / closing state index value and the vehicle movement index value). Will be Here, as described above, the “drowsiness level” is a degree of drowsiness of a person that expresses a degree from a state in which a person does not feel drowsiness at all to a state in which a person feels strong drowsiness in a plurality of stages. And can be determined by observation of the expression of the subject (by the evaluator), subjective evaluation by the subject himself, brain wave measurement, and the like. The labeling of the drowsiness level in the learning data may be performed by any of the above methods, for example, the evaluator observes a video image of the facial expression of the driver while driving, and opens and closes the eyelids. It may be performed by a method of determining the drowsiness level of the driver in the video image at each time point in the time series data of the feature amount and the vehicle motion feature amount (see Non-Patent Document 1). Then, the time point S when the drowsiness level reaches the predetermined level A is labeled in the time-series data of the eyelid opening / closing feature amount and the vehicle movement feature amount (drowsiness occurrence label). Here, as described above, the predetermined level A may be appropriately set by a designer or a user of the device in consideration of driving safety of the vehicle or the moving body. Thus, a set of time-series data (D1, D2, D3...) Of eyelid opening / closing feature amounts and vehicle movement feature amounts labeled with a plurality of sets of drowsiness occurrence times are prepared as supervised learning data.

  It should be noted that in the following classifier trial production and verification processing, a part of the learning data is used as the teacher data and another part of the learning data is used as the verification data. May be used in turn for teacher data and verification data.

Preparation of discriminator (Steps 11 to 17)
When the learning data is prepared as described above, internal parameters for determining the characteristics of the time width F and the discriminant function Ψ in the feature amount collection time range (hereinafter, referred to as a group of internal parameters of the classifier including the time width F). Is appropriately set, the learning data is used, and the distribution range of the feature amount state to be identified in the feature amount space (the predetermined level of drowsiness during the predicted time range L after the allowance time range K has elapsed from the current time) A classifier is created by determining a discriminant function Ψ which defines a feature amount state distribution area that reaches A. As a method of creating a classifier, for example, an arbitrary method that can define a distribution region of a specific feature amount group in a feature amount space as an identification region, such as a support vector machine method and a k-NN method, can be used. . However, it is difficult to determine a priori an internal parameter group of an appropriate classifier that realizes a classifier having good classification performance, that is, capable of accurately predicting the occurrence of drowsiness. In other words, while changing the internal parameters of the classifier using the learning data, trial production and verification of the classifier are repeated, and an appropriate internal parameter group of the classifier is experimentally found, and the classifier is Generate the discriminant function 使用 used for. Specifically, the discriminator may be prepared according to the following procedure.

(I) Setting of internal parameters of the discriminator (step 11)
In a specific process of preparing the classifier, first, an appropriate group of internal parameters of the classifier (a plurality of parameters usually include the time width F) are set. In this regard, the feature value state determined by the feature value within the time width F of the feature value collection time range is, as described above, the eyelid opening / closing feature value at each calculation time point of the feature value within the feature value collection time range. And a vector having the vehicle motion feature amount as a variable. That is, as schematically illustrated in FIG. 9B, the dimension number N of the feature amount state (vector) at a certain point in time is nx (the number of eyelid opening / closing feature amounts and the vehicle motion feature amount) × nf ( (The number of calculation points within the time width F). That is, when the time width F of the feature amount collection time range is changed, the dimension number N of the feature amount vector is also changed. Here, if the time width F of the feature amount collection time range is too short, the number of data to be referred to decreases and the prediction performance decreases. On the other hand, if the time width F is too long, the calculation load increases and the prediction load increases. Therefore, it is preferable that the value be set to an appropriate range by a designer or a user. For example, the time width F may be appropriately set between 60 and 1200 seconds. The time width F is changed in step 16 described later, and the value actually used in the classifier is searched for in the course of the iterative process of trial production and verification of the classifier.

  More specifically, the internal parameters that determine the characteristics of the discriminant function are hyperparameters that determine characteristics such as the shape of the discriminant function in the feature space. As is known to those skilled in the art, for example, when a discriminator is created by a support vector machine method, a type of a kernel function for determining a discriminant function, a parameter for determining a shape of the kernel function, an order of the kernel function, etc. Is an internal parameter. The operation for generating the identification function is performed using a function or module prepared in an arbitrary programming language, and the types of the internal parameters of the identification function that are specifically set are the functions of the programming language used. Alternatively, it will be understood by those skilled in the art that the specific values of the internal parameters of the discriminant function are appropriately set according to the specifications, since they are determined by the specifications of the module. For example, when the API function sklearn.svm.SVC in the scikit-learn library of python 3 is used as a function of the programming language for generating the identification function, the internal parameters may be set as follows. : C ← {1,10,100,1000} [Used when selecting other than linear as the kernel. ]; Kernel ← ｛linear, rbf, sigmoid, poly｝ [type of kernel function]; gamma ← ｛0.01, 0.001, 0.0001｝ [coefficient of kernel function]; degree ← ｛2, 3, 4｝ [policy as kernel The order used when selecting]

(Ii) Prototype of classifier (Step 12)
When the group of internal parameters of the classifier is set as described above, a prototype of the classifier is performed by using a part of the learning data as teacher data. In this process, first, in a manner as illustrated in FIG. 9B, the feature amount vectors are configured in time series using the feature amounts of the teacher data. Then, the characteristic amount vector (x) at each time point of the time-series data as schematically illustrated in FIG. 9C is obtained as shown in FIG. 9D. It will be distributed in the feature space defined by the variables (x1,... Xi,...) Constituting the vector. By the way, referring to FIG. 9 (C), the time range indicated by the time width L, that is, the time range K corresponding to the spare time range from the drowsiness occurrence time point S, and the prediction time range from that time point Each time point in the time range [time range from the time point (SKL) to the time point (SK)] that is traced back by the time width L is the time width after the lapse of the allowance time range of the time width K from that time point. This is the time point at which the drowsiness level reaches A at time point S within the predicted time range of L. Then, it is considered that each feature amount vector in the time range given the time width L in FIG. 9C reflects a sign that the drowsiness level increases to A. Therefore, in the feature space shown in FIG. 9D, the area す る in which the distribution of the feature vector in the time domain L shown in FIG. A drowsiness level can reach A (a drowsiness occurs) within the predicted time range of the time width L after the elapse of the time range. When entering the area 予 測, it is predicted that the drowsiness level will reach A (the drowsiness will occur) within the predicted time range of the time width L after the allowance time range of the time width K has elapsed from that time. It is possible to do. Thus, in the classifier trial production process (step 12) of the present embodiment, the internal parameter group set in step 11 is used in the teacher data in accordance with the above-described arbitrary classifier creation method. A discrimination function 遡 that defines a region Γ in which a feature amount vector is distributed in a time domain that goes back a time width K from the drowsiness occurrence time S and goes back a time width L from that time point is generated.

(Iii) Verification of prototyped classifier (step 13)
Since the discrimination function ス テ ッ プ generated in step 12 does not always enable the prediction of drowsiness as expected in data other than the teacher data used when generating the discrimination function Ψ, the discrimination function Ψ Using learning data different from the teacher data used for the determination as the verification data, it is verified whether the expected drowsiness can be predicted as expected by the discriminant function 生成 generated in step 12. Here, the prediction of drowsiness occurrence as expected means that the feature vector obtained from the verification data moves in a time series in the feature space as schematically illustrated in FIG. In the process of entering, when the discrimination function Ψ enters from the outside of the discrimination area 画 defined by the discrimination function Ψ to the inside of the discrimination area Γ, the prediction time range of the time width L after the lapse of the time margin K from the time Ti If the sleepiness occurrence time point S in the verification data exists, in other words, the time point Ti is a time period that is further back by the time width L from the time point before the sleepiness occurrence time point S by the time width K in the verification data. This is the case where the area is within the area (FIG. 10B). In this case, the prediction is successful. On the other hand, after the elapse of the predicted time range of the time width L after the elapse of the allowance time area of the time width K from the time Ti (FIG. 10C), and before the elapse of the allowance time area of the time width K from the time Ti (FIG. 10). If the sleepiness occurrence time point S exists in (D)), and if the feature amount vector does not enter the identification area 前 before the sleepiness occurrence time point S (FIG. 10E), the prediction fails. Thus, in step 13, the above-described verification is performed using a plurality of verification data, and whether or not the prediction is successful is recorded, and the success rate of the classifier is calculated by the following equation. It may be recorded together with the internal parameters of the discriminator used (step 14).
Success rate = (number of data sets for which prediction was successful) / number of verified data sets)
The higher the success rate of such a classifier, the more accurately the classifier can determine that it is possible to predict the occurrence of drowsiness in the prediction time range after the allowance time range has elapsed.

(Iv) Repetition of trial production and verification of a classifier (steps 15 and 16)
The process of trial production and verification of the discriminator in steps 11 to 14 is repeated using different internal parameter groups, and the verification result is recorded together with the internal parameter group of the discriminator. As described above, in step 16, the internal parameters that determine the characteristics of the discriminant function Ψ are changed, including the time width F of the feature amount collection time range. When changing the internal parameter group, a certain number of candidates for the internal parameter group are prepared in advance by the designer or user of the apparatus, and a prototype of the classifier and verification are performed. Then, when all prepared internal parameter groups are used, the processing may be completed.

(V) Preparation of Classifier Upon completion of the above-described process of trial production and verification of the classifier, a drowsiness occurrence prediction processing unit in the apparatus according to the present embodiment based on the verification result (step 5 in FIG. 1B) The discriminator to be used in is selected. Specifically, the classifier with the highest success rate or the classifier with the success rate exceeding a predetermined target value can accurately predict the drowsiness occurrence in the prediction time range after the lapse of the allowance time range. The internal parameter group constituting these classifiers is determined as an internal parameter group used in the drowsiness occurrence prediction device, and the identification function た generated using them is used in the drowsiness occurrence prediction processing unit. The classification function の of the classifier used in the drowsiness occurrence prediction processing unit may be generated using the teacher data at the stage of trial production of the classifier, or may be a group of determined internal parameters. It may be generated using the teacher data and the verification data. When there are a plurality of available classifiers, the remaining time G (the time from the drowsiness occurrence prediction time Ti to the drowsiness occurrence time S) becomes the longest, or the remaining time G + the characteristic amount collection time. The discriminator having the longest range F may be selected and used in the drowsiness occurrence prediction processing unit. This is because the longer the remaining time G, the longer the time margin is given to the driver after the drowsiness is predicted, and the longer the feature amount collection time range F, the longer the feature value reference section. This is because the prediction performance is considered to be high.

  Thus, in the drowsiness occurrence predicting device according to the above-described embodiment, not only the index value obtained from the eyelid opening / closing motion of the driver of the vehicle or the like, but also the motion state of the vehicle or the like that changes due to the driving operation of the driver. With reference to the obtained index value, drowsiness occurrence is predicted, and according to this configuration, even if there is a change in the eyelid opening / closing movement caused by the driver's driving operation, that fact is considered. Since prediction of sleepiness occurrence is determined in the state, it is expected that erroneous detection or oversight of sleepiness occurrence due to a change in eyelid opening / closing movement caused by the driver's driving operation is reduced. In addition, the configuration of the apparatus according to the present embodiment is configured such that a drowsiness occurrence can be predicted by previously detecting a sign of occurrence of strong drowsiness from the eyelid opening / closing feature amount and the vehicle movement feature amount. And the like can take appropriate measures or driving actions before actually feeling strong drowsiness, and it is expected that the driving safety of a vehicle or the like can be further enhanced.

  Although the above description has been made in connection with the embodiments of the present invention, many modifications and changes are easily possible for those skilled in the art, and the present invention is limited to only the above-exemplified embodiments. It will be apparent that the invention is not limited and can be applied to various devices without departing from the concept of the invention.

Claims (1)

A device for predicting the occurrence of drowsiness of a driver while driving a vehicle or a moving body,
Means for detecting a physiological state of the driver;
Means for sequentially calculating the physiological state feature quantity from the time series data of the physiological state,
Means for detecting the motion state of the vehicle or the moving body that changes according to the driving operation of the driver,
Means for sequentially calculating a vehicle motion feature amount from time-series data of the motion state of the vehicle or the moving body,
Using the physiological state feature amount and the vehicle motion feature amount, the drowsiness occurrence prediction means for predicting in advance the occurrence of drowsiness of the driver, the first time width in the feature amount collection time range The feature amount state determined by the physiological state feature amount and the vehicle motion feature amount is the third time width after the elapse of the second time width margin time period from the last time point of the feature amount collection time range. When the drowsiness level of the driver reaches a predetermined level within the predicted time range, the predicted time after the lapse of the spare time range from the last time point of the feature value collection time range when the feature amount state that the driver's drowsiness level reaches a predetermined level Means for predicting drowsiness occurrence when the drowsiness level of the driver reaches a predetermined level within a range,
Sleepiness occurrence prediction result output means for outputting a sleepiness occurrence prediction result by the sleepiness occurrence prediction means,
The drowsiness occurrence prediction means performs the first learning by machine learning using a plurality of previously prepared time-series data of the physiological state feature amount and the vehicle motion feature amount labeled with a drowsiness level as supervised learning data. The drowsiness level in the first group of the learning data defined in the feature space defined by the physiological state feature and the vehicle motion feature in the time width of A second group of the learning data is placed in a distribution range of the feature amount state in a time range that is traced back by the third time width from a time point before the second time width before reaching the level. In the time range of the third time width after the elapse of the second time width from the point in time when the feature amount state of the learning data enters, the drowsiness level in the second group of the learning data is Reach the predetermined level Devices prepared as the time to enter the.

Images