JP5139470B2 - Sleepiness level estimation device and sleepiness level estimation method - Google Patents

Description

  The present invention relates to a sleepiness level estimation apparatus and a sleepiness level estimation method for estimating a sleepiness level of a measurement subject.

  An apparatus for detecting a specific type of blink of a driver of a vehicle as a measurement subject and determining the driver's arousal state from the detection result is disclosed in Patent Document 1 below. In this conventional apparatus, when specifying the type of blink, the time X2 from the start of blinking to the peak height is compared with the time X3 from the peak height to the end of blinking. And the relationship of X2 <X3 is disclosed as a general example of blinking. Also. In order to avoid false detection of blinking, as shown in FIG. 8, a value obtained by smoothing the waveform of the eye opening degree for a certain past time (that is, a value of the eye opening degree from which only a high frequency noise component is removed) is used. It is shown.

JP 2007-31824 A JP 2009-3644 A

  However, in the smoothing process in the conventional apparatus described above, the low-frequency noise component as shown in FIG. 8 is not removed, and sleepiness estimation using a waveform of the eye opening degree including such a noise component is performed. It is carried out. In other words, for example, when the direct sunlight changes suddenly, the eye opening / closing operation at that time may be mistakenly detected as blinking, and sleepiness may be estimated based on the false detection of blinking. It was. As described above, the conventional technique has not been able to sufficiently improve the accuracy of sleepiness estimation.

  Therefore, the present invention has been made to solve the above-described problem, and an object thereof is to provide a sleepiness level estimation device and a sleepiness level estimation method that can estimate sleepiness level with higher accuracy. .

  A sleepiness level estimation device according to the present invention is a sleepiness level estimation device that estimates a sleepiness level of a measurement target person, and is detected by a feature point detection unit that detects a feature point of the face of the measurement target person, and the feature point detection unit The eye opening degree calculation means for calculating the degree of eye opening indicating the degree of eye opening of the measurement subject based on the facial feature points, and the eye opening degree exceeds a predetermined threshold based on the eye opening degree calculated by the eye opening degree calculation means. Detecting an open eye state and a closed eye state in which the degree of eye opening is below a predetermined threshold, and calculating a time Tclose until the eye-open state is changed from the open-eye state and a time Topen until the eye-open state is changed from the closed-eye state Based on Tclose and Topen calculated by the closed eye opening determination unit and the closed eye opening determination unit, the eye opening / closing operation of the measurement subject satisfies a relationship of Tclose ≦ Topen In addition, a blink determination unit that determines that the eye opening / closing operation at that time is a blink, and a blink feature that calculates a blink feature amount related to the eye opening / closing operation at that time only when the blink determination unit determines that the eye is blinking An amount calculating unit; and an estimating unit configured to estimate a sleepiness level of the measurement subject based on the blink feature amount calculated by the blink feature amount calculating unit.

  In this sleepiness level estimation device, the eye opening state and the eye closing state based on the eye opening degree are detected by the eye closing eye opening determination means, and the time Tclose from the eye opening state to the eye closing state and the time from the eye closing state to the eye opening state. Topen is calculated. Then, when the relationship of Tclose ≦ Topen is satisfied by the blink determination means, it is determined that the eye opening / closing operation at that time is blinking. Furthermore, only the blink feature amount of the opening / closing operation when the blink determination unit determines that the blink is present is used for sleepiness level estimation by the estimation unit. In this way, since blink determination is performed based on Tclose and Topen based on the degree of eye opening, erroneous detection of blinking when estimating sleepiness level is significantly reduced. Therefore, in the sleepiness level estimation device according to the present invention, it is possible to estimate the sleepiness level with higher accuracy than the conventional technology that may erroneously detect blinking.

  A sleepiness level estimation method according to the present invention is a sleepiness level estimation method for estimating a sleepiness level of a measurement target person, which is detected in a feature point detection step for detecting a feature point of the measurement target person's face and a feature point detection step The eye opening degree calculation step for calculating the degree of eye opening indicating the degree of eye opening of the measurement subject based on the facial feature points, and the eye opening degree exceeds a predetermined threshold based on the eye opening degree calculated in the eye opening degree calculation step. Detecting an open eye state and a closed eye state in which the degree of eye opening is below a predetermined threshold, and calculating a time Tclose until the eye-open state is changed from the open-eye state and a time Topen until the eye-open state is changed from the closed-eye state Based on Tclose and Topen calculated in the closed eye opening determination step and the closed eye opening determination step, the opening / closing operation of the eye of the measurement subject is expressed as Tclose ≦ T a blink determination step for determining that the eye opening / closing operation at that time is a blink when the pen relationship is satisfied, and a blink feature relating to the eye opening / closing operation at that time only when it is determined to be a blink in the blink determination step A blink feature amount calculation step for calculating the amount, and an estimation step for estimating the sleepiness level of the measurement subject based on the blink feature amount calculated in the blink feature amount calculation step.

  In this drowsiness level estimation method, the eye-opening state and the eye-closed state based on the eye-opening degree are detected in the eye-opening eye determination step, and a time Tclose from the eye-opening state to the eye-closing state and a time Topen from the eye closing to the eye opening are calculated. In the blink determination step, when the relationship of Tclose ≦ Topen is satisfied, it is determined that the eye opening / closing operation at that time is blink. Furthermore, only the blink feature quantity of the opening / closing operation when it is determined that the blink is determined in the blink determination step is used for sleepiness level estimation in the estimation step. In this way, since blink determination is performed based on Tclose and Topen based on the degree of eye opening, erroneous detection of blinking when estimating sleepiness level is significantly reduced. Therefore, in the sleepiness level estimation method according to the present invention, the sleepiness level can be estimated with higher accuracy than the conventional technique that may erroneously detect blinking.

  ADVANTAGE OF THE INVENTION According to this invention, the sleepiness degree estimation apparatus and sleepiness degree estimation method which can estimate a sleepiness degree with higher precision are provided.

FIG. 1 is a system block diagram showing a sleepiness level estimation apparatus according to an embodiment of the present invention. FIG. 2 is a flowchart showing a procedure of a method for estimating sleepiness using the apparatus of FIG. FIG. 3 is a diagram showing image processing performed on the face image taken by the image sensor of FIG. FIG. 4 is an eye-opening degree-time graph showing threshold value processing for the degree of eye opening in the closed-eye opening determination unit of FIG. FIG. 5 is an eye-opening degree-time graph showing an eye-opening degree section determined by the erroneous detection section estimating unit in FIG. FIG. 6 is a graph showing the feature amount (ratio occupied by the closed eye state) calculated by the blink feature amount calculation unit of FIG. FIG. 7 is a probability density-feature amount graph showing the relationship between the sleepiness level and the blink feature amount used in the sleepiness level estimation unit of FIG. FIG. 8 is an eye-opening time-time graph showing the noise processing of the eye opening degree according to the prior art.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. In addition, the same code | symbol is attached | subjected about the same or equivalent element, and the description is abbreviate | omitted when description overlaps.

  FIG. 1 is a system block diagram showing a sleepiness level estimation device 10 according to an embodiment of the present invention. The sleepiness level estimation device 10 includes an arithmetic processing device (CPU, etc.) and a recording device (ROM, RAM, hard disk, etc.) in addition to the image sensor 11 as physical components. The sleepiness level estimation device 10 includes, as functional components, a face position detection unit 12, a face feature point detection unit (feature point detection unit) 13, an eye opening degree calculation unit (eye opening degree calculation unit) 14, and eyes closed / open eyes. A determination unit (closed eye opening determination unit) 15, an erroneous detection section estimation unit (blink determination unit) 16, a blink feature amount calculation unit (blink feature amount calculation unit) 17, and a sleepiness level estimation unit (estimation unit) 18 are provided.

  The image sensor 11 is a sensor directed to the face of the driver who is the subject of measurement of sleepiness, and takes an image of the face of the driver.

  The face position detection unit 12 is a part that detects the position of the face from the face image photographed by the image sensor 11 using Boosting, a neural network, or the like.

  The face feature point detection unit 13 is a part that detects, from a face image captured by the image sensor 11, a feature point of the measurement subject's face (eg, the corner of the eye, the head of the nose, the center of the nasal cavity, the mouth end) using a neural network or the like It is.

  Based on the feature points detected by the face feature point detection unit 13, the eye opening degree calculation unit 14 projects a plurality of curves on the edge image around the corners and corners of the feature points, and calculates the curved edge of the edges. This is a part for detecting the vertical eyelid position from the intensity, and further calculating the eye opening degree (value indicating the degree of eye opening) of the measurement subject from the difference in the Y coordinate of the vertical eyelid position.

  The eye-closed / open-eye determination unit 15 is a part that performs threshold processing of the eye-opening degree calculated by the eye-opening degree calculation unit 14 and determines the eye-opening state / eye-closing state. Specifically, the eye closing / opening determination unit 15 determines the eye opening timing when the eye opening degree exceeds a predetermined threshold based on the eye opening degree calculated by the eye opening degree calculation unit 14, and the eye opening degree satisfies the predetermined threshold value. It detects the timing of the closed eye in the lower state. Further, the closed / open eye determination unit 15 calculates a time Tclose from the eye opening to the eye closing and a time Topen from the eye closing to the eye opening using the detected eye opening timing and the eye closing timing.

  The false detection section estimation unit 16 is a part that analyzes the eye opening / closing operation (blink behavior) of the measurement subject and extracts the false detection section. Specifically, based on Tclose and Topen calculated by the closed / open eye determination unit 15, when the eye opening / closing operation of the measurement subject satisfies the relationship of Tclose ≦ Topen, the eye opening / closing operation at that time blinks. Is determined. However, when the relationship of Tclose ≦ Topen is not satisfied (that is, when the relationship of Tclose> Topen is satisfied), the section is determined as a false detection section.

  The blink feature amount calculation unit 17 determines the degree of opening of the eye opening / closing operation determined to be blinking only when the erroneous detection interval estimation unit 16 determines blinking (that is, other than the opening degree of the erroneous detection interval). This is a part that uses data to calculate a blink feature amount (for example, a time when the eyes are closed) having a correlation with sleepiness.

  The sleepiness level estimation unit 18 estimates the sleepiness level of the measurement target person based on the blink feature value calculated by the blink feature value calculation unit 17 using the relationship between the sleepiness level learned in advance and the blink feature value. Part.

  Next, a specific procedure of the method for estimating sleepiness level using the sleepiness level estimation device 10 will be described with reference to the flowchart of FIG.

  First, in step S01, a face image as shown in FIG. 3A is photographed by the image sensor 11, and the face position detection unit 12 uses a technique such as neural network or boosting with the photographed face image as an input. Used to scan the entire range of the image and identify the position of the face of the person being measured. Note that the face detection method is not limited to the neural network, and may be another method.

  Next, in step S02, the face feature point detection unit 13 uses a neural network or the like for the face existing range detected in step S01, as shown in FIG. Right eye, right eye corner, left eye head, left eye corner, nasal cavity center, left and right mouth edges). Note that the method for detecting facial feature points is not limited to a neural network, and other methods may be used.

  In subsequent step S03, the eye opening degree calculation unit 14 calculates the eye opening degree of each of the left and right eyes. In the following, only the method for calculating the degree of opening of the left eye is shown, but the degree of opening of the right eye is also calculated in the same way. In addition, about the calculation method of an eye opening degree, other well-known methods can also be used.

(1) First, an image (edge image) in which an edge is emphasized is created by applying a Sobel filter to a certain region including the corner of the eye and the head detected in step S02.
(2) Next, a plurality of curves (for example, Bezier curves) created from the corners of the eyes and the eyes detected in step S02 are projected onto the edge image (see FIG. 3C), and the edge strength (edge Image pixel value) is calculated.
(3) Then, from the plurality of curves in (2), the one having the strongest edge strength is selected as the vertical curve.
(4) Finally, as shown in FIG. 4, the degree of eye opening [pixel] is calculated from the difference in the Y coordinates of the midpoints of the upper and lower eyelid curves.

  Then, in step S04, the eye-closed / open-eye determination unit 15 performs an eye-closed state (a state in which the eyes are closed) in which the eye-opening degree is lower than the eye-opening threshold with respect to the eye-opening degree waveform obtained in step S03. An eye opening state (a state where the eyes are open) in which the degree of eye opening exceeds the threshold is determined by threshold processing. Note that states other than the closed eye state and the open eye state are determined to be intermediate states. Each threshold value may use a predetermined value, or may use other values.

  In step S04, the closed eye / open eye determination unit 15 performs a time Tclose (a time required for closing the eyes) until the eye is closed from the open state and a time Topen (the eye is changed from the closed state to the open state). Calculate the time it takes to open.

  In subsequent step S05, the erroneous detection interval estimation unit 16 estimates an interval in which blink detection is erroneously detected by the following method.

  As a first method, when an interval satisfying the relationship of Tclose ≦ Topen (see FIG. 5A) is observed, it is determined that the interval is a normal blink, while the relationship of Tclose ≦ Topen is satisfied. If a non-observed section (see FIG. 5B) is observed, it is determined that it does not apply to the blink characteristics of the measurement subject, and the section is estimated as a false detection section. The reason for making such a determination is that the inventors have obtained knowledge that the relationship of Tclose ≦ Topen is often established as a result of analyzing the characteristics of the driver's blink during driving.

  In the “Visual Information Processing Handbook” edited by the Visual Society of Japan, p45, chapter 1.8.2, “Type of blink and movement of upper eyelid during blink” (by Takahiro Niida), Regardless of voluntary blinks, it is described that Topen is slower than Tclose, and anatomical knowledge of the ocular and upper levator ani muscles involved in blinks The relationship between Tclose and Topen is a well-known fact.

  As a second method different from the first method, when the drowsiness level of the latest driver is low, it is determined whether or not the relationship of Tclose ≦ Topen is satisfied. The section is estimated as a false detection section. The reason for making such a determination is that the inventors analyzed the characteristics of the driver's blink during driving, and obtained the knowledge that the relationship Tclose ≦ Topen is more often established when the drowsiness level is low. This is because.

  In both the first method and the second method, the relational expression of Tclose ≦ Topen can be adjusted using a constant as necessary, such as Tclose ≦ (Topen-constant). This is because, for example, when the variation in the degree of eye opening detected (so-called “abare”) is large, the relationship between Tclose and Topen may be reversed depending on the threshold value for determining the degree of eye opening.

  In addition, in both of the first method and the second method, when the time in which the eyes are closed is long (eg, closed eyes for a long time), the relationship of Tclose ≦ Topen may not be established. It is preferable to exclude from detection. This is because when the driver is extremely sleepy, he closes his eyelids slowly and closes his eyes, but is prevented from being excluded from the false detection section when he suddenly opens his eyes.

  In step S06, as shown in FIG. 6, the blink feature value calculation unit 17 uses the eye-closed / open-eye / intermediate state within a certain time (for example, 10 seconds) to blink the feature value ( For example, “the ratio of the closed eye state”) is calculated. However, if there is a section determined as an erroneous detection section, the data in that section is not used for calculating the feature amount. Note that a feature value other than the above may be used as the feature value.

  In step S07, as shown in the graph of FIG. 7, the sleepiness level estimation unit 18 estimates the driver's sleepiness level using the relationship (statistical distribution of feature values) between the sleepiness level and blinking feature values learned in advance. To do. For example, the probability density is calculated from the statistical distribution (probability density) of the feature quantity when the sleepiness degree obtained by prior learning (offline processing) is high and when the sleepiness degree is low, and the extracted feature quantity. To estimate the sleepiness of the current driver.

  As described above, in the sleepiness level estimation apparatus 10, the eye-opening state and the eye-closed state based on the eye-opening degree are detected by the eye-opening eye determination unit 15, and the time Tclose from the eye-opening state to the eye-closing state and the eye-closing state are determined. A time Topen until the eye is opened is calculated. And when the relationship of Tclose <= Topen is satisfy | filled by the misdetection area estimation part 16, it determines with the opening / closing operation | movement of the eye at that time blinking. Furthermore, only the blink feature amount of the opening / closing operation when it is determined that the blink is detected by the erroneous detection section estimation unit 16 is used for the sleepiness level estimation by the sleepiness level estimation unit 18.

  That is, Tclose and Topen used for sleepiness level estimation are values based on the degree of eye opening, and are values calculated on the basis of the eye opening state and the eye closing state of the measurement subject. In this way, by performing blink determination based on Tclose and Topen based on the degree of eye opening, the low-frequency noise component of the degree of eye opening as shown in FIG. 8 is significantly removed, and false detection of blinking at the time of sleepiness estimation Is effectively reduced.

  Therefore, in the sleepiness level estimation apparatus 10 according to the present invention, the sleepiness level can be estimated with higher accuracy than the conventional technique that may erroneously detect blinking.

  DESCRIPTION OF SYMBOLS 10 ... Sleepiness degree estimation apparatus, 12 ... Face position detection part, 13 ... Face feature point detection part, 14 ... Eye opening degree calculation part, 15 ... Eye closure / open eye determination part, 16 ... False detection area estimation part, 17 ... Blink feature-value Calculation part, 18 ... sleepiness degree estimation part.

Claims (2)

A sleepiness level estimation device for estimating a sleepiness level of a measurement subject,
Feature point detecting means for detecting feature points of the face of the measurement subject;
An eye opening degree calculating means for calculating an eye opening degree indicating a degree of eye opening of the measurement subject based on the feature point of the face detected by the feature point detecting means;
Based on the eye opening degree calculated by the eye opening degree calculating means, an eye opening state in which the eye opening degree exceeds a predetermined threshold value and a closed eye state in which the eye opening degree is below a predetermined threshold value are detected. A closed eye opening determination means for calculating a time Tclose until the eye is closed and a time Topen until the eye is opened from the closed state;
Based on Tclose and Topen calculated by the closed eye opening determination means, when the eye opening / closing operation of the measurement subject satisfies the relationship of Tclose ≦ Topen, the eye opening / closing operation at that time is determined to blink Means,
Only when it is determined that the blink is determined by the blink determination unit, the blink feature amount calculation unit that calculates the blink feature amount related to the eye opening / closing operation at that time,
A sleepiness level estimation device comprising: estimation means for estimating the sleepiness level of the measurement subject based on the blink feature value calculated by the blink feature value calculation means. A method for estimating sleepiness of a measurement subject, comprising:
A feature point detecting step of detecting a feature point of the face of the measurement subject;
An eye opening degree calculating step for calculating an eye opening degree indicating a degree of eye opening of the measurement subject based on the feature point of the face detected in the feature point detecting step;
Based on the eye opening degree calculated in the eye opening degree calculating step, an eye opening state in which the eye opening degree exceeds a predetermined threshold value and a closed eye state in which the eye opening degree is lower than the predetermined threshold value are detected, and from the eye opening state A closed eye opening determination step of calculating a time Tclose until the closed eye state and a time Topen from the closed eye state to the opened eye state;
Based on Tclose and Topen calculated in the closed eye opening determination step, when the eye opening / closing operation of the measurement subject satisfies a relationship of Tclose ≦ Topen, the eye opening / closing operation at that time is determined as blinking Steps,
Only when it is determined that the blink is determined in the blink determination step, the blink feature amount calculation step that calculates the blink feature amount related to the eye opening / closing operation at that time,
A sleepiness level estimation method comprising: an estimation step of estimating a sleepiness level of the measurement subject based on the blink feature value calculated in the blink feature value calculation step.

Images