JP6399311B2 - Dozing detection device - Google Patents

Description

  The present invention relates to a snoozing detection apparatus, and more particularly to an apparatus for detecting a driver's snoozing while driving a vehicle and preventing the snoozing operation.

  As a snooze driving detection device, there are various snoozing driving detection devices, such as a device that captures a driver's face with a camera and determines a snoozing state from an image or a device that determines a snoozing state from a change in a driver's pulse Proposed.

  For example, Patent Document 1 discloses that an extreme value is extracted from a frequency distribution of the driver's eye opening degree, and a change in the state of the driver is estimated by detecting a temporal change in the extreme value. In this technique, an extreme value at the time of opening the eye and an extreme value at the time of closing the eye are extracted, and it is determined that the driver is drowsy when the extreme value in the closed eye state tends to be high.

  However, the above technique erroneously determines that the extreme value of the closed eye state is high even when the closed eye state and the open eye state are repeated in a short time, for example, when blinking is repeated continuously. There is a problem. In addition, detection may be temporarily interrupted by disturbance light, and measurement is not always continued.

  Furthermore, since the closed eye state is determined from the frequency distribution in a fixed measurement time, there is a problem that the closed eye state cannot be detected unless the measurement time has elapsed. That is, it is not possible to specify at which point in the measurement time the eye-closed state is shifted, and there is a possibility that a time difference occurs between the time when the eye-closed state is actually moved and the time when the eye-closed state is detected, resulting in a delay.

  On the other hand, in Patent Document 2, a threshold is used to eliminate the influence of blinking or disturbance light as noise. However, if a threshold is used when determining the closed eye state, an error due to the definition of the threshold is inevitable. . In the first place, there are individual differences in the size of the eyes, and since it is influenced by various factors such as facial expressions and postures, it is difficult to set the threshold itself.

Tokukai 2008-99884 JP 2010-184067 A

  The present invention has been made in view of the above circumstances, and an object of the present invention is to eliminate the influence of blinking, disturbance light, and the like, and improve detection accuracy and responsiveness in the dozing detection device. .

In order to solve the above problems, a dozing detection device according to the present invention is
Means for detecting eyelid opening from the binocular image of the subject;
Means for recording the heel opening as time-series data at a predetermined frame rate;
A boundary frame that divides a predetermined number of consecutive detection target frames including the latest frame of the time series data into a preceding group and a succeeding group, and separating the time series data of the preceding group and the time series data of the succeeding group And a doze determination unit that extracts a boundary frame having the maximum degree as the eye-closing start time.

  According to the above configuration, in normal implementation of the dozing detection device, since dozing does not start immediately after the start of detection, immediately after the start, the eyelid opening is at the open eye level in the entire detection target frame, and the preceding group and the subsequent group The segregation degree continues to take a very small value, but if a tendency to fall asleep appears, a low level of heel opening appears in the succeeding group, and the segregation degree increases. Therefore, the change points before and after can be directly detected from continuous time-series data, which is advantageous in improving detection accuracy and responsiveness.

  In addition to short-term changes such as blinking or lack of data due to ambient light, the degree of separation hardly changes, and individual differences such as eye size and opening, posture changes, face tilting movements, etc. Even if there is a difference in the size acquired in the images, they are considered in advance when they are recorded in time series as the data of the preceding group, so there is no need for a processing procedure or pre-processing to eliminate them. In addition to simplifying the processing and the apparatus, it is advantageous in preventing erroneous detection.

  In a preferred aspect of the present invention, the doze determination unit assumes that, for all the frames of the predetermined number of detection target frames, each frame is a boundary frame, and the time series data of the preceding group and the time series of the subsequent group Variance is obtained as a degree of separation from data, and a boundary frame that maximizes the variance is extracted as a closed eye start time. According to this aspect, the number of frames is weighted for each group in determining the degree of separation between the preceding grape and the succeeding group, so that the influence of temporary and sporadic fluctuations such as blinking is eliminated and stable doze is achieved. This is advantageous for detection.

The dozing detection device according to the present invention is
Detecting eyelid opening from the binocular image of the subject;
Recording the heel opening as time-series data at a predetermined frame rate;
Reading time-series data of a predetermined number of consecutive detection target frames including the latest frame from the time-series data recording means;
For all frames of the predetermined number of detection target frames, assuming that each frame is a boundary frame, the time series data of the detection target frame is divided into a preceding group and a subsequent group, and the time series data of the preceding group and the Calculating a degree of separation between time-series data of subsequent groups;
Extracting a boundary frame having a maximum degree of separation between the time-series data of each preceding group and the succeeding group as a closed eye start time;
The present invention can also be implemented by providing an arithmetic processing device (computer) that can execute the above.

In a further preferred aspect of the present invention, the number N of the predetermined number of detection target frames, the average value μ1 of the time series data of the preceding group, the average value μ2 of the time series data of the subsequent group, the boundary frame The degree of separation ηk at k is given by: ηk = (μ1−μ2) ²
(However, if μ2> μ1, ηk = 0)
It is comprised so that it may calculate from. According to this aspect, there is an advantage that calculation and processing can be simplified while maintaining detection accuracy.

  In a preferred aspect of the present invention, the dozing determination unit is configured to determine dozing after the eye-closed state has elapsed for a predetermined time after detecting the eye-closing start time. According to this aspect, by setting the predetermined time to a time that can be allowed according to the purpose of use of the dozing detection device, it is possible to eliminate the eye-closing operation at a level that does not affect the purpose of use.

  The dozing detection device according to the present invention is suitable to be implemented as a vehicle dozing driving prevention system. For example, the target person is a driver of a vehicle, and an alarm to the driver or a control signal to the vehicle is output when the doze determination unit performs the doze determination.

It is a block diagram which shows embodiment of the dozing detection apparatus which concerns on this invention. It is the schematic which shows the vehicle by which the dozing detection apparatus which concerns on this invention is implemented. It is the schematic which shows the detection of the eaves opening degree. It is a flowchart which shows operation | movement of the dozing detection apparatus which concerns on this invention. It is a graph which shows a time-dependent change of the eaves opening degree. It is a flowchart which shows the detection process of the eye closure start time in the dozing detection apparatus which concerns on this invention. It is a graph which shows the detection of a class separation degree and eye closure start time. It is a graph which shows a time-dependent change of the eyelid opening degree when a posture changes. It is the graph (left side) which shows the time-dependent change of the eyelid opening in time series (a)-(d), and the graph (right side) which shows the time-dependent change of a separation degree.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 2 shows an outline of a vehicle 4 in which the dozing detection device 1 according to the present invention is implemented as a dozing operation prevention system. In the figure, the dozing operation prevention system includes a dozing detection device 1, vehicle information detection means 20, and a camera. 21 and the alarm device 22, etc., and from the image taken by the camera 21, the driver's 30 eyelid opening is detected over time, and it is determined that the driver 30 is shifting to the dozing state. In such a case, a preventive measure such as urging the driver 30 to wake up through the alarm device 22 is executed.

  The vehicle information detection means 20 acquires the sensor output of each part of the vehicle as a wired or wireless signal, and wirelessly transmits vehicle information such as the vehicle speed, steering wheel steering angle, accelerator opening, brake switch, blinker switch operation state, etc. A vehicle-mounted network (CAN) acquired as a signal is preferably used.

  The camera 21 is installed on an instrument panel or a column cover facing the driver 30 so that at least both eyes of the driver 30 can be photographed, and is a digital camera using a solid-state image sensor such as a CCD or CMOS. Are preferably used.

  As shown in FIG. 1, the dozing detection device 1 includes a vehicle information recording unit 10, an image recording unit 11, a eyelid opening degree detection unit 12, a time series data recording unit 13, a class separation degree calculation unit 14, and an eye closure start time detection unit. 15, a closed eye end time detection unit 16, a dozing state determination unit 17, and a signal output unit 18.

  These are preferably a ROM for storing a program and data operable to execute each function, a CPU for performing arithmetic processing, a work area of the CPU from which the program is read, and a temporary storage area for arithmetic results And a computer comprising an input interface to which input side external devices (20, 21) are connected, an output interface to which output side external devices (22) are connected, and the like.

  Hereinafter, the operation of each unit of the dozing detection device 1 will be described in detail with reference to the block diagram of FIG. 1 and the flowchart of FIG. 4.

  The vehicle information recording unit 10 records the vehicle information acquired through the vehicle information detection means 20, for example, when the vehicle speed is zero, the vehicle information recording unit 10 considers that the vehicle is stopped, the steering wheel steering angle, the accelerator opening, the brake switch, the winker switch When an operation such as this is detected, the driver 30 assumes that he is awake and makes the dozing detection device 1 inactive (S101). Moreover, the duration in the dozing state determination described later can be changed according to the vehicle speed.

  The image recording unit 11 records an image acquired through the camera 21 as time-series data at a predetermined frame rate (S102). A method of temporarily holding and sequentially updating data of the number of frames necessary for doze determination, which will be described later, or a method of sequentially overwriting after storing up to a predetermined storage capacity may be used.

The eyelid opening degree detection unit 12 performs image processing for processing the recorded image to obtain the eyelid opening degree (S103). The image processing procedure for obtaining the eyelid opening can use a known technique related to image processing and image recognition. For example, when the images of both eyes 2L and 2R as shown in FIG. 3 are binarized, the eyelid line and the pupil portion become black pixels. Therefore, in the binary image obtained, the maximum value of the number of black pixels in each pixel row across both eyes 2L, the 2R vertically corresponds to the vertical center of the width of the left and right pupils, eyelid opening O _L, the O _R.

It should be noted that immediately after the start of recording (immediately after the start of operation), it can be considered that the state is awakened as indicated by 3 'in the figure, so the ratio of the opening degree ( _OLMAX , _ORMAX ) in this case is set as the opening degree _{O L,} O R _(%) also good as, in the present invention, as described below, for detecting the degree of separation of open-eye state and the closed-eye state, eyelid opening was absolute value and relative to that of eyelid opening it is not necessary to seek itself, it can be used the maximum number of black pixels as eyelid opening O _L, as O _R. Further, left and right eyelid opening O _L, may be adopted an average value of O _R, from the viewpoint of either one may be in the open-eye state, it may be employed the larger of the left and right . Conversely, the smaller value of the left and right may be adopted.

Time series data recording unit 13 records eyelid opening _O L obtained in eyelid opening detector 12, the _{O R} as time series data at a predetermined frame rate (S104). The frame rate is not particularly limited, but is preferably 10 to 30 frames / second in consideration of the characteristics of blinking operation. If the frame rate is too high, the load on the device increases, and the device cost required to ensure the required processing speed increases.

  Further, the time-series data recording unit 13 holds data for the minimum number of frames necessary for determining the drowsiness state, the past data is sequentially deleted, and the data of the current frame is newly added. . As a result, the time series data for the past fixed time is operated so as to be constantly updated.

  FIG. 5 is a graph showing an example of time-series data of eyelid opening, and the eyelid opening is about 90% in the vicinity of −10 to −3 seconds with reference to the current time. There is a temporary loss of data due to a drastic drop in dredging and ambient light. From such an open eye state, it has changed greatly to a closed eye state after -3 seconds.

  As will be described later in detail, in order to detect from the degree of separation when the closed-eye state continues for 2 seconds or more in consideration of the vehicle speed or the like, at least twice that time series data for 4 seconds may be held. preferable. In the present embodiment, assuming that time series data for 5 seconds is held so that reliable detection can be performed, time series data of N = 150 frames is held at a frame rate of 30 frames / second.

  The class separation degree calculation unit 14 divides the time series data for N frames acquired as described above into a preceding group C1 and a succeeding group C2, as shown in FIG. The degree of separation of the time series data of the subsequent group C2 is calculated, and a boundary value at which this degree of separation is maximized is obtained (S105). Basically, at the start of driving, it can be considered that the driver 30 is awake. Therefore, at the beginning of detection, time series data of an eye-opening state class with a high eyelid opening degree is recorded in the preceding group C1, When dozing is detected, time-series data of a closed eye state class with a low eyelid opening degree appears in the subsequent group.

The degree of separation between the two classes can be obtained by the following equation as the ratio η between the interclass variance σ _B ² and the total variance σ _T ² .
η = σ _B ² / σ _T ² = ω ₁ ω ₂ (μ1-μ2) ² / (ω ₁ ten ω ₂ ) ² σ _T ²
Here, μ1 and μ2 are average values of each class, and ω ₁ and ω ₂ are the number of data of each class.

By obtaining a boundary value between classes that maximizes the degree of separation η, it is possible to specify the change point from the open eye state class to the closed eye state class. At this time, since the total variance σ _T ² is a constant value with respect to individual data, a boundary value that maximizes the square of the difference between the average values μ 1 and μ ² of each class in the inter-class variance σ _B ² may be obtained. .

  Specifically, the class separation degree calculation unit 14 determines the time series of the preceding group C1 when the boundary frame k that divides the N frames to be detected into the preceding group C1 and the succeeding group C2 is each frame of 0 to N. The degree of separation η between the data and the time-series data of the subsequent group C2 is calculated, and the boundary frame k that maximizes the degree of separation η is set as the eye closure start time.

  The closed eye start time detection unit 15 measures the passage of time from the closed eye start time (start frame) obtained by the class separation degree calculation unit 14 (S106). The dozing state determination unit 17 determines that dozing has occurred when the maximum value of the separation degree η is continuously detected over a predetermined time (predetermined frame) set in accordance with the vehicle speed ( S109).

  In the case where the closing point from the closed eye state class to the open eye state class is detected by the closed eye end time detecting unit 16 after the closed eye start time detecting unit 15 detects the closed eye start time (start frame) (S107). Then, the required time is calculated (S108), and it is determined that a drowsiness has occurred (S109).

  In the signal output unit 18, when the dozing state is determined by the dozing state determination unit 17, the current vehicle information acquired in the vehicle information recording unit 10 is referred to (S110). When the accelerator, the brake, the blinker, etc. are not operated, a control signal is output to the alarm device 22 (S111), and an alarm for prompting the driver to wake up is issued. At the same time, a warning may be displayed on the display unit of the vehicle, or a process such as automatically stopping the vehicle may be executed. On the other hand, when it is determined from the vehicle information that the vehicle is driving, no control signal is output.

  Next, a specific example of the process of detecting the eye closure start time from the time series data will be described with reference to the flowchart of FIG. 6 and the graph of FIG.

  First, the eyelid opening degree data of a certain number of frames (0 to N) recorded in the time series data recording unit 13 is read (S201). Here, the dozing state is set to 2 seconds or more, and in order to detect the dozing state, data of the number of frames (N = 150) corresponding to 5 seconds is read at a frame rate of 30 frames / second. In the illustrated example, the current frame number is set to 0, and the frame number is increased as going back in the past. However, the oldest frame number may be set to 0 as long as the correspondence between the time and the frame number is obtained.

  Next, a calculation flag k corresponding to a boundary frame between the preceding group C1 (open eye state class) and the subsequent group C2 (closed eye state class) is initialized (S202).

  Next, the average value μ2 of the eyelid closure degree data in the range of the subsequent group C2 (0 to k) is calculated (S203), and the average value μ1 of the eyelid opening degree data in the range of the preceding group C1 (k + 11 to N) is calculated. Calculate (S204).

The degree of separation ηk in the calculation flag k corresponding to the boundary frame is calculated by the following equation and recorded (S205).
ηk = (μ1−μ2) ²
However, if μ2> μ1, ηk = 0

  The value of the calculation flag k corresponding to the boundary frame is counted up (S206).

  It is determined whether the value of the calculation flag k corresponding to the boundary frame has reached N (S207). If the value of the calculation flag k has not reached N, the process returns to step (S203), and the loop up to step (S206) Repeat the process.

  When the value of the calculation flag k corresponding to the boundary frame reaches N, the loop process is terminated, and the boundary frame kmax having the maximum degree of separation is selected from the degrees of separation η1 to ηN−1 obtained by the loop process. Determine (S208).

  The boundary frame kmax is output as the eye closure start time (S209).

(Responding to changes in face orientation)
Next, FIG. 8 shows the change over time of the eyelid opening when the face orientation changes up and down. When the face orientation of the driver 30 is turned upward or downward, the apparent image of FIG. 3, since the reflected as eye narrowed, eyelid opening O _L which is detected eyelid opening detector _12, O _R is generally lower than the _heel opening ( _OLMAX , _ORMAX ) in the initial state.

However, the doze detecting device 1 according to the invention, the eyelid opening O _L, O _R itself without determining threshold, by obtaining a predecessor group C1 separation of the subsequent group C2, directly closed eyes start time ( Since the boundary frame k) can be detected, parameter adjustment and calibration are not required even if individual differences occur when the driver changes, as well as when the face orientation changes up and down.

(Detection of eye closure start time, doze determination example)
The graphs on the left side of FIGS. 9A to 9D show the temporal changes in eyelid opening that change with a time difference of 1 second (30 frames), respectively. An example of a doze determination result will be described with reference to the drawings.

  First, in FIG. 9 (a), although the eye-closed state due to blinking and data loss due to ambient light have passed, they do not significantly affect the degree of separation and the eye-open state continues. However, at time A, the heel opening decreases and the degree of separation becomes maximum. At this time, although the time A is set as the eye closing start time, as is apparent from the subsequent drawings, the time A is excluded from the eye closing start time when the time A is no longer at the maximum difficulty level thereafter.

  On the other hand, in FIG. 9B, the degree of difficulty is maximized at time B and is the eye closure start time, and in the subsequent FIGS. 9C and 9D, the maximum degree of difficulty at time B continues. In (d), it is determined that the patient is in a dozing state by continuing the closed eye state for a predetermined time (2 seconds).

  In the above embodiment, as a boundary value between classes that maximizes the degree of separation η, a boundary value that maximizes the square of the difference between the average values μ1 and μ2 of each class is obtained. Although the case where the change point to the closed eye state class is specified is described, the change from the open eye state class to the closed eye state class is obtained by obtaining a boundary value that maximizes the difference between the average values μ1 and μ2 of each class. A point can also be specified. However, the change in the degree of separation η is small.

On the other hand, when the degree of separation is obtained as the ratio η between the inter-class variance σ _B ² and the total variance σ _T ² , the number of data ω ₁ and ω _{2 of} each class is reflected. After reaching the point of change, the degree of separation does not increase rapidly as shown in FIG. 9B, but it can be said that the stability against temporary fluctuations such as blinking is high.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, Based on the technical idea of this invention, various deformation | transformation and a change are further possible.

  For example, in the above embodiment, the case where the present invention is applied to a vehicle doze driving prevention system has been described. Can be implemented.

DESCRIPTION OF SYMBOLS 1 Dozing detection apparatus 2L, 2R Both eyes 3 Upper eyelid 4 Vehicle 10 Vehicle information recording part 11 Image recording part 12 Eyelid opening degree detection part 13 Time series data recording part 14 Class separation degree calculation part 15 Eye closure start time detection part 16 Eye closing end time detecting unit 17 dozing state determining unit 18 warning display control signal output unit 20 the vehicle information detection means 21 camera 22 alarm device 30 driver N detection target frame number _O L, _{O R} eyelid opening η, ηk separability .mu.1, .mu.2 average value

Claims (6)

Means for detecting eyelid opening from the binocular image of the subject;
Means for recording the heel opening as time-series data at a predetermined frame rate;
A boundary frame that divides a predetermined number of consecutive detection target frames including the latest frame of the time series data into a preceding group and a succeeding group, and separating the time series data of the preceding group and the time series data of the succeeding group A dozing determination unit that extracts a boundary frame that has a maximum degree as a closed eye start time;
A dozing detection device with   The dozing determination unit assumes that each frame is a boundary frame for all frames of the predetermined number of detection target frames, and the degree of separation between the time-series data of the preceding group and the time-series data of the subsequent group The dozing detection apparatus according to claim 1, wherein the device is configured to obtain a variance and to extract a boundary frame that maximizes the variance as a closing eye start time. Detecting eyelid opening from the binocular image of the subject;
Recording the heel opening as time-series data at a predetermined frame rate;
Reading time-series data of a predetermined number of consecutive detection target frames including the latest frame from the time-series data recording means;
For all frames of the predetermined number of detection target frames, assuming that each frame is a boundary frame, the time series data of the detection target frame is divided into a preceding group and a subsequent group, and the time series data of the preceding group and the Calculating a degree of separation between time-series data of subsequent groups;
Extracting a boundary frame having a maximum degree of separation between the time-series data of each preceding group and the succeeding group as a closed eye start time;
Dozing detection device provided with an arithmetic processing device capable of executing the above. The number N of the predetermined number of detection target frames, the average value μ1 of the time-series data of the preceding group, the average value μ2 of the time-series data of the subsequent group, and the degree of separation ηk in the boundary frame k are expressed by the following equation ηk = (Μ1-μ2) ²
However, if μ2> μ1, ηk = 0
The dozing detection device according to claim 1, wherein the device is configured to calculate from the following.   The dozing detection unit according to any one of claims 1 to 4, wherein the dozing determination unit is configured to determine dozing after the eye-closing start time is detected and the eye-closed state has elapsed for a predetermined time. apparatus.   The object person is a driver of a vehicle, and is configured to output an alarm to the driver or a control signal to the vehicle at the time of dozing determination by the dozing determination unit. The dozing detection device according to any one of 1 to 5.

Images