JP7745829B2 - Driver state determination method and device - Google Patents

Description

The present invention relates to a driver condition determination method and device for detecting abnormal conditions of a driver operating a vehicle.

BACKGROUND ART Driver support devices have been known that accurately grasp a driver's state of consciousness, particularly the driver's state of alertness, and support the driver in driving a vehicle in accordance with the driver's state of alertness.
An abnormal state of a driver, including illness or drowsiness, is determined based on head-related information such as the driver's head posture and facial state acquired by an imaging device such as a camera. Among the head-related information, the head posture is detected based on, for example, the driver's facial direction and head position (three-dimensional coordinates), and the facial state is detected based on, for example, the driver's facial expression and eye opening state (eyelid state).

The vehicle alarm device of Patent Document 1 has a drowsiness detection unit that detects whether the driver is drowsy, an inattentive detection unit that detects whether the driver is looking away, an abnormal posture detection unit that detects whether the driver is in an abnormal posture, a vehicle speed detection unit that detects the vehicle speed, a notification unit that issues an alarm, and an alarm control unit that activates the notification unit based on detection by either detection unit.When the vehicle speed is below a predetermined threshold, the alarm control unit issues an alarm when drowsiness is detected, but does not issue an alarm when inattentive detection is detected.

Japanese Patent Application Laid-Open No. 2019-133402

The vehicle alarm device of Patent Document 1 issues an alarm in the event of drowsiness or an abnormal posture, but can avoid issuing unnecessary alarms in situations where the driver is driving the vehicle at extremely low speeds to search for a parking spot, etc., thereby limiting the occurrence of situations that the driver finds annoying.
However, although the vehicle warning device of Patent Document 1 can avoid false detection in the inattentive driving judgment, there is a risk that it cannot avoid false detection in the abnormal posture judgment.

Generally, when a driver develops a disease, an abnormal posture, in other words, an unconscious collapse of posture without the driver's awareness, is recognized by external observation.
Therefore, unconscious posture changes are detected by facial orientation and head position.
On the other hand, when the driver consciously adopts an unsuitable driving posture, i.e., intentionally loses posture, if an abnormality is determined based on the driver's facial direction and head position, the driver's abnormal state is determined.
However, since the driver intentionally loses his/her posture, the warning is issued even though there is no particular problem in terms of consciousness or driving operation, which may cause inconvenience to the driver.

Some driver assistance devices automatically move the vehicle toward the shoulder of the road and autonomously stop the vehicle when it is determined that the driver is in an abnormal state.
Such driver assistance devices automatically stop the vehicle even if it is determined that the driver has intentionally lost posture, which not only increases the inconvenience for the driver but also poses the risk of an unexpected situation in which the vehicle makes an emergency stop in front of other vehicles.
That is, although it is essential to improve the accuracy of determining whether a driver is in an abnormal posture, it is not easy.

The object of the present invention is to provide a driver condition determination method and device that can improve the accuracy of determining whether a driver is in an abnormal state.

The driver state determination method of claim 1 is a driver state determination method for detecting an abnormal state of a vehicle driver, the method comprising: performing a control process for determining whether the driver's eyes are closed and a control process for determining a head posture of the driver in parallel based on an image of the driver; the control process for determining whether the driver's eyes are closed and a control process for determining a head posture of the driver; the control process for determining whether the driver is in an abnormal state when the driver does not respond to an alarm that is issued when the driver's eyes remain closed; and the control process for determining whether the driver is in an abnormal state when the driver does not respond to an alarm that is issued when the driver's eyes remain closed, the control process for determining whether the driver is in an abnormal state, the control process for determining whether the driver is in an abnormal state when the driver does not respond to an alarm that is issued when the driver's eyes remain closed ... The method includes a gaze determination index calculation step for calculating a saliency index, a saccade amplitude index, and a saccade frequency index as multiple gaze determination indexes for the driver; a head posture acquisition step for acquiring the facial orientation and head position of the driver based on the captured image ; and a driver abnormal state determination step for determining whether the driver is in an abnormal state or not.The driver abnormal state determination step is characterized in that it determines that the driver has intentionally lost posture if the head posture of the driver acquired in the head posture acquisition step is an abnormal posture and the multiple gaze determination indexes of the driver calculated in the gaze determination index calculation step are all normal, and it determines that the driver is in an abnormal state if the head posture of the driver acquired in the head posture acquisition step is an abnormal posture and any of the multiple gaze determination indexes of the driver calculated in the gaze determination index calculation step is abnormal.

In this driver state determination method, a control process for determining whether the driver's eyes are closed and a control process for determining the driver's head posture are performed in parallel based on an image of the driver. The control process for determining whether the driver's eyes are closed determines that the driver is in an abnormal state if the driver does not respond to a warning that is issued when the driver's eyes remain closed. The head posture determination control process includes a peripheral information acquisition process for acquiring peripheral information regarding the surrounding conditions of the vehicle, a field of view image acquisition process for acquiring a field of view image corresponding to the driver's field of view, an attraction distribution generation process for generating an attractiveness distribution of the driver for the field of view image, a gaze behavior acquisition process for acquiring the driver's gaze behavior based on the captured image and the field of view image, a gaze determination index calculation process for calculating a saliency index, a saccade amplitude index, and a saccade frequency index as multiple gaze determination indexes for the driver based on the peripheral information, the field of view image , the attractiveness distribution, and the gaze behavior, a head posture acquisition process for acquiring the driver's facial direction and head position based on the captured image, and a driver abnormal state determination process for determining whether the driver is in an abnormal state.Therefore, multiple gaze determination indexes related to the driver's level of attention can be included in the determination conditions for the driver's abnormal state.
The driver abnormal state determination process determines that the driver has intentionally lost posture if the driver's head posture acquired in the head posture acquisition process is abnormal and the driver's multiple gaze determination indices calculated in the gaze determination index calculation process are all normal, thereby making it possible to determine whether a driver has an inappropriate driving posture even if there are no problems with their consciousness or driving operation.
The driver abnormal state determination process determines that the driver is in an abnormal state if the head posture of the driver acquired in the head posture acquisition process is abnormal and any of the driver's multiple gaze determination indices calculated in the gaze determination index calculation process is abnormal, so that unconscious posture deviations of the driver can be determined with high accuracy from their posture deviations.
Therefore, the abnormal state of the driver is determined based on the driver's eye closure state, abnormal head posture, and multiple gaze determination indices related to the driver's level of attention, thereby improving the accuracy of determining the abnormal state of a driver who is unable to drive normally.

The invention of claim 2 is characterized in that in the invention of claim 1, the driver abnormal state determining step determines that the head posture is abnormal based on at least the angle of the driver's face.
According to this configuration, it is possible to easily determine whether the driver's head posture is abnormal.

The invention of claim 3 is characterized in that, in the invention of claim 1 or 2, the driver abnormal state determination process determines that the driver is in an abnormal state if the head posture of the driver acquired in the head posture acquisition process is abnormal and any of the driver's multiple gaze determination indices calculated in the gaze determination index calculation process is abnormal, and this situation continues for a predetermined time.
This configuration makes it possible to reliably determine whether the driver is in an abnormal state.

a driver's state determination device for determining whether a driver of a vehicle is in an abnormal state, the driver's state determination device comprising: a peripheral information acquisition means for acquiring peripheral information relating to a peripheral situation of the vehicle; a driving operation amount acquisition means for acquiring a driving operation amount of the driver; a field of view image acquisition means for acquiring a field of view image corresponding to the field of view of the driver based on the peripheral information; an attraction distribution generation means for generating an attractiveness distribution of the driver for the field of view image; a gaze behavior acquisition means for acquiring a gaze behavior of the driver based on an image captured by an internal camera that images the driver; a gaze determination index calculation means for calculating a saliency index, a saccade amplitude index, and a saccade frequency index as a plurality of gaze determination indexes of the driver based on the peripheral information, the field of view image, the attractiveness distribution, and the gaze behavior; a head posture acquisition means for acquiring a facial orientation and a head position of the driver based on the image ; and an abnormal driver state determination means for determining whether the driver is in an abnormal state. and a determination means, wherein the driver abnormal state determination means performs a control process for determining whether the driver's eyes are closed and a control process for determining the driver's head posture in parallel based on the captured image, and the eye closure determination control process determines that the driver is in an abnormal state if the driver does not react based on the driving operation amount and the captured image to an alarm that is issued when the driver's eyes remain closed, and the head posture determination control process determines that the driver has intentionally lost posture if the head posture of the driver acquired by the head posture acquisition means is an abnormal posture and all of the plurality of gaze determination indices of the driver calculated by the gaze determination index calculation means are normal, and determines that the driver is in an abnormal state if the head posture of the driver acquired by the head posture acquisition means is an abnormal posture and any of the plurality of gaze determination indices of the driver calculated by the gaze determination index calculation means is abnormal.

This driver condition determination device detects an abnormal state of the driver of a vehicle and includes a peripheral information acquisition means for acquiring peripheral information related to the surrounding conditions of the vehicle, a driving operation amount acquisition means for acquiring the driving operation amount of the driver, a field of view image acquisition means for acquiring a field of view image corresponding to the driver's field of view based on the peripheral information, an attraction distribution generation means for generating an attractiveness distribution of the driver for the field of view image, a gaze behavior acquisition means for acquiring the gaze behavior of the driver based on an image captured by an internal camera that images the driver, a gaze determination index calculation means for calculating a saliency index, a saccade amplitude index, and a saccade frequency index as multiple gaze determination indices of the driver based on the peripheral information, the field of view image, the attractiveness distribution, and the gaze behavior, a head posture acquisition means for acquiring the facial direction and head position of the driver based on the captured image , and a driver abnormal condition determination means for determining whether the driver is in an abnormal state or not.Therefore, multiple gaze determination indices related to the driver's level of attention can be included in the determination conditions for the driver's abnormal state.
The driver abnormal state determination means determines that the driver has intentionally lost posture if the head posture of the driver acquired by the head posture acquisition means is abnormal and the multiple gaze determination indices of the driver calculated by the gaze determination index calculation means are all normal, and therefore can determine that a driver has an inappropriate driving posture even if there are no problems with their state of consciousness or driving operation.
The driver abnormal state determination means determines that the driver is in an abnormal state if the head posture of the driver acquired by the head posture acquisition means is an abnormal posture and any of the driver's multiple gaze determination indices calculated by the gaze determination index calculation means is abnormal, and therefore can accurately determine whether the driver's posture is unconscious.
Therefore, the abnormal state of the driver is determined based on the driver's eye closure state, abnormal head posture, and multiple gaze determination indices related to the driver's level of attention, thereby improving the accuracy of determining the abnormal state of a driver who is unable to drive normally.

According to the driver condition determination method and device of the present invention, the accuracy of determining the driver's abnormal condition can be improved by using the driver's closed eye state, abnormal head posture, and multiple gaze determination indicators as conditions for determining the driver's abnormal condition.

1 is a block diagram of a driver state determination device according to a first embodiment; FIG. 1 is an explanatory diagram of a monitoring support device. FIG. 2 is an explanatory diagram of a braking assist device. FIG. 2 is an explanatory diagram of a vehicle stopping assistance device. 10 is a step chart of a line-of-sight determination control process. 10 is a graph for explaining extraction of a saccade. 10 is a graph for explaining a noise removal process of a saccade. FIG. 10 is a diagram showing an example of a time change in saliency related to a driver's gaze point. FIG. 9 is a diagram showing an example of time-dependent change in saliency related to random points whose coordinates are randomly specified in the same environment as in FIG. 8 . FIG. 1 is an explanatory diagram of a saliency index. 10 is a flowchart showing a control process for determining whether the eyes are closed. 10 is a flowchart showing a head posture determination control process.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
The following description exemplifies the application of the present invention to a driver state determination device for a vehicle, and does not limit the present invention, its applications, or its uses.

A first embodiment of the present invention will be described below with reference to FIGS. 1 to 12. FIG.
The driver state determination device M is a driving assistance device that determines the driver's state of consciousness, particularly an abnormal state that makes it difficult for the driver to operate the vehicle (hereinafter also referred to as the driver's own vehicle), and can autonomously make an emergency stop if an abnormal state of the driver is determined.
Hereinafter, an abnormal state of the driver will be described as a state in which the driver is not consciously awake and is physically unwell, making it impossible for the driver to perform driving operations (including the onset of illness or drowsiness).
The following description also includes a description of a driver state determination method.

As shown in Figure 1, the driver state determination device M mainly comprises multiple detection means 1 capable of detecting the driver's operating behavior and the vehicle's driving behavior, a driver assistance device 2 that autonomously provides vehicle driving assistance to the driver from the vehicle side, and an ECU (Electronic Control Unit) 3 that performs various calculations based on input signals from the detection means 1 and outputs various command signals to the driver assistance device 2.

First, the detection means 1 will be described.
1, the detection means 1 includes an external camera 10 that captures an image ahead of the vehicle in the traveling direction, an internal camera 11 that captures an image of the driver, a vehicle speed sensor 12 that detects the speed of the vehicle, a steering angle sensor 13 that detects the steering angle due to the operation of the steering wheel (not shown), a grip sensor 14 that detects the driver's grip on the steering wheel, an accelerator sensor 15 that detects the amount of depression of the accelerator pedal (not shown), a brake sensor 16 that detects the amount of depression of the brake pedal (not shown), and a hazard switch 17 that activates hazard lamps (not shown). Each of the sensors 12 to 16 corresponds to a driving operation amount acquisition means that acquires the driving operation amount at a predetermined timing, i.e., the current time immediately before the determination time.

The external camera 10 is provided to cover the entire area around the vehicle. The external camera 10 captures an image of the environment surrounding the vehicle (external environment around the vehicle) to obtain an image of the area ahead in the direction of travel, which is part of the external environment of the vehicle.
The image information acquired by the external camera 10 is transmitted to the ECU 3. The external camera 10 is a monocular camera with a wide-angle lens, and is configured using a solid-state image sensor such as a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS).

The internal camera 11 is, for example, a CCD camera, and is mounted in the upper center of an instrument panel (not shown) in a position facing the driver.
The internal camera 11 is configured to be able to capture an image of the head, including the face, of a driver seated in the driver's seat and fastening a seat belt from the front. The captured head image is configured to be able to extract an image of the driver's face and detect the direction of the face (direction of face orientation), identify the driver's facial expression, enlarge and detect the iris to detect the direction of the line of sight, enlarge and detect the eyelid to detect the degree of eye opening and closing, and the three-dimensional coordinate position of the driver's head (center of gravity).
The internal camera 11 is required to be capable of capturing at least the necessary captured images, and may be equipped with two cameras, a front-facing camera and an upward camera installed directly above the driver's seat, or three or more cameras.

Next, the driver assistance device 2 will be described.
As shown in FIG. 1, the driver assistance device 2 is composed of a monitoring assistance device 21 that notifies the driver of an alert state, a braking assistance device 22 that causes the vehicle to perform a braking operation, and a stopping assistance device 23 that causes the vehicle to perform an emergency stop operation.

The monitoring support device 21 is configured to recognize the state of consciousness, that is, the wakefulness state of the driver, and, if it determines that recovery is necessary, to notify the driver and encourage the driver to recover consciousness.
When the monitoring support device 21 receives an inattentive driving signal or a drowsiness estimation signal from the ECU 3, it issues an alarm to the driver from one or more speakers (not shown) provided in the vehicle cabin.
2, a speedometer section 21a disposed on the instrument panel is provided with a display section 21b capable of displaying a warning. When the monitoring support device 21 receives an inattentive driving signal or a drowsiness estimation signal from the ECU 3, the monitoring support device 21 displays a warning such as "Watch out for the road ahead" on the display section 21b.

The braking assist device 22 is configured to issue a warning when the vehicle approaches another vehicle ahead while traveling, and to apply emergency braking when the braking limit distance at which a collision can be avoided is reached.
As shown in FIG. 3, the braking assist device 22 issues a warning to the driver from a speaker when the host vehicle approaches the leading vehicle B to the position of the host vehicle A1.
The separation distance between the other vehicle B and the host vehicle A1 is a distance corresponding to the time (for example, 0.8 seconds) that the driver can normally avoid a collision by his/her own driving operation (manual operation).
The braking assist device 22 automatically applies emergency braking when the host vehicle approaches the leading vehicle B to the position of the host vehicle A2. The separation distance between the leading vehicle B and the host vehicle A2 is the limit distance at which a collision can be avoided by applying emergency braking equivalent to the maximum braking force.

As shown in FIG. 3, when the braking assist device 22 receives an inattentive driving signal or a drowsiness estimation signal from the ECU 3, it issues an alarm to the driver from the speaker when the vehicle approaches a leading vehicle B to a position of vehicle A3 that is further away from the leading vehicle B than the position of vehicle A1. Here, the separation distance between vehicle B and vehicle A3 corresponds to the time (e.g., 1.2 seconds) that the driver can use their own driving operations to avoid a collision even if they are distracted.

When the stopping assistance device 23 receives a drowsiness estimation signal from the ECU 3, it issues an alarm to the driver from a speaker, and after receiving a drowsiness confirmation signal, it autonomously brings the vehicle to an emergency stop.
As shown in Figure 4, when the vehicle stopping assistance device 23 receives a drowsiness estimation signal corresponding to an eyes-closed state at the position of the host vehicle X1, it activates the speaker to issue an alarm, and when it receives a drowsiness confirmation signal corresponding to an unoperated driving state or the like at the position of the host vehicle X2, it starts automatic driving at the position of the host vehicle X3, even if the host vehicle had been manually driven by the driver until then. After starting automatic driving, the host vehicle moves to a safety zone such as the shoulder of the road and then makes an autonomous emergency stop at the position of the host vehicle X4.

Furthermore, when the stopping assistance device 23 receives an abnormality signal from the ECU 3, it autonomously brings the host vehicle to an emergency stop. As shown in FIG. 4, the stopping assistance device 23 receives an abnormality signal from the driver at the position of host vehicle X2. Next, at the position of host vehicle X3, automatic driving begins even if the driver has been driving manually up until then. After starting automatic driving, the host vehicle moves to a safety zone such as the shoulder of the road and then makes an autonomous emergency stop at the position of host vehicle X4. Thereafter, the stopping assistance device 23 makes an emergency call to an emergency center or the like, for example, after a predetermined vehicle stop maintenance period has elapsed. Hereinafter, this autonomous emergency stop control is referred to as abnormality sequence control.

Next, the ECU 3 will be described.
The ECU 3 determines whether the driver is in an abnormal state based on the driver's symptoms at the time of an abnormality (hereinafter referred to as "abnormal symptoms"), the driver's physical condition at the time of the abnormality (hereinafter referred to as "abnormal state"), and gaze abnormality, which is one of the driver's gaze characteristics. The ECU 3 is composed of a CPU (Central Processing Unit), ROM, RAM, an internal interface, an external interface, etc. The ROM stores various programs and data for brake control, display control, etc., and the RAM has a processing area used when the CPU performs a series of processes.

As shown in Figure 1, the ECU 3 includes a gaze determination unit 30 (field of view image acquisition means, gaze determination index calculation means) that determines whether the driver has an abnormality in their gaze, a head posture determination unit 31 that determines the state of the driver's head posture, an eye state determination unit 32 that determines whether the driver's left and right eyes are open or closed, and a driver state determination unit 33 that determines the driver's state that may be an obstacle to driving the vehicle.

First, the line-of-sight determination unit 30 will be described.
1, the gaze determination unit 30 includes a saliency (attractiveness) determination unit 30a (attractiveness distribution generating means) and a saccade (saccade eye movement) determination unit 30b (gaze behavior acquiring means). The saliency determination unit 30a determines gaze abnormality based on a saliency index, and the saccade determination unit 30b determines gaze abnormality based on a saccade amplitude index and a saccade frequency index. The gaze determination unit 30 determines the gaze to be normal if all of these gaze determination indices are determined to be normal, and determines the gaze to be abnormal if any one of the three gaze determination indices is determined to be abnormal.

As shown in the step chart of FIG. 5, the gaze determination control process executed by the gaze determination unit 30 is carried out in the following order: peripheral information acquisition step S1, field of view image acquisition step S2, attention distribution generation step S3, gaze behavior acquisition step S4, gaze determination index calculation step S5, and gaze index determination step S6.
In the peripheral information acquisition step S1, the image captured by the external camera 10 and the image captured by the internal camera 11 are received and processed. This image processing includes distortion correction processing to correct distortion in the image and white balance adjustment processing to adjust the white balance of the image.

In the field of view image acquisition step S2, a field of view image that appears in the field of view of the driver seated in the driver's seat is created based on input from the external camera 11 and the internal camera 12.
The field of view image is created by combining an image of the external environment in front of the vehicle captured by the external camera 10 with vehicle body components such as pillars that are in the driver's field of view while the vehicle is traveling.
In addition, the driver's gaze point is calculated from the driver's line of sight direction and the field of view image.

In the attention distribution generation process S3, saliency is calculated for each feature, such as color-based saliency, brightness-based saliency, and movement-based saliency, for parts of the field of view image acquired in the field of view image acquisition process S2 other than the vehicle body components, and a saliency map for each feature is generated. The final saliency map is created by adding together these generated saliency maps for each feature.

In the gaze behavior acquisition step S4, the amplitude and frequency of saccades are acquired.
6, the period sandwiched between adjacent fixation periods is a saccade period. The saccade amplitude ds is the distance the line of sight moves during the saccade period.
7, saccade candidates are extracted based on changes in the distance of gaze movement, and a regression curve L10 is derived based on the multiple saccade candidates. A saccade range R10 is set between a first reference curve L11 obtained by shifting the regression curve L10 in the direction of increasing the movement speed and a second reference curve L12 obtained by shifting the regression curve L10 in the direction of decreasing the movement speed, and saccade candidates included in the saccade range R10 are extracted as saccades from among the multiple saccade candidates.

In the gaze determination index calculation step S5, a saliency index, a saccade amplitude index, and a saccade frequency index are calculated.
The saccade amplitude index is calculated by averaging the amplitude of saccades within a predetermined period, and the saccade frequency index is calculated by dividing the number of saccades within a predetermined period by the duration of the period.

In calculating the saliency index, the driver's gaze point is detected in the saliency map obtained within the measurement time, and a saliency graph (see Figure 8) is created by extracting the gaze point for each specified time, and a random point is specified in the saliency map obtained within the measurement time, and a saliency graph (see Figure 9) is created by extracting the random point for each specified time.
The probability of exceeding the threshold at the fixation point is calculated by dividing the number of fixation point saliencies that exceed the threshold by the total number of fixation point saliencies. Also, the probability of exceeding the threshold at the random point is calculated by dividing the number of random point saliencies that exceed the threshold by the total number of random point saliencies.

As shown in Figure 10, a receiver operating characteristic (ROC) curve C is derived based on a combination of the probability of exceeding a threshold at the gaze point and the probability of exceeding a threshold at a random point. This ROC curve C changes depending on the strength of the tendency for the driver's gaze to be attracted to high saliency areas. Finally, the saliency index (AUC: Area Under Curve), which is the area below the ROC curve C (hatched area), is calculated.

In the gaze index determination process S6, if the saliency index exceeds a predetermined standard AUC value, or if the saccade amplitude index is equal to or greater than a determination threshold, or if the saccade frequency index is equal to or greater than a determination threshold, it is determined that the gaze abnormality is causing a decrease in attention function.
If the saliency index, the saccade amplitude index, and the saccade frequency index are all determined to be normal, the driver's line of sight is determined to be normal.

Next, the head posture determination unit 31 will be described.
The head posture determination unit 31 determines the head posture state of the driver based on the image information captured by the internal camera 11. When the head posture determination unit 31 detects that the driver's head posture state is one of the following: head down, lying sideways, leaning face down, or leaning backward, it determines that the head posture is out of alignment.
When a head posture error is detected, the first counter C1 is set to 1. The initial value of the first counter C1 is 0.

For example, the head posture determination unit 31 outputs a head-down determination signal when the absolute value of the face angle in the pitch direction is equal to or greater than a predetermined threshold, and outputs a sideways (leaning) determination signal when the absolute value of the face angle in the roll direction is equal to or greater than the predetermined threshold. Furthermore, the head posture determination unit 31 outputs a head-down determination signal when the absolute value of the face angle in the pitch direction is equal to or greater than the predetermined threshold and the current head position is equal to or greater than the reference head position minus the head position threshold, and outputs a backward lean determination signal when the face angle in the pitch direction is equal to or greater than the predetermined threshold and the current head position is smaller than the reference head position plus the head position threshold. The thresholds associated with the angles and head positions used to determine head-down, sideways, leaning, and backward leaning correspond to head posture determination thresholds used to determine abnormal head posture in the driver.

The eye state determination unit 32 determines whether the driver's eyes are closed or not.
For example, the eye state determination unit 32 detects the driver's upper and lower eyelids captured by the internal camera 11 and detects the degree of eye opening based on the number of pixels between the edges of the eyelids. If the detected degree of eye opening is equal to or greater than a predetermined determination threshold, it is determined that the driver's eyes are open. If the detected degree of eye opening is less than the predetermined determination threshold, it is determined that the driver's eyes are closed. When the eye state is determined to be closed, 1 is assigned to the second counter C2. The initial value of the second counter C2 is 0.
In this embodiment, the eyes are determined to be closed only when both the left and right eyes of the driver are determined to be closed, but in other cases, the eyes are not determined to be closed.

The driver state determination unit 33 includes an inattentiveness determination unit 33a that determines whether the driver is looking in a direction other than the forward direction in the direction of travel; a drowsiness determination unit 33b that determines whether the driver is in a drowsy state, in which it is estimated that the driver is drowsy, or a confirmed drowsy state, in which it is highly likely that the driver is drowsy; and an abnormality determination unit 33c (driver abnormal state determination means) that determines whether the driver is in an abnormal state, in which it is difficult for him or her to drive the vehicle.

The inattentive driving determining unit 33a will be described.
The inattentiveness determination unit 33a determines whether the driver is looking away from the driver based on the driver's face direction information, gaze direction information, and vehicle information, and if it is determined that the driver is looking away, it outputs an inattentiveness signal to the monitoring support device 21 and the braking support device 22.
When the count value of the driver's gaze direction in the yaw direction falls outside the range of the yaw direction threshold, it is determined that the driver is looking aside. The yaw direction threshold is changed according to the vehicle speed or the radius of curvature when the vehicle is traveling. Also, when the count value of the driver's gaze direction in the pitch direction falls outside the range of the pitch direction threshold, it is determined that the driver is looking aside. The pitch direction threshold is changed according to the vehicle speed.

The yaw gaze direction count value and pitch gaze direction count value are calculated using the long-term continuous inattentive count or the short-term cumulative inattentive count. The long-term continuous inattentive count refers to the number of times the determination results for each processing cycle are consecutive in chronological order. The short-term cumulative inattentive count refers to the cumulative count when the determination results for each processing cycle match the conditions during the specified period prior to that.

The reliability of the gaze direction information in the yaw and pitch directions is determined using a preset reliability threshold. The reliability of each gaze direction information is set based on the clarity of the captured image of the driver (clarity of the contours of the face, eyelids, pupils, etc.).
If the reliability of each gaze direction information is low and each gaze direction is invalid, it is interpolated using face direction information in the yaw and pitch directions.If the reliability of face direction information in the yaw and pitch directions is low and each face direction is invalid, it is interpolated using the previous valid gaze direction.

Next, the drowsiness determining unit 33b will be described.
The drowsiness determination unit 33b determines whether the driver is drowsy or not based on the duration of eye closure and vehicle information, etc., and if it is estimated that the driver is drowsy, it outputs a drowsiness estimation signal to the monitoring support device 21, the braking support device 22, and the stopping support device 23, and if it is confirmed that the driver is drowsy, it outputs a drowsiness confirmation signal to the stopping support device 23.

The drowsiness determining unit 33b estimates a drowsy state when the driver's continuous eye-closure time exceeds a predetermined time threshold, or when the driver repeatedly closes his/her eyes for short periods exceeding the predetermined time threshold within a certain period of time.
In addition, the drowsiness determination unit 33b determines the drowsy state if the driver's continuous eye closure time exceeds a predetermined time threshold, or if the continuous eye closure and no driving operation state exceeds a time threshold, when the vehicle is traveling at a certain speed or above after a drowsiness warning.

The non-operation state is determined as inappropriate driving operation while driving by comparing the detection signal from the detection means 1 with a judgment threshold. Specifically, it is determined by at least one of the following operating states: non-operation of the steering wheel, non-grasping of the steering wheel, and non-operation of the accelerator or brake. It is also possible to set an extremely small judgment threshold for each operating element to determine inappropriate driving operation.

Next, the abnormality determination unit 33c will be described.
The abnormality determination unit 33c determines that the driver has intentionally lost posture if the driver's head posture acquired by the internal camera 11 is abnormal and the three gaze determination indices of the driver calculated by the gaze determination unit 30 are all normal, and determines that the driver is in an abnormal state if the driver's head posture acquired by the internal camera 11 is abnormal and any of the three gaze determination indices of the driver calculated by the gaze determination unit 30 is abnormal.

The inventor has found that the driver's symptoms of an abnormality can be detected using eye closure as a parameter, the driver's physical state when an abnormality occurs can be detected using head posture as a parameter, and the driver's behavior when an abnormality occurs can be detected using driving inactivity as a parameter. However, head posture abnormalities can be classified into unconscious posture abnormalities caused by illness or the like and intentional posture abnormalities, and if both are determined to be head posture abnormalities, the determination accuracy will decrease.
Therefore, in this embodiment, an unconscious posture error and an intentional posture error are distinguished from each other using a gaze determination index.

Next, the eye closure determination control process will be described with reference to the flowchart of FIG.
Incidentally, Si (i=11, 12, . . . ) indicates a step for each process.

First, as shown in FIG. 11, the ECU 3 receives input of various information such as outputs from various detecting means 1, such as the internal camera 11, and the first and second counters C1 and C2 (S11), and then proceeds to S12.
In S12, it is determined whether or not the driver's eyes are closed.
If the result of the determination in S12 is that the driver has his/her eyes closed, the process proceeds to S13 because closed eyes, which is a sign of an abnormality (disease onset or drowsiness), have been detected. If the result of the determination in S12 is that the driver does not have his/her eyes closed, the process returns because at least one of the eyes is open or the reliability of the imaging information is low.

In S13, 1 is added to the current count value of the second counter C2, and then the process proceeds to S14. In S14, it is determined whether the count value of the second counter C2 is equal to or greater than the second determination threshold L2. If the result of the determination in S14 is that the count value of the second counter C2 is equal to or greater than the second determination threshold L2, the eyes are still closed, and it is estimated that the patient is experiencing a disease or is dozing, so the process proceeds to S15.
If the result of the determination in S14 is that the count value of the second counter C2 is less than the second determination threshold L2, the onset of a disease or drowsiness cannot be estimated at present, and therefore the process returns.
In S15, an alarm is issued, and then the process proceeds to S16.

In S16, it is determined whether or not the first determination condition is met.
The first determination conditions are the following three conditions.
(1) The driver's head posture is abnormal. (2) The driver has not operated the vehicle for a first non-operation determination period. (3) The driver has closed his or her eyes for a certain period.

If the result of the determination in S16 is that the first determination condition is met, the driver will not start driving even if a warning is issued, so the process proceeds to S17.
If the result of the determination in S16 is that the first determination condition is not met, the driver has started driving in response to the warning, so the process returns.

In S17, it is determined whether a cancel signal for stopping the alarm has been generated.
If the result of the determination in S17 is that a cancel signal has been generated, the driver has performed an operation to generate a cancel signal, so the count values of the first and second counters C1 and C2 are each set to 0 (S18), and then the process returns.If the result of the determination in S17 is that a cancel signal has not been generated, the driver cannot cancel, so the process proceeds to S19.
In S19, the abnormality sequence control is executed, and then the process returns to S6. In S6, 0 is substituted into the first and second counters C1 and C2.

Next, the head posture determination control process will be described with reference to the flowchart of Fig. 12. This head posture determination control process is controlled independently of and in parallel with the eye closure determination control process.
Si (i=21, 22...) indicates the step for each process.

First, as shown in FIG. 12, the ECU 3 receives an input of an output from the detection means 1 such as the internal camera 11 and the hazard switch 17 (S21), and then proceeds to S22.
In S22, it is determined whether the driver's head posture is out of alignment.
If the result of the determination in S22 is that the driver's head posture is out of alignment, the process proceeds to S23 to distinguish between unconscious and intentional posture deviations. If the result of the determination in S22 is that the driver's head posture is not out of alignment, the process returns because the driver is not in an abnormal state.

In S23, it is determined whether the driver's line of sight is abnormal.
The line-of-sight determination unit 30 determines line-of-sight abnormalities using line-of-sight determination indices (saliency index, saccade amplitude index, saccade frequency index) extracted from the driver's line of sight.
If the result of the determination in S23 is that the driver has an abnormal line of sight, the driver's attention function is impaired and the process proceeds to S24. If the result of the determination in S23 is that the driver does not have an abnormal line of sight, the driver's attention function is not impaired and the process proceeds to S32.

In S24, it is determined whether the driver is operating the various operating parts (steering wheel, accelerator pedal, brake pedal, etc.) appropriately.
If the result of the determination in S24 is that the driver is operating the vehicle appropriately, even if the head posture is out of alignment, the process returns. If the result of the determination in S24 is that the driver is not operating the vehicle appropriately, for example, if no operation units are being operated, the process proceeds to S25.

In S25, it is determined whether the vehicle speed is equal to or greater than a determination threshold value.
If the result of the determination in S25 is that the vehicle speed is equal to or greater than the determination threshold value,
Since the driver is unconsciously losing his posture and is not operating the vehicle even though the vehicle is moving, the process proceeds to S26. If the result of the determination in S25 is that the vehicle speed is below the determination threshold, the driver may be stopped to wait for recovery, so the process returns. In S26, 1 is added to the current count value of the first counter C1, and the process proceeds to S27.

In S27, it is determined whether the count value of the first counter C1 is greater than or equal to the first determination threshold L1. If the result of the determination in S27 is that the count value of the first counter C1 is greater than or equal to the first determination threshold L1, the onset of a disease is suspected, and the process proceeds to S28. If the result of the determination in S27 is that the count value of the first counter C1 is less than the first determination threshold L1, the onset of a disease cannot currently be suspected, and the process returns. In S28, an alarm is issued, and then the process proceeds to S29.

In S29, it is determined whether a cancel signal for stopping the alarm has been generated.
If the result of the determination in S29 is that a cancel signal has been generated, the driver has performed an operation to generate a cancel signal, so the count value of the first counter C1 is set to 0 (S30), and then the process returns. If the result of the determination in S29 is that a cancel signal has not been generated, the driver is not in a state where cancellation is possible, so the process proceeds to S31. In S31, abnormality sequence control is executed, and then the process returns. In this S31, the count values of the first and second counters C1 and C2 are each set to 0.

In S32, since the driver's posture has been intentionally changed and this does not cause any problems in terms of the driver's consciousness or driving operation, an alarm is issued as a warning, and then the process proceeds to S33.
In S33, it is determined whether a cancel signal for stopping the alarm has been generated.
If the result of the determination in S33 is that a cancel signal has been generated, the driver has noticed the warning and performed an operation to generate a cancel signal, so the warning is stopped (S34), and then the process returns.
If the result of the determination in S33 is that the cancel signal is not generated, the driver has not noticed the warning, so the warning continues.

Next, the operation and effects of the driver state determination method and the device M will be described.
According to this driver state determination method, there are provided a peripheral information acquisition process S1 for acquiring peripheral information relating to the vehicle's surrounding conditions, a field of view image acquisition process S2 for acquiring a field of view image that appears in the driver's field of view, an attraction distribution generation process S3 for generating a driver's attention distribution for the field of view image created in the field of view image acquisition process S2, a gaze behavior acquisition process S4 for acquiring the driver's gaze behavior, a gaze determination index calculation process S5 for calculating a saliency index, a saccade amplitude index, and a saccade frequency index as multiple gaze determination indexes for the driver based on the peripheral information, field of view image, attraction distribution, and gaze behavior, a head posture acquisition process S21 for acquiring the driver's facial direction and head position, and driver abnormal state determination processes S22, S23, S27 for determining whether the driver is in an abnormal state or not, and therefore multiple gaze determination indexes related to the driver's level of attention can be included in the determination conditions for the driver's abnormal state.
In the driver abnormal state determination processes S22, S23, and S27, if the driver's head posture acquired in the head posture acquisition process S21 is abnormal and the driver's multiple gaze determination indices calculated in the gaze determination index calculation process S5 are all normal, it is determined that the driver has intentionally lost posture, making it possible to determine whether a driver has an inappropriate driving posture even if there are no problems with their consciousness or driving operation.
The driver abnormal state determination steps S22, S23, and S27 determine that the driver is in an abnormal state if the head posture of the driver acquired in the head posture acquisition step S21 is abnormal and any of the multiple gaze determination indices of the driver calculated in the gaze determination index calculation step S5 is abnormal, so that it is possible to determine with high accuracy whether the driver is in an abnormal state or not from the driver's posture deviation. Therefore, since the abnormal state of the driver is determined based on the driver's eye closure state, abnormal head posture, and multiple gaze determination indices related to the driver's attention level, it is possible to improve the accuracy of determining the abnormal state of a driver who cannot drive normally.

The driver abnormal state determination process S22 determines whether the driver's head posture is abnormal based at least on the driver's face direction angle, making it easy to determine whether the driver's head posture is abnormal.

The driver abnormal state determination process S27 determines that the driver is in an abnormal state if the driver's head posture acquired in the head posture acquisition process S21 is abnormal and any of the driver's multiple gaze determination indices calculated in the gaze determination index calculation process S5 is abnormal for a period corresponding to the first determination threshold L1. This allows for reliable determination of the driver's abnormal state.

The gaze determination indices are the saliency index, saccade amplitude index, and saccade frequency index, so it is possible to determine the driver's abnormal state based on the driver's level of attention.

This driver condition determination device M detects an abnormal state of the driver of a vehicle and includes an external camera 10 that acquires peripheral information related to the situation around the vehicle, a gaze determination unit 30 that corresponds to a field of view image acquisition means that acquires a field of view image that appears in the driver's field of view, a saliency determination unit 30a that generates a visual attention distribution of the driver for the field of view image created by the field of view image acquisition means, a saccade determination means 30b that acquires the driver's gaze behavior, a gaze determination unit 30 that corresponds to a gaze determination index calculation means that calculates a saliency index, a saccade amplitude index, and a saccade frequency index as multiple gaze determination indexes of the driver based on the peripheral information, field of view image, visual attention distribution, and gaze behavior, an internal camera 11 that acquires the driver's facial direction and head position, and an abnormality determination unit 33c that determines whether the driver is in an abnormal state or not.Therefore, multiple gaze determination indexes related to the driver's level of attention can be included in the determination conditions for the driver's abnormal state.
If the driver's head posture acquired by the internal camera 11 is abnormal and the multiple gaze determination indices of the driver calculated by the gaze determination unit 30, which corresponds to the gaze determination index calculation means, are all normal, the abnormality determination unit 33c determines that the driver has intentionally lost posture, and is therefore able to determine that a driver has an inappropriate driving posture even if there are no problems with their state of consciousness or driving operation.
The abnormality determination unit 33c determines that the driver is in an abnormal state if the head posture of the driver acquired by the internal camera 11 is an abnormal posture and any of the plurality of gaze determination indices of the driver calculated by the gaze determination unit 30, which corresponds to the gaze determination index calculation means, is abnormal, so that it is possible to determine with high accuracy whether the driver is in an abnormal state or not from the driver's posture deviation. Therefore, since the abnormal state of the driver is determined based on the driver's eye closure state, abnormal head posture, and a plurality of gaze determination indices related to the driver's attention level, it is possible to improve the accuracy of determining the abnormal state of a driver who cannot drive normally.

In addition, those skilled in the art may implement the above-described embodiments in various modified forms without departing from the spirit of the present invention, and the present invention also encompasses such modifications.

1 Detection means 10 External camera 11 Internal camera 30 Line of sight determination unit 30a Saliency determination unit 30b Saccade determination unit 33c Abnormality determination unit M Driver state determination device

Claims (4)

A driver condition determination method for detecting an abnormal condition of a driver of a vehicle, comprising:
performing a control process for determining whether the driver has eyes that are closed and a control process for determining a head posture of the driver in parallel based on a captured image of the driver;
The eye closure determination control process determines that the driver is in an abnormal state when the driver does not respond to a warning issued when the driver's eyes remain closed, and
The head posture determination control process includes:
a surrounding information acquisition step of acquiring surrounding information relating to a surrounding situation of the vehicle;
a field of view image acquisition step of acquiring a field of view image corresponding to the driver's field of view based on the peripheral information ;
an attractiveness distribution generating step of generating an attractiveness distribution of the driver with respect to the field of view image;
a gaze behavior acquisition step of acquiring a gaze behavior of the driver based on the captured image and the field of view image ;
a gaze determination index calculation step of calculating a saliency index, a saccade amplitude index, and a saccade frequency index as a plurality of gaze determination indexes of the driver based on the peripheral information, the field of view image, the visual attraction distribution, and the gaze behavior;
a head posture acquiring step of acquiring a facial orientation and a head position of the driver based on the captured image ;
a driver abnormal state determination step of determining whether the driver is in an abnormal state,
The driver abnormal state determination step includes:
determining that the driver has intentionally lost posture when the head posture of the driver acquired in the head posture acquisition step is abnormal and the plurality of gaze determination indices of the driver calculated in the gaze determination index calculation step are all normal ;
a driver state determination method characterized by determining that the driver is in an abnormal state when the head posture of the driver acquired in the head posture acquisition step is an abnormal posture and any of the plurality of gaze determination indices of the driver calculated in the gaze determination index calculation step is abnormal. The driver condition determination method described in claim 1, characterized in that the driver abnormal condition determination step determines whether the head posture is abnormal based at least on the driver's face direction angle. The driver state determination method described in claim 1 or 2, characterized in that the driver abnormal state determination process determines that the driver is in an abnormal state if the driver's head posture acquired in the head posture acquisition process is abnormal and any of the driver's multiple gaze determination indices calculated in the gaze determination index calculation process is abnormal, for a predetermined period of time. A driver condition determination device for detecting an abnormal condition of a vehicle driver,
a surrounding information acquisition means for acquiring surrounding information relating to the surrounding conditions of the vehicle;
a driving operation amount acquisition means for acquiring a driving operation amount of the driver;
a field of view image acquisition means for acquiring a field of view image corresponding to the driver's field of view based on the peripheral information;
an attractiveness distribution generating means for generating an attractiveness distribution of the driver with respect to the field of view image;
a gaze behavior acquisition means for acquiring a gaze behavior of the driver based on an image captured by an internal camera that captures an image of the driver;
a gaze determination index calculation means for calculating a saliency index, a saccade amplitude index, and a saccade frequency index as a plurality of gaze determination indexes of the driver based on the peripheral information, the field of view image, the attractiveness distribution, and the gaze behavior;
a head posture acquisition means for acquiring a facial orientation and a head position of the driver based on the captured image;
a driver abnormal state determination means for determining whether the driver is in an abnormal state;
the driver abnormal state determination means performs a control process for determining whether the driver has closed eyes and a control process for determining a head posture of the driver in parallel based on the captured image,
In the eye closure determination control process, when the driver does not respond to a warning issued when the driver's eyes remain closed based on the driving operation amount and the captured image, it is determined that the driver is in an abnormal state;
The head posture determination control process determines that the driver has intentionally lost posture if the head posture of the driver acquired by the head posture acquisition means is an abnormal posture and the multiple gaze determination indices of the driver calculated by the gaze determination index calculation means are all normal, and determines that the driver is in an abnormal state if the head posture of the driver acquired by the head posture acquisition means is an abnormal posture and any of the multiple gaze determination indices of the driver calculated by the gaze determination index calculation means is abnormal.