JP6992417B2 - Driver status detector - Google Patents

Description

The present invention relates to a driver state detection device.

Conventionally, a device for detecting a driver state such as inattentiveness by using an image pickup means provided in a vehicle is known. For example, in the inattentive determination device described in Patent Document 1, the driver's line of sight is detected from the driver's face image output by the imaging means, and the driver is inattentive when the detected line of sight is out of a predetermined range. Is determined.

Japanese Unexamined Patent Publication No. 2012-08769

By the way, when the lane of a vehicle is changed, the driver needs to shift his / her line of sight from the front to see the lane of the destination for safety confirmation. However, in the inattentive determination device described in Patent Document 1, it is determined that the driver is inattentive even though the driver is facing sideways or diagonally backward for safety confirmation.

In view of the above problems, an object of the present invention is to prevent a driver from erroneously determining that he / she is looking aside when the driver confirms safety for changing lanes.

In order to solve the above problems, in the present invention, a driver monitor camera that photographs the face of a driver of a vehicle and generates a face image of the driver, and detects the face orientation or the line of sight of the driver based on the face image. The driver state determination unit includes a face information detection unit and a driver state determination unit that determines that the driver is looking aside when the driver's face orientation or line of sight satisfies the inattentiveness determination condition. The driver state determination unit is the vehicle. When it is determined that there is a high possibility that the lane change will be performed, the driver state detection device is provided, which relaxes the inattentive determination condition or does not perform the inattentive determination.

According to the present invention, when the driver confirms the safety due to a lane change, it is possible to prevent the driver from erroneously determining that he / she is looking aside.

FIG. 1 is a block diagram showing a configuration of a driver state detection device according to the first embodiment of the present invention. FIG. 2 is a diagram schematically showing the inside of a vehicle equipped with a driver state detection device. FIG. 3 is a flowchart showing a control routine of the inattentive determination condition setting process in the first embodiment. FIG. 4 is a flowchart showing a control routine of the driver state determination process in the first embodiment. FIG. 5 is a flowchart showing a control routine of the lane change prediction process in the second embodiment. FIG. 6 is a flowchart showing a control routine of the driver state determination process in the second embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description, similar components are given the same reference numbers.

<First Embodiment>
Hereinafter, the first embodiment of the present invention will be described with reference to FIGS. 1 to 4. FIG. 1 is a block diagram showing a configuration of a driver state detection device according to the first embodiment of the present invention. The driver state detection device 1 is mounted on the vehicle and detects the state of the driver of the vehicle. The driver state detection device 1 includes a driver monitor camera 10 and an electronic control unit (ECU) 20.

FIG. 2 is a diagram schematically showing the inside of a vehicle equipped with a driver state detection device. The driver monitor camera 10 photographs the driver's face of the vehicle 80 (own vehicle) and generates a driver's face image. The driver monitor camera 10 is provided inside the vehicle 80. Specifically, as shown in FIG. 2, the driver monitor camera 10 is provided on the upper part of the steering column 81 of the vehicle 80. In FIG. 2, the projection range of the driver monitor camera 10 is shown by a broken line. The driver monitor camera 10 may be provided on the steering 82, the rearview mirror, the meter panel, the meter hood, or the like of the vehicle 80. Further, the driver state detection device 1 may include a plurality of driver monitor cameras 10.

The driver monitor camera 10 is composed of a camera and a floodlight. For example, the camera is a CMOS (complementary metal oxide semiconductor) camera or a CCD (charge-coupled device) camera, and the floodlight is an LED (light emitting diode). Further, the floodlight is preferably a near-infrared LED so that the driver's face can be photographed without causing discomfort to the driver even in low illuminance such as at night. For example, the floodlight is two near-infrared LEDs located on either side of the camera. Further, the camera may be provided with a filter such as a visible light cut filter. The driver's face image generated by the driver monitor camera 10 is transmitted from the driver monitor camera 10 to the ECU 20.

The ECU 20 is a microcomputer provided with a memory such as a read-only memory (ROM) and a random access memory (RAM), a central processing unit (CPU), an input port, an output port, a communication module, and the like. In this embodiment, one ECU 20 is provided, but a plurality of ECUs may be provided for each function. The ECU 20 has a face information detection unit 21 and a driver status determination unit 22.

The face information detection unit 21 detects the driver's face information. Specifically, the face information detection unit 21 detects the driver's face orientation or line of sight based on the driver's face image generated by the driver monitor camera 10.

The face information detection unit 21 detects the driver's face orientation by, for example, the following method. The face information detection unit 21 stores in advance the face shape data when the driver is facing the front. The face shape data may be general human face data or may be acquired for each driver. The face shape data is stored in the memory of the ECU 20. The face information detection unit 21 matches the generated face image of the driver with the face shape data. The face information detection unit 21 rotates the driver's face image so that the matching rate between the two is maximized, and detects the driver's face orientation from the rotation angle when the matching rate is maximized.

Further, the face information detection unit 21 may detect the driver's face orientation by the following method. The face information detection unit 21 stores in advance a plurality of face shape data having different face orientations of the driver. The face shape data may be general human face data or may be acquired for each driver. The face shape data is stored in the memory of the ECU 20. The face information detection unit 21 matches the generated face image of the driver with a plurality of face shape data. The face information detection unit 21 detects the face orientation of the face shape data having the maximum matching rate between the two as the face orientation of the driver. Further, the face information detection unit 21 may detect the face orientation of the driver by another known method as described in JP-A-2000-9767662 or JP-A-2003-44853.

The face information detection unit 21 detects the driver's line of sight by, for example, the following method. The face information detection unit 21 identifies a face region from the generated face image of the driver, and detects the face parts by extracting feature points of the face parts such as eyes, nose, and mouth. Further, the face information detection unit 21 detects the position of the Purkinje image (corneal reflex image) and the position of the center of the pupil, and detects the line of sight of the driver based on the positional relationship between the Purkinje image and the center of the pupil. The face information detection unit 21 may detect the driver's line of sight based on the positional relationship between the Purkinje image and the center of the pupil and the detected face orientation of the driver.

The driver status determination unit 22 determines the driver status. Specifically, the driver state determination unit 22 determines that the driver is looking aside when the face orientation or the line of sight of the driver detected by the face information detection unit 21 satisfies the inattentive determination condition. For example, the driver state determination unit 22 determines that the driver is looking aside when the driver's face orientation or line of sight is maintained outside the threshold range for the threshold time or longer.

When the driver status determination unit 22 determines that the driver is looking aside, the driver status determination unit 22 gives an alarm to the driver. For example, the driver state determination unit 22 gives an alarm to the driver visually or audibly via the Human Machine Interface (HMI) 30. The HMI 30 is an interface for inputting / outputting information between the driver and the vehicle 80. The HMI 30 is composed of, for example, a display for displaying character or image information, a speaker for generating sound, an operation button for a driver to perform an input operation, a touch panel, a microphone, and the like.

Since the human eye has an effective visual field, the left visual field is narrowed when the driver is facing the right side, and the right visual field is narrowed when the driver is facing the left side. Therefore, in order for the driver to monitor a wide range on both the left and right sides, the driver needs to be facing forward as much as possible. Therefore, the threshold range for inattentive determination is set to a predetermined range (for example, ± 20 to 40 °) centered on the front surface (0 °).

On the other hand, when the lane of the vehicle 80 is changed, the driver needs to look at the lane of the destination by shifting his / her face or line of sight from the front for safety confirmation. However, if the inattentive judgment conditions (threshold range and threshold time) are maintained even when the lane is changed, the driver is looking aside even though the driver is facing sideways or diagonally backward for safety confirmation. It is determined and the driver is alerted. In addition, the driver may confirm the safety even before the turn signal is activated and the lane is actually changed.

Therefore, in the present embodiment, when the driver state determination unit 22 determines that the lane change of the vehicle 80 is likely to be performed, the driver state determination unit 22 relaxes the inattentive determination condition. As a result, when the driver confirms the safety due to a lane change, it is possible to prevent the driver from erroneously determining that he / she is looking aside.

When the vehicle 80 is automatically driven, the automatic driving system proposes a lane change as needed, and the lane change is executed with the consent of the driver. Therefore, the driver state determination unit 22 determines that there is a high possibility that the lane change of the vehicle 80 will be performed when the lane change is proposed by the automatic driving system.

Further, when the vehicle 80 approaches the preceding vehicle, the lane is often changed to the overtaking lane in order to avoid a collision with the preceding vehicle. Therefore, the driver state determination unit 22 determines that there is a high possibility that the lane change of the vehicle 80 will be performed when the collision margin time (TTC: Time To Collection) with the preceding vehicle is within a predetermined time. The TTC is a predicted value of the time until the vehicle 80 collides with the preceding vehicle, and is obtained by dividing the relative distance between the vehicle 80 and the preceding vehicle by the relative speed between the vehicle 80 and the preceding vehicle.

The vehicle 80 is provided with another vehicle detection device 40, and the output of the other vehicle detection device 40 is transmitted to the ECU 20. The other vehicle detection device 40 detects other vehicles around the vehicle 80. The driver state determination unit 22 calculates the relative distance between the vehicle 80 and the preceding vehicle and the relative speed between the vehicle 80 and the preceding vehicle based on the output of the other vehicle detection device 40. The other vehicle detection device 40 is, for example, a millimeter wave radar, a laser radar, a stereo camera, or the like.

In addition, when the own lane disappears on the traveling route, it is necessary to change lanes in order to continue traveling. Therefore, the driver state determination unit 22 determines that there is a high possibility that the lane of the vehicle 80 will be changed when the own lane disappears within a predetermined distance in the traveling direction of the vehicle 80.

The vehicle 80 is provided with a navigation device 50, and the information stored in the navigation device 50 is transmitted to the ECU 20. The navigation device 50 has a GPS receiver, and detects the current position of the vehicle 80 by the GPS receiver. Further, the navigation device 50 stores map information. The map information includes road position information, road shape information (for example, type of curve and straight part, radius of curvature of curve, road slope, etc.), position information such as confluence and diversion points, and information such as road type. included.

The driver state determination unit 22 detects the disappearance of the own lane based on the information acquired from the navigation device 50. For example, the number of lanes on a running road decreases or the lane disappears due to merging. The driver state determination unit 22 may detect the disappearance of the own lane based on the information received from the outside of the vehicle 80.

In addition, if the vehicle cannot travel in its own lane on the travel route, it is necessary to change lanes in order to continue traveling. Therefore, the driver state determination unit 22 determines that there is a high possibility that the lane of the vehicle 80 will be changed when the own lane cannot travel within a predetermined distance in the traveling direction of the vehicle 80.

The vehicle 80 is provided with a road traffic information receiving device 60, and the road traffic information received by the road traffic information receiving device 60 is transmitted to the ECU 20. The road traffic information receiving device 60 receives road traffic information such as road regulation information due to accidents, construction, disasters, weather conditions, etc., and traffic jam information from an external communication center such as a road traffic information communication system center. The road traffic information receiving device 60 may receive road traffic information by vehicle-to-vehicle communication between the vehicle 80 and another vehicle. Further, the navigation device 50 may function as a road traffic information receiving device.

The driver state determination unit 22 detects that the own lane cannot travel based on the information acquired from the road traffic information receiving device 60. For example, due to construction work, accidents, fires, landslides, collapses, etc., the vehicle cannot drive in its own lane.

When the driver state determination unit 22 determines that there is a high possibility that the lane of the vehicle 80 will be changed, the driver state determination unit 22 extends the threshold range, for example. Specifically, when the driver state determination unit 22 determines that the vehicle 80 is likely to move to the right lane due to a lane change, the driver state determination unit 22 expands the threshold range on the right side (plus side). For example, the threshold range is extended to + 180 °. On the other hand, when the driver state determination unit 22 determines that the vehicle 80 is likely to move to the left lane due to the lane change, the driver state determination unit 22 expands the threshold range on the left side (minus side). For example, the threshold range is extended to -180 °.

Further, the driver state determination unit 22 may extend the threshold time instead of expanding the threshold range. For example, the threshold time is extended to a longer time than the predicted time until the lane change is completed. Further, the driver state determination unit 22 may extend the threshold range and extend the threshold time.

<Inattentive judgment condition setting process>
Hereinafter, the control for detecting the inattentiveness of the driver by using the driver state detection device 1 will be described in detail with reference to the flowcharts of FIGS. 3 and 4. FIG. 3 is a flowchart showing a control routine of the inattentive determination condition setting process in the first embodiment. This control routine is repeatedly executed by the ECU 20 at predetermined time intervals. In this control routine, inattentive judgment conditions (threshold value determination and threshold time) are set.

First, in step S101, it is determined whether or not the automatic driving system has proposed a lane change. If it is determined by the autonomous driving system that no lane change has been proposed, the control routine proceeds to step S102.

In step S102, it is determined whether or not the TTC is T1 or less for a predetermined time. The predetermined time T1 is, for example, 1 to 5 seconds. If it is determined that the TTC is longer than the predetermined time T1, the control routine proceeds to step S103.

In step S103, it is determined whether or not the own lane disappears within a predetermined distance in the traveling direction of the vehicle 80. The predetermined distance is, for example, 100 to 800 m. If it is determined that the own lane does not disappear within a predetermined distance in the traveling direction of the vehicle 80, the control routine proceeds to step S104.

In step S104, it is determined whether or not the own lane cannot travel within a predetermined distance in the traveling direction of the vehicle 80. The predetermined distance is, for example, 100 to 800 m. If it is determined that the own lane can travel within a predetermined distance in the traveling direction of the vehicle 80, the control routine proceeds to step S105.

In step S105, the threshold range is initialized. That is, the threshold range is returned to the original value before expansion. Then, in step S106, the threshold time is initialized. That is, the threshold time is returned to the original value before the extension. After step S106, this control routine ends.

On the other hand, if the determination in any one of steps S101 to S104 is affirmed, it is determined that there is a high possibility that the lane change of the vehicle 80 will be performed, and the control routine proceeds to step S107. In step S107, it is determined whether or not the predicted moving direction of the lane change is on the right side.

For example, when the automatic driving system proposes a lane change to the right side, it is determined that the predicted moving direction of the lane change is the right side. On the other hand, when the automatic driving system proposes a lane change to the left side, it is determined that the predicted moving direction of the lane change is the left side. Further, when the TTC is T1 or less for a predetermined time and the overtaking lane is on the right side, it is determined that the predicted movement direction of the lane change is on the right side. On the other hand, when the TTC is T1 or less for a predetermined time and the overtaking lane is on the left side, it is determined that the predicted moving direction of the lane change is on the left side. Further, when the disappearance of the own lane is detected and the destination lane is on the right side, it is determined that the predicted movement direction of the lane change is on the right side. On the other hand, when the disappearance of the own lane is detected and the destination lane is on the left side, it is determined that the predicted movement direction of the lane change is on the left side. Further, when it is detected that the own lane becomes inoperable and the destination lane is on the right side, it is determined that the predicted movement direction of the lane change is on the right side. On the other hand, when it is detected that the own lane becomes inoperable and the destination lane is on the left side, it is determined that the predicted movement direction of the lane change is on the left side.

If it is determined in step S107 that the predicted moving direction of the lane change is on the right side, the control routine proceeds to step S108. In step S108, the threshold range on the right side is expanded. Then, in step S109, the threshold time is extended. After step S109, this control routine ends.

On the other hand, if it is determined in step S107 that the predicted moving direction of the lane change is on the left side, the control routine proceeds to step S110. In step S110, the threshold range on the left side is expanded. Then, in step S111, the threshold time is extended. After step S111, this control routine ends.

If the TTC decreases sharply, there is a high possibility that the lane will be changed to the overtaking lane. Therefore, in step S102, it is determined whether or not the amount of decrease in the TTC per unit time is equal to or greater than a predetermined value. May be done.

<Driver status judgment process>
FIG. 4 is a flowchart showing a control routine of the driver state determination process in the first embodiment. This control routine is repeatedly executed by the ECU 20 at predetermined time intervals. In this control routine, it is determined whether or not the driver is looking aside.

First, in step S201, the driver's face image is acquired from the driver monitor camera 10. The driver's face image is generated by the driver monitor camera 10.

Then, in step S202, the driver's face orientation or line of sight is detected based on the driver's face image. For example, the driver's face orientation or line of sight is detected by any of the methods described above.

Next, in step S203, it is determined whether or not the driver's face orientation or line of sight is within the threshold range. The threshold range is set in the control routine of FIG. If it is determined that the driver's face orientation or line of sight is within the threshold range, the control routine proceeds to step S208. In step S208, the integrated time ET is reset to zero. After step S208, this control routine ends.

On the other hand, if it is determined in step S203 that the face orientation or the line of sight is out of the threshold range, the control routine proceeds to step S204. In step S204, the integrated time ET is updated. Specifically, the value obtained by adding the minute time Δt to the integrated time ET is taken as a new integrated time ET. The integrated time ET is the time during which the driver's face orientation or line of sight is maintained outside the threshold range. The initial value of the integrated time ET is zero. Further, the minute time Δt is a value corresponding to the execution interval of this control routine.

Next, in step S205, it is determined whether or not the integrated time ET is equal to or greater than the threshold time TF. The threshold time Tref is set in the control routine of FIG. If it is determined that the integrated time ET is less than the threshold time Tref, this control routine ends.

On the other hand, if it is determined that the integrated time ET is equal to or greater than the threshold time TF, the control routine proceeds to step S206. In step S206, it is determined that the driver is looking aside. Then, in step S207, an alarm is given to the driver via the HMI 30 for a predetermined time. Then, in step S208, the integrated time ET is reset to zero. After step S208, this control routine ends.

Note that steps S106, S109 and S111 in FIG. 3 may be omitted. In this case, the threshold time is fixed regardless of whether or not the lane is changed. Further, step S105, step S108 and step S110 in FIG. 3 may be omitted. In this case, the threshold range is constant regardless of whether or not the lane is changed. That is, when it is determined that the lane change of the vehicle 80 is likely to be performed, at least one of the threshold range and the threshold time is changed.

<Second embodiment>
The configuration and control of the driver state detection device according to the second embodiment are basically the same as the configuration and control of the driver state detection device according to the first embodiment, except for the points described below. Therefore, hereinafter, the second embodiment of the present invention will be described focusing on the parts different from the first embodiment.

In the second embodiment, the driver state determination unit 22 does not perform an inattentive determination when it is determined that there is a high possibility that the lane change of the vehicle 80 will be performed. As a result, when the driver confirms the safety due to a lane change, it is possible to prevent the driver from erroneously determining that he / she is looking aside.

<Lane change prediction processing>
FIG. 5 is a flowchart showing a control routine of the lane change prediction process in the second embodiment. This control routine is repeatedly executed by the ECU 20 at predetermined time intervals. In this control routine, it is determined whether or not there is a high possibility that the lane change of the vehicle 80 will be performed.

In steps S301 to S304, the same determination as in steps S101 to S104 of FIG. 3 is performed. If all the determinations in steps S301 to S304 are denied, it is determined that there is a low possibility that the lane change of the vehicle 80 will be performed, and the control routine proceeds to step S305. In step S305, the lane change flag Frc is set to zero. The initial value of the lane change flag Frc is zero. After step S305, this control routine ends.

On the other hand, if the determination in any one of steps S301 to S304 is affirmed, it is determined that there is a high possibility that the lane change of the vehicle 80 will be performed, and the control routine proceeds to step S306. In step S306, the lane change flag Frc is set to 1. After step S306, this control routine ends.

<Driver status judgment process>
FIG. 6 is a flowchart showing a control routine of the driver state determination process in the second embodiment. This control routine is repeatedly executed by the ECU 20 at predetermined time intervals. In this control routine, it is determined whether or not the driver is looking aside.

First, in step S401, it is determined whether or not the lane change flag Frc is 1. If it is determined that the lane change flag Frc is zero, the control routine proceeds to step S402. Since steps S402 to S409 are the same as steps S201 to S208 in FIG. 4, the description thereof will be omitted. In this case, the threshold range and the threshold range are predetermined.

On the other hand, if it is determined in step S401 that the lane change flag Frc is 1, this control routine ends. That is, when it is determined that there is a high possibility that the lane of the vehicle 80 will be changed, the inattentive determination is not performed.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the claims.

1 Driver status detection device 10 Driver monitor camera 20 Electronic control unit (ECU)
21 Face information detection unit 22 Driver status determination unit 80 Vehicle

Claims (1)

A driver monitor camera that captures the driver's face and generates a driver's face image,
A face information detection unit that detects the face orientation or line of sight of the driver based on the face image,
It is provided with a driver state determination unit that determines that the driver is looking aside when the driver's face orientation or line of sight is maintained outside the threshold range for a threshold time or longer.
The driver state determination unit disappears when a lane change is proposed by the automatic driving system, when the collision margin time with the preceding vehicle is within a predetermined time, or within a predetermined distance in the traveling direction of the vehicle. In some cases, a driver state detection device that extends the threshold range or extends the threshold time.

Images