WO2020031873A1 - Visually-distracted driving detection device - Google Patents

Visually-distracted driving detection device Download PDF

Info

Publication number
WO2020031873A1
WO2020031873A1 PCT/JP2019/030415 JP2019030415W WO2020031873A1 WO 2020031873 A1 WO2020031873 A1 WO 2020031873A1 JP 2019030415 W JP2019030415 W JP 2019030415W WO 2020031873 A1 WO2020031873 A1 WO 2020031873A1
Authority
WO
WIPO (PCT)
Prior art keywords
driver
sight
line
vehicle
inattentive
Prior art date
Application number
PCT/JP2019/030415
Other languages
French (fr)
Japanese (ja)
Inventor
一輝 三浦
Original Assignee
オムロン株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by オムロン株式会社 filed Critical オムロン株式会社
Publication of WO2020031873A1 publication Critical patent/WO2020031873A1/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
    • A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
    • A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R11/00—Arrangements for holding or mounting articles, not otherwise provided for
    • B60R11/04—Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems

Definitions

  • the present invention relates to an inattentive driving detection device that detects that a driver of a vehicle is performing inattentive driving.
  • the driver's face is imaged with a camera, and based on the face direction and gaze direction obtained from the image, whether or not the driver is driving inattentively
  • a driver monitor for monitoring whether or not the operation is completed.
  • the driver monitor is provided in the driver's seat of the vehicle, and when an inattentive driving is detected from a captured image of the camera, outputs a warning to alert the driver.
  • the driver monitor has a function of detecting not only inattentive driving but also drowsy driving and poor physical condition of the driver.
  • Patent Documents 1 and 2 describe detecting a driver's line of sight with a driver monitor.
  • Patent Literature 1 while the timing of the warning output is changed according to the angle between the driver's line of sight and the traveling direction of the vehicle, the warning output is prohibited when the driver's line of sight faces the side mirror. By doing so, the alarm device operates properly.
  • Patent Literature 2 when the driver's line of sight is directed to the rearview mirror or the side mirror, it is determined that the driver has an intention to change the lane, and the steering control for changing the lane is executed. This allows the driver to change lanes in accordance with the driver's intention.
  • Patent Document 3 discloses a configuration in which a camera for detecting a pedestrian in the vicinity of a vehicle is provided on a side mirror in addition to a driver monitor for monitoring a driver.
  • the direction of the driver's face is detected by the camera of the driver monitor, and when it is determined that the driver does not visually recognize the side sufficiently when turning right or left, the pedestrian is detected by the camera of the side mirror. If detected, an alarm is issued to the driver.
  • the driver monitor even if the line of sight obtained from the driver's face image captured by the driver monitor deviates from the traveling direction of the vehicle, it may not be said that the driver is driving aside.
  • the driver looks at the side mirror and checks the vehicle status of the lane to be changed.
  • the driver's line of sight is directed to the side mirror, but the act of looking at the side mirror when changing lanes is necessary for safe driving of the vehicle. It is an act, not an inattentive driving. Nevertheless, if the driver monitor recognizes that the driver is looking aside, an alarm is erroneously output.
  • An inattentive driving detection device includes an inattentive driving camera provided at a driver's seat of a vehicle, the main camera capturing an image of a driver's face, and an inattentive driving direction obtained from an image captured by the main camera.
  • An inattentiveness determination unit that determines whether or not to drive
  • an alarm output unit that outputs an alarm when the inattentiveness determination unit determines that the driver is inattentive driving
  • installation of a specific in-vehicle component that is visually recognized by a driver while the vehicle is traveling.
  • an auxiliary camera that is provided at a location and captures the driver's face from a location different from the main camera.
  • the inattentiveness determination unit determines that the driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and that the driver's line of sight obtained from the captured image of the auxiliary camera indicates the direction of the specific vehicle-mounted component.
  • the driver's gaze is directed toward a specific vehicle-mounted component. If the time exceeds a predetermined reference time, it is determined that the driver is looking aside. It is determined that it is not driving.
  • the driver's line of sight obtained from the captured image of the main camera is facing the inattentive direction
  • the driver's line of sight obtained from the captured image of the auxiliary camera is the reference line. If the vehicle is facing the direction of a specific vehicle-mounted component within the time, the inattentive operation is regarded as an action necessary for driving, and the inattentive operation is not determined. Therefore, it is possible to avoid outputting an alarm for a driver's legitimate action.
  • the driver's line of sight is facing the direction of a specific vehicle-mounted component beyond the reference time, it is determined that inattentive driving is a dangerous act, and that inattentive driving is determined. For this reason, it is possible to output a warning and ensure safety for dangerous behavior of the driver.
  • the fact that the driver's line of sight is directed to a specific vehicle-mounted component is directly detected based on the line of sight obtained from an image captured by an auxiliary camera attached to the vehicle-mounted component. For this reason, the detection accuracy is improved as compared with the case where the detection is indirectly performed based on the line of sight obtained from the image captured by the main camera, and it is possible to clearly distinguish between the inattentive driving and the action that is not performed.
  • the driver's line of sight obtained from the captured image of the main camera is directed to the inattentive direction
  • the driver's line of sight obtained from the captured image of the auxiliary camera is a specific vehicle-mounted component.
  • the time when the driver's line of sight does not face the direction of the specific vehicle-mounted component exceeds a predetermined time shorter than the reference time, it is determined that the driver is looking aside, and the predetermined time is determined. If not, it may be determined that the driver is not looking aside.
  • the specific in-vehicle component visually recognized by the driver includes at least a left side mirror provided on the left side of the vehicle body, a right side mirror provided on the right side of the vehicle body, and a room mirror provided on the front side of the vehicle interior. It may include a mirror-based vehicle-mounted component composed of one.
  • the reference time is the time set as the first threshold
  • the inattentiveness determination unit determines that the driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and If the driver's line of sight obtained from the captured image of the camera is directed to the mirror-based vehicle-mounted component and the time during which the driver's line of sight is directed to the mirror-based vehicle-mounted component exceeds the first threshold, It is determined that the driving is inattentive driving, and if it does not exceed the first threshold value, it is determined that the driving is not inattentive driving.
  • the inattentiveness determination unit determines that the driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is the mirror-based vehicle-mounted component.
  • the driver does not face the direction
  • the specific in-vehicle component visually recognized by the driver is at least one of a display and a meter provided on the instrument panel of the vehicle in addition to or instead of the mirror-based in-vehicle component described above.
  • a display system in-vehicle component composed of two components.
  • the inattentiveness determination unit determines that the driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and that the driver's line of sight obtained from the captured image of the auxiliary camera is a display system in-vehicle component.
  • the third threshold value may be a value smaller than the first threshold value.
  • the inattentiveness determination unit determines that the driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is the direction of the display system onboard component.
  • the driver's gaze is not facing the direction of the in-vehicle display component when the driver's gaze exceeds the fourth threshold smaller than the third threshold, it is determined that the driver is looking aside, and the driver must not exceed the fourth threshold. It is determined that the driver is not looking aside.
  • the main camera, the inattentiveness determination unit, and the alarm output unit may be provided in a driver monitor that is mounted on the vehicle and monitors the state of the driver.
  • a first image processing unit that processes an image captured by a main camera, a second image processing unit that processes an image captured by an auxiliary camera, and a first image processing unit and a second image processing unit
  • a gaze detection unit that detects the direction of the gaze of the driver based on the processed image
  • the second image processing unit processes a single image processed by the plurality of auxiliary cameras. It may be a processing unit.
  • a single image processing unit that processes an image captured by the main camera and an image captured by the auxiliary camera may be provided.
  • the act of inattentive driving can be clearly distinguished from an unattended driving.
  • FIG. 1 is a block diagram of an inattentive driving detection device according to an embodiment of the present invention.
  • FIG. 2 is a block diagram showing the configuration of the main camera and the auxiliary camera.
  • FIG. 3 is a diagram illustrating imaging of a driver's face.
  • FIG. 4 is a diagram showing an example of installation of a main camera and an auxiliary camera.
  • FIG. 5A is a diagram illustrating an example of a driver's line of sight during traveling of the vehicle.
  • FIG. 5B is a diagram illustrating another example of the driver's line of sight while the vehicle is traveling.
  • FIG. 6 is a flowchart showing the operation of the inattentive driving detection device.
  • FIG. 7 is a continuation of the flowchart of FIG. FIG.
  • FIG. 8 is a block diagram of an inattentive driving detection device according to another embodiment of the present invention.
  • FIG. 9 is a block diagram of an inattentive driving detection device according to another embodiment of the present invention.
  • FIG. 10 is another example of a flowchart showing the operation of the inattentive driving detection device.
  • FIG. 11 is another example of a flowchart showing the operation of the inattentive driving detection device.
  • the inattentive driving detection apparatus 100 includes a driver monitor 50, auxiliary cameras 2 to 6, and image processing units 15 to 19.
  • the driver monitor 50 is a device that is mounted on a vehicle and monitors the state of the driver, and includes a main camera 1, an image processing unit 11, a line-of-sight detection unit 12, an inattentiveness determination unit 13, and an alarm output unit 14. I have.
  • the line-of-sight detection unit 12, the inattentiveness determination unit 13, and the alarm output unit 14 constitute the control unit 10.
  • the image processing unit 11 may be included in the control unit 10.
  • the control unit 10 includes, in addition to the line-of-sight detection unit 12 and the inattentiveness determination unit 13, blocks for detecting the movement of the driver's eyelids, falling asleep, and the like, but since these are not directly related to the present invention, The illustration is omitted.
  • the main camera 1 is provided in the driver's seat of the vehicle and captures a driver's face as described later.
  • the image processing unit 11 processes an image captured by the main camera 1, generates a driver's face image, and extracts feature points such as eyes from the face image.
  • the line-of-sight detection unit 12 detects the direction of the driver's line of sight based on the face image and the feature points obtained by the image processing unit 11.
  • the inattentiveness determination unit 13 determines the presence or absence of inattentive driving based on the driver's gaze direction detected by the gaze detection unit 12.
  • the alarm output unit 14 outputs an alarm when the inattentive driving determination unit 13 determines that the driver is looking aside.
  • the auxiliary cameras 2 to 6 are provided at locations where specific in-vehicle components, such as mirrors and displays, which are visually recognized by a driver while the vehicle is running, as described later.
  • the image processing units 15 to 19 are provided corresponding to the auxiliary cameras 2 to 6, respectively, and like the image processing unit 11, the characteristics such as the face to the eyes of the driver imaged by the auxiliary cameras 2 to 6 are provided. Extract points.
  • the outputs of the image processing units 15 to 19 are sent to the visual line detection unit 12 of the driver monitor 50 together with the output of the image processing unit 11.
  • vehicle information such as a running speed (vehicle speed) of the vehicle is input to an inattentiveness determination unit 13 of the driver monitor 50 from an ECU (electronic control unit) not shown.
  • the image processing unit 11 corresponds to the “first image processing unit” in the present invention
  • the image processing units 15 to 19 correspond to the “second image processing unit” in the present invention.
  • FIG. 2 shows the configuration of the main camera 1 and the auxiliary cameras 2 to 6.
  • Each of the cameras 1 to 6 includes a lens 20 that forms an optical system, an imaging element 21 that captures an image of a driver as a subject through the lens 20, and an amplification circuit that amplifies an analog electric signal photoelectrically converted by the imaging element 21.
  • a signal output circuit 23 that converts the signal amplified by the amplifier circuit 22 into a digital signal and outputs the digital signal as an image signal.
  • the image sensor 21 is formed of, for example, a CMOS image sensor.
  • the main camera 1 is provided with a light source for irradiating the driver's face with infrared light or the like as necessary (not shown).
  • FIG. 3 is a diagram showing a state in which the main camera 1 captures an image of the face of the driver DR sitting on the seat 34 in the driver seat 31 of the vehicle 30.
  • a broken line indicates an imaging range of the main camera 1.
  • the vehicle 30 is, for example, an automobile.
  • FIG. 4 is a view of the windshield W and the instrument panel 32 viewed from the interior of the vehicle in the traveling direction of the vehicle, and shows an example in which the main camera 1 and the auxiliary cameras 2 to 6 are installed.
  • the main camera 1 is installed at a location on the instrument panel 32 facing the driver DR, and captures an image of the driver DR's face through the steering 33 as shown in FIG.
  • the auxiliary cameras 2 to 6 are provided at locations where specific in-vehicle components are visually recognized by the driver DR, and capture images of the driver DR's face from a location different from the main camera 1.
  • the auxiliary camera 2 is attached to a left side mirror 35 provided on the left side of the vehicle body
  • the auxiliary camera 3 is attached to a right side mirror 36 provided on the right side of the vehicle body.
  • the auxiliary camera 4 is attached to a room mirror 37 provided in front of the vehicle interior (near the upper portion of the windshield W).
  • the auxiliary camera 5 is attached to a display 38 provided on the instrument panel 32.
  • the display 38 displays an image such as a car navigation or a television.
  • the auxiliary camera 6 is attached to a meter 39 provided on the instrument panel 32.
  • the meter 39 displays a running speed, an engine speed, and the like.
  • the left side mirror 35, the right side mirror 36, and the room mirror 37 are examples of the “mirror-based vehicle-mounted parts" in the present invention, and the display 38 and the meter 39 are examples of the "display-based vehicle-mounted parts" in the present invention. .
  • 5A and 5B show an example of the line of sight of the driver DR while the vehicle is traveling.
  • the driver DR is imaged by the main camera 1
  • a face image is generated by the image processing unit 11 (FIG. 1)
  • the line of sight (the line of sight viewed from the main camera 1) is detected by the line of sight detection unit 12.
  • the driver DR is imaged by the auxiliary cameras 2 to 6
  • a face image is generated by the image processing units 15 to 19, and the line of sight (the line of sight viewed from the auxiliary cameras 2 to 6) is detected by the line of sight detection unit 12.
  • the inattentiveness determination unit 13 determines whether or not the driver DR is performing inattentive driving based on each line of sight detected by the line of sight detection unit 12. Hereinafter, this determination method will be specifically described.
  • the example of the line of sight shown in FIG. 5A is based on the legitimate action of the driver DR required for driving the vehicle, and in any case, it does not basically correspond to inattentive driving.
  • FIG. 5A (a) shows a state where the driver DR is facing the front, and the line of sight coincides with the traveling direction.
  • the driver DR is looking straight ahead, so it is not necessary to refer to the images captured by the auxiliary cameras 2 to 6 and the driver DR does not correspond to inattentive driving. Is determined.
  • 5A (b) shows a state in which the driver DR is looking at the left side mirror 35 (FIG. 4), and his line of sight is obliquely directed to the lower left with respect to the traveling direction.
  • the driver DR looks aside to the left, but in the image captured by the auxiliary camera 2, the driver DR points in the direction of the left side mirror 35. It is in a state of facing.
  • the driver DR looking at the left side mirror 35 is an action necessary for turning left or changing lanes to the left lane, so that it is determined that the driver DR does not correspond to inattentive driving in this case.
  • the driver keeps looking at the left side mirror 35 for more than a certain period of time, the driver becomes careless in the front and hinders safe driving.
  • FIG. 5A (c) shows a state in which the driver DR is looking at the right side mirror 36 (FIG. 4), and the line of sight is obliquely directed to the lower right with respect to the traveling direction.
  • the driver DR looks sideways to the right, but in the image captured by the auxiliary camera 3, the driver DR points in the direction of the right side mirror 36. It is in a state of facing.
  • the driver DR looks at the right side mirror 36 because it is necessary to turn right or change lanes to the right lane, it is determined that the driver DR does not correspond to inattentive driving in this case.
  • the driver keeps looking at the right side mirror 36 for more than a certain period of time, the driver becomes careless in the forward direction and hinders safe driving.
  • FIG. 5A (d) shows a state in which the driver DR is looking at the rear-view mirror 37 (FIG. 4), and the line of sight is directed obliquely to the upper left with respect to the traveling direction.
  • the driver DR looks sideways upward, but in the image captured by the auxiliary camera 4, the driver DR faces the room mirror 37. State. Then, the driver DR looks at the rear-view mirror 37 because it is necessary to confirm the vehicle traveling behind, and in this case, it is determined that the driver DR does not correspond to inattentive driving. However, if the user continues to look at the rear-view mirror 37 for a certain period of time, the driver becomes careless in the forward direction and hinders safe driving.
  • FIG. 5A (e) shows a state in which the driver DR is looking at the display 38 (FIG. 4), and his line of sight is obliquely directed to the lower left with respect to the traveling direction.
  • the driver DR looks aside at the lower side, but in the image captured by the auxiliary camera 5, the driver DR faces the display 38.
  • the driver DR looks at the display 38, for example, is an action necessary for confirming the traveling route by a car navigation system, it is determined that the driver DR does not correspond to inattentive driving in this case.
  • the driver becomes careless in the front and hinders safe driving.
  • FIG. 5A (f) shows a state in which the driver DR is looking at the meter 39 (FIG. 4), and the line of sight is obliquely directed to the lower left with respect to the traveling direction.
  • the driver DR looks aside at the lower side, but in the image captured by the auxiliary camera 6, the driver DR faces the direction of the meter 39. State. Then, since the driver DR looks at the meter 39 because it is necessary to check the traveling speed and the like, in this case, it is determined that the driver DR does not correspond to the inattentive driving. However, if the user continues to look at the meter 39 for more than a certain period of time, the driver becomes careless in the front and hinders safe driving.
  • the example of the line of sight shown in FIG. 5B is based on the dangerous action of the driver DR that is not necessary for driving the vehicle, and in any case, basically corresponds to the inattentive driving.
  • FIG. 5B (u) shows a state in which the driver DR is looking at a scene outside the vehicle on the left side, and the line of sight is directed leftward with respect to the traveling direction.
  • the driver DR looks aside to the left, and in the image captured by the auxiliary cameras 2 to 6, the driver DR is selected from any of the in-vehicle components 35 to 39. Is also not oriented. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is pointing to the left is extremely short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
  • FIG. 5B (v) shows a state in which the driver DR is looking at the scenery outside the vehicle on the right side, and the line of sight is to the right with respect to the traveling direction.
  • the driver DR looks aside to the right, and in the image captured by the auxiliary cameras 2 to 6, the driver DR is selected from any of the in-vehicle components 35 to 39. Is also not oriented. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is pointing to the right is very short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
  • FIG. 5B (w) shows a state where the driver DR is looking up at the sky, and the line of sight is directed upward with respect to the traveling direction.
  • the driver DR in the image captured by the main camera 1, the driver DR is looking aside at the upper side, and in the images captured by the auxiliary cameras 2 to 6, the driver DR is selected from any of the in-vehicle components 35 to 39. Is also not oriented. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is facing upward is very short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
  • FIG. 5B (x) shows a state in which the driver DR is looking at the feet and the like, and the line of sight is directed downward with respect to the traveling direction.
  • the driver DR in the image captured by the main camera 1, the driver DR is looking aside at the lower side, and in the images captured by the auxiliary cameras 2 to 6, the driver DR is connected to the in-vehicle components 35 to 39. It is in a state where it is not oriented in any direction. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is facing downward is extremely short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
  • FIG. 5B (y) shows a state in which the driver DR is looking at an obliquely upper left outside scenery of the vehicle, and the line of sight is obliquely directed to the upper left with respect to the traveling direction.
  • the driver DR in the image captured by the main camera 1, the driver DR is looking aside at the upper side, and in the images captured by the auxiliary cameras 2 to 6, the driver DR is selected from any of the in-vehicle components 35 to 39. Is also not oriented. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is pointing diagonally to the upper left is very short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
  • 5B (z) shows a state in which the driver DR is looking at a portion below the display 38, and the line of sight is obliquely directed to the lower left with respect to the traveling direction.
  • the driver DR in the image captured by the main camera 1, the driver DR is looking aside at the lower side, and in the images captured by the auxiliary cameras 2 to 6, the driver DR is connected to the in-vehicle components 35 to 39. It is in a state where it is not oriented in any direction. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is pointing diagonally to the lower left is very short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
  • step S1 the main camera 1 and the auxiliary cameras 2 to 6 image the driver DR's face.
  • step S2 the image processing units 11, 15 to 19 generate a face image of the driver DR.
  • step S3 the gaze detecting unit 12 detects the gaze of the driver DR from the face image.
  • step S4 the inattentiveness determination unit 13 determines the driving direction of the driver DR based on the line of sight in the face image captured by the main camera 1 and the vehicle information acquired from the ECU out of the line of sight detected in step S3. It is determined whether the player looks aside to the left. More specifically, when the vehicle speed is equal to or higher than a predetermined value (for example, equal to or higher than 10 km / h) and the line of sight of the driver DR is directed leftward with respect to the traveling direction, the inattentiveness determination unit 13 determines that the driver is looking leftward. If the driver is looking aside to the left, the process proceeds to step S5. If the driver is not looking aside to the left, the process proceeds to step S9.
  • a predetermined value for example, equal to or higher than 10 km / h
  • step S5 the inattentiveness determination unit 13 determines whether the driver DR is looking at the left side mirror 35. This determination is performed based on the direction of the driver's line of sight detected by the line-of-sight detection unit 12 from the driver's DR face image generated by the image processing unit 15 captured by the auxiliary camera 2. As a result of the determination, when the driver DR is looking at the left side mirror 35 as shown in FIG. 5A (b), the process proceeds to step S6a. When the driver DR does not look at the left side mirror 35 as shown in FIG. 5B (u), the process proceeds to step S6b.
  • a threshold T1 (first threshold) for inattentive driving determination is set.
  • This threshold T1 is stored in advance in a memory (not shown) of the control unit 10, and corresponds to the “reference time” of the present invention.
  • the inattentiveness determination unit 13 determines whether the inattentiveness time has exceeded the threshold T1.
  • the inattentive time in this case is the time during which the driver DR is looking at the left side mirror 35 and is measured based on the image captured by the auxiliary camera 2.
  • a threshold T2 (second threshold) for inattentive driving determination is set.
  • This threshold value T2 is also stored in the memory (not shown) of the control unit 10 in advance.
  • the threshold value T2 is smaller than the threshold value T1 (T1> T2), and corresponds to the "predetermined time shorter than the reference time" of the present invention.
  • the inattentiveness determination unit 13 determines whether the inattentiveness time has exceeded the threshold T2.
  • the inattentive time in this case is the time during which the driver DR looks aside to the left without looking at the left side mirror 35, and is measured based on the image captured by the main camera 1.
  • step S7 If the result of the determination in step S7 is that the inattentive time exceeds the threshold value T1 or T2, the process proceeds to step S8, and if not, the process returns to the first step S1.
  • step S8 the inattentiveness determination unit 13 determines that the driver is looking aside in the left direction, and the alarm output unit 14 outputs an alarm. This warning is output in a form such as voice or display (the same applies to steps S13, S18, S23, and S28).
  • the inattentiveness determination unit 13 determines whether the driver DR looks aside to the right in step S9. This determination is made based on the line of sight in the face image captured by the main camera 1 and the vehicle information acquired from the ECU. Specifically, when the vehicle speed is equal to or higher than a certain value and the driver DR's line of sight is directed to the right with respect to the traveling direction, the inattentiveness determination unit 13 determines that the driver is looking rightward. If the driver is looking aside to the right, the process proceeds to step S10. If the driver is not looking aside to the right, the process proceeds to step S14.
  • step S10 the inattentiveness determination unit 13 determines whether the driver DR is looking at the right side mirror 36. This determination is made based on the direction of the driver's line of sight detected by the line of sight detection unit 12 from the driver DR's face image captured by the auxiliary camera 3 and generated by the image processing unit 16. As a result of the determination, when the driver DR is looking at the right side mirror 36 as shown in FIG. 5C, the process proceeds to step S11a. In addition, as shown in FIG. 5B (v), when the driver DR does not look at the right side mirror 36, the process proceeds to step S11b.
  • step S11a the above-described threshold value T1 is set.
  • step S12 the inattentiveness determination unit 13 determines whether or not the inattentive time has exceeded the threshold value T1.
  • the inattentive time in this case is a time during which the driver DR is looking at the right side mirror 36, and is measured based on an image captured by the auxiliary camera 3.
  • step S11b the above-described threshold value T2 is set.
  • step S12 the inattentiveness determination unit 13 determines whether the inattentiveness time has exceeded the threshold T2.
  • the inattentive time in this case is a time during which the driver DR looks aside to the right without looking at the right side mirror 36, and is measured based on the captured image of the main camera 1.
  • step S12 determines that the inattentive time exceeds the threshold value T1 or T2
  • the process proceeds to step S13, and if not, the process returns to the first step S1.
  • step S13 the inattentiveness determination unit 13 determines that the driver is looking aside in the right direction, and the alarm output unit 14 outputs an alarm.
  • step S9 when the driver DR is not looking aside to the right, the inattentiveness determination unit 13 determines whether the driver DR is looking aside upward at step S14. This determination is made based on the line of sight in the face image captured by the main camera 1 and the vehicle information acquired from the ECU. More specifically, when the vehicle speed is equal to or higher than a certain value and the driver DR's line of sight is facing upward with respect to the traveling direction, the inattentiveness determination unit 13 determines that the driver is looking at the upper side. If the driver looks aside at the upper side, the process proceeds to step S15. If the driver does not look aside at the upper side, the process proceeds to step S19 (FIG. 7).
  • step S15 the inattentiveness determination unit 13 determines whether the driver DR is looking at the rear-view mirror 37. This determination is performed based on the direction of the driver DR's line of sight detected by the line of sight detection unit 12 from the driver's DR face image captured by the auxiliary camera 4 and generated by the image processing unit 17. As a result of the determination, when the driver DR is looking at the rear-view mirror 37 as shown in FIG. 5D, the process proceeds to step S16a. If the driver DR is not looking at the room mirror 37 as shown in FIG. 5B (w) (y), the process proceeds to step S16b.
  • step S16a the above-described threshold value T1 is set.
  • step S17 the inattentiveness determination unit 13 determines whether the inattentiveness time has exceeded the threshold T1.
  • the inattentive time in this case is a time during which the driver DR is looking at the rear-view mirror 37, and is measured based on an image captured by the auxiliary camera 4.
  • step S16b the above-described threshold value T2 is set.
  • step S17 the inattentiveness determination unit 13 determines whether or not the inattentiveness time has exceeded the threshold T2.
  • the inattentive time is a time during which the driver DR looks aside upward without looking at the room mirror 37, and is measured based on the image captured by the main camera 1.
  • step S17 If the result of determination in step S17 is that the inattentive time exceeds the threshold value T1 or T2, the process proceeds to step S18, and if not, the process returns to the first step S1.
  • step S18 the inattentiveness determination unit 13 determines that the driver is looking aside in an upward direction, and the alarm output unit 14 outputs an alarm.
  • step S14 when the driver DR does not look aside at the upper side, the inattentiveness determination unit 13 determines whether the driver DR looks down at the lower side at step S19 in FIG. This determination is made based on the line of sight in the face image captured by the main camera 1 and the vehicle information acquired from the ECU. More specifically, when the vehicle speed is equal to or higher than a certain value and the driver DR's line of sight is directed downward with respect to the traveling direction, the inattentiveness determination unit 13 determines that the driver is looking at the lower side. If the driver looks aside at the lower side, the process proceeds to step S20 and step S24. Steps S20 to S23 and steps S24 to S27 are executed in parallel. If the driver DR does not look down at the side in step S19, the process returns to step S1 in FIG.
  • step S20 the inattentiveness determination unit 13 determines whether the driver DR is looking at the display 38. This determination is performed based on the direction of the driver DR's line of sight detected by the line-of-sight detection unit 12 from the driver DR's face image captured by the auxiliary camera 5 and generated by the image processing unit 18. As a result of the determination, when the driver DR is looking at the display 38 as shown in FIG. 5A (e), the process proceeds to step S21a. When the driver DR does not look at the display 38 as shown in FIG. 5B (z), the process proceeds to step S21b.
  • step S21a a threshold value T3 smaller than the above-described threshold value T1 is set (T1> T3).
  • the inattentiveness determination unit 13 determines whether or not the inattentive time has exceeded the threshold T3.
  • the inattentive time in this case is the time during which the driver DR is looking at the display 38, and is measured based on the image captured by the auxiliary camera 5.
  • step S21b a threshold T4 smaller than the threshold T3 is set.
  • the inattentiveness determination unit 13 determines whether or not the inattentiveness time has exceeded the threshold T4.
  • the inattentive time in this case is the time during which the driver DR looks aside downward without looking at the display 38, and is measured based on the image captured by the main camera 1.
  • step S22 If the inattentive time exceeds the threshold T3 or T4 as a result of the determination in step S22, the process proceeds to step S23, and if not, the process returns to step S1 in FIG.
  • step S23 the inattentiveness determination unit 13 determines that the vehicle is looking aside in the downward direction, and the alarm output unit 14 outputs an alarm.
  • step S24 the inattentiveness determination unit 13 determines whether the driver DR is looking at the meter 39. This determination is made based on the direction of the driver DR's line of sight detected by the line of sight detection unit 12 from the driver DR's face image captured by the auxiliary camera 6 and generated by the image processing unit 19. As a result of the determination, when the driver DR is looking at the meter 39 as shown in FIG. 5A (f), the process proceeds to step S25a. When the driver DR does not look at the meter 39 as shown in FIG. 5B (z), the process proceeds to step S25b.
  • step S25a the above-described threshold value T3 is set.
  • step S26 the inattentiveness determination unit 13 determines whether or not the inattentiveness time has exceeded the threshold value T3.
  • the inattentive time in this case is the time during which the driver DR is looking at the meter 39, and is measured based on the image captured by the auxiliary camera 6.
  • step S25b the above-described threshold value T4 is set.
  • step S26 the inattentiveness determination unit 13 determines whether or not the inattentive time has exceeded the threshold T4.
  • the inattentive time in this case is the time during which the driver DR looks aside downward without looking at the meter 39, and is measured based on the image captured by the main camera 1.
  • step S26 If the result of determination in step S26 is that the inattentive time has exceeded the threshold value T3 or T4, the process proceeds to step S27; otherwise, the process returns to step S1 in FIG.
  • step S27 the inattentiveness determination unit 13 determines that the driver is looking aside in the downward direction, and the alarm output unit 14 outputs an alarm.
  • the auxiliary cameras 2 to 6 are provided at places where the driver DR can view. Then, even if the line of sight of the driver DR obtained from the captured image of the main camera 1 is looking in the inattentive direction, the line of sight of the driver DR obtained from the captured images of the auxiliary cameras 2 to 6 remains at the reference time (threshold T1, In T3), when the vehicle is facing a specific vehicle-mounted component such as the mirrors 35 to 37, the display 38, and the meter 39, the inattentive driving is regarded as an action necessary for driving, and the inattentive driving is not determined.
  • the reference time threshold T1, In T3
  • the fact that the line of sight of the driver DR faces the direction of a specific vehicle-mounted component is directly determined based on the line of sight obtained from the captured images of the auxiliary cameras 2 to 6 attached to the vehicle-mounted component. Detected. For this reason, compared with the method of indirectly detecting based on the line of sight obtained from the image captured by the main camera 1 as in Patent Document 1, the detection accuracy is improved, and the act of looking aside and the act that is not so are clearly distinguished. It can be determined separately.
  • the line of sight of the driver DR obtained from the image captured by the main camera 1 faces in the inattentive direction, and the line of sight of the driver DR obtained from the images captured by the auxiliary cameras 2 to 6 is specified. If the vehicle component is not facing the direction of the vehicle-mounted component, it is determined whether the driver is looking aside or not depending on the time during which the vehicle component is not facing the direction of the vehicle-mounted component. For this reason, if the inattentive time is long, an alarm can be output to ensure safety. If the inattentive time is momentary that does not hinder safe driving, It is possible to suppress output of an alarm.
  • the line of sight of the driver DR obtained from the captured image of the main camera 1 is directed in the inattentive direction, and the line of sight of the driver DR obtained from the captured images of the auxiliary cameras 2 to 6 is
  • the vehicle faces the mirror-based vehicle-mounted components such as the mirrors 35 and 36 and the room mirror 37
  • the time in which the mirror-based vehicle-mounted components are directed exceeds the first threshold T1
  • the time during which the driver DR does not face the mirror-based vehicle-mounted component is set to the first threshold. If it exceeds the second threshold value T2 smaller than T1, it is determined that the driving is inattentive driving, and if it does not exceed the second threshold value T2, it is determined that it is not inattentive driving. For this reason, when the driver DR is looking at the mirror-based in-vehicle components, the threshold value can be increased to secure time for checking the side mirrors 35 and 36 and the room mirror 37. When the in-vehicle component is not viewed, safety can be ensured by reducing the threshold value and outputting an alarm in a short time.
  • the line of sight of the driver DR obtained from the image captured by the main camera 1 is directed in the inattentive direction, and the line of sight of the driver DR obtained from the images captured by the auxiliary cameras 2 to 6 is displayed on the display.
  • the vehicle faces the direction of the display system in-vehicle component such as 38 or the meter 39, if the time in which the direction of the display system in-vehicle component is directed exceeds the third threshold value T3 smaller than the first threshold value T1, it is the inattentive driving. If it does not exceed the third threshold value T3, it is determined that the driver is not looking aside.
  • the time during which the driver DR does not face the display system vehicle component is the third threshold value. If it exceeds the fourth threshold T4 smaller than T3, it is determined that the driver is looking aside, and if the driver does not exceed the fourth threshold T4, it is determined that the driver is not looking aside. For this reason, when the driver DR is looking at the display vehicle components, the threshold value can be increased to some extent to secure time for checking the display 38 and the meter 39. If the user does not see the warning, the threshold can be made as small as possible and an alarm can be output immediately to ensure safety.
  • the main camera 1 is a dedicated camera for the driver monitor 50.
  • the driver monitor 50 not only detects inattentiveness based on the line of sight, but also detects eyelid movements and falling asleep, as described above. Therefore, it is necessary to use a high-performance camera as the main camera 1.
  • the auxiliary cameras 2 to 6 detect that the driver DR is looking at the vehicle-mounted components 35 to 39 to which the auxiliary cameras 2 to 6 are attached, and it is sufficient if the driver DR can detect the line of sight of the driver DR. It does not need to be a high-performance camera like the main camera 1. Therefore, the cost of the auxiliary cameras 2 to 6 can be suppressed, and the inattentive driving detection device 100 can be configured at a low cost.
  • the image processing units 15 to 19 are provided corresponding to the plurality of auxiliary cameras 2 to 6, respectively.
  • a single image processing unit 40 for processing images captured by each of 2 to 6 may be provided.
  • This image processing unit 40 corresponds to the “second image processing unit” in the present invention. By doing so, it is not necessary to provide an image processing unit for each of the auxiliary cameras 2 to 6, so that the inattentive driving detection device 200 can be configured at lower cost.
  • images taken by the auxiliary cameras 2 to 6 may be processed by the image processing unit 11 of the driver monitor 50.
  • the inattentive driving detection device 300 can be configured at a lower cost.
  • FIGS. 6 and 7 show an example in which an inattentive driving determination (FIG. 7) for the display-system in-vehicle component is performed following the inattentive driving determination for the mirror-based in-vehicle component (FIG. 6). It may be executed separately.
  • FIG. 10 and FIG. 11 show flowcharts in that case.
  • the main camera 1 and the auxiliary cameras 2 to 4 image the face of the driver DR in step S1 in FIG.
  • steps S31 to S33 are added to the flowchart of FIG. It is the main camera 1 and the auxiliary cameras 5 and 6 that image the face of the driver DR in step S31.
  • the contents of steps S32 to S33 are the same as the contents of steps S2 to S3 in FIG.
  • both the inattentive driving determination for the mirror-based in-vehicle component and the inattentive driving determination for the display-based in-vehicle component are performed. Alternatively, only one of them may be performed.
  • the thresholds T1 and T2 for the side mirrors 35 and 36 are applied to the room mirror 37, but the thresholds may be different between the side mirrors 35 and 36 and the room mirror 37.
  • the thresholds T3 and T4 for the display 38 are applied to the meter 39, but the thresholds may be different between the display 38 and the meter 39.
  • the threshold value may be changed according to the display content of. For example, if the display 38 is a car navigation screen, the act of the driver DR looking at the display 38 during traveling is an act necessary for confirming a traveling route or the like. Set to T3. On the other hand, if the display 38 is a television screen, the act of the driver DR watching the display 38 while driving is a dangerous act unrelated to driving, and the threshold value is smaller than T3. (May be T4).
  • the display 38 and the meter 39 have been described as examples of the display system in-vehicle components. Further, as the in-vehicle components visually recognized by the driver DR while the vehicle is traveling, operation-related in-vehicle components may be included in addition to the mirror-based in-vehicle components and the display-based in-vehicle components.
  • the on-board operation system components include, for example, a switch for opening and closing a door, a switch for opening and closing a window, a switch for an air conditioner and audio, a change lever, and the like.
  • the inattentive driving detection device of the present invention is mounted on a four-wheeled vehicle.
  • the present invention is also applied to an inattentive driving detecting device mounted on another vehicle such as a truck or a bus. can do.

Abstract

A visually-distracted driving detection device (100) comprises: a main camera (1) that is provided to the driver seat of a vehicle and that takes an image of the face of a driver; a visual-distraction determination unit (13) for determining whether or not visually-distracted driving is being performed, on the basis of the direction of the line of sight of the driver obtained from the image taken by the main camera (1); an alert output unit (14) for outputting an alert when it is determined that visually-distracted driving is being performed; and auxiliary cameras (2 to 6) provided around the locations at which respective specific vehicle-mounted components to be visually recognized by the driver during running of the vehicle are disposed. In a case where the line of sight of the driver obtained from the image taken by the main camera (1) is oriented to a direction of visual-distraction and the line of sight of the driver obtained from an image taken by any of the auxiliary cameras (2 to 6) is oriented to the direction toward the corresponding one of the specific vehicle-mounted components, the visual-distraction determination unit (13) determines, if the period during which the line of sight of the driver is oriented in the direction toward the specific vehicle-mounted component exceeds a reference period, that visually-distracted driving is being performed, and determines, if this period does not exceed the reference period, that visually-distracted driving is not being performed.

Description

脇見運転検出装置Inattentive driving detection device
　本発明は、車両の運転手が脇見運転をしていることを検出する脇見運転検出装置に関する。 The present invention relates to an inattentive driving detection device that detects that a driver of a vehicle is performing inattentive driving.
　脇見運転による交通事故を未然に防止するために、運転手の顔をカメラで撮像し、その画像から得られる顔の向きや視線の方向などに基づいて、運転手が脇見運転をしているか否かを監視するドライバモニタが知られている。ドライバモニタは、車両の運転席に装備され、カメラの撮像画像から脇見運転が検出された場合は、警報を出力して運転手に注意を喚起する。また、ドライバモニタは、脇見運転だけでなく、居眠り運転や運転手の体調不良なども検出する機能を備えている。 In order to prevent traffic accidents caused by inattentive driving, the driver's face is imaged with a camera, and based on the face direction and gaze direction obtained from the image, whether or not the driver is driving inattentively There is known a driver monitor for monitoring whether or not the operation is completed. The driver monitor is provided in the driver's seat of the vehicle, and when an inattentive driving is detected from a captured image of the camera, outputs a warning to alert the driver. In addition, the driver monitor has a function of detecting not only inattentive driving but also drowsy driving and poor physical condition of the driver.
　特許文献１、２には、ドライバモニタで運転手の視線方向を検出することが記載されている。特許文献１では、運転手の視線方向と車両の進行方向とがなす角度に応じて警報出力のタイミングを変更する一方、運転手の視線がサイドミラーの方向を向いているときは警報出力を禁止することで、警報装置が適切に作動するようにしている。特許文献２では、運転手の視線がルームミラーまたはサイドミラーの方向を向いているときは、運転手に車線を変更する意志があると判定し、車線を変更するための操舵制御を実行することで、運転手の意図に合致した車線変更が行えるようにしている。 Patent Documents 1 and 2 describe detecting a driver's line of sight with a driver monitor. In Patent Literature 1, while the timing of the warning output is changed according to the angle between the driver's line of sight and the traveling direction of the vehicle, the warning output is prohibited when the driver's line of sight faces the side mirror. By doing so, the alarm device operates properly. According to Patent Literature 2, when the driver's line of sight is directed to the rearview mirror or the side mirror, it is determined that the driver has an intention to change the lane, and the steering control for changing the lane is executed. This allows the driver to change lanes in accordance with the driver's intention.
　また、特許文献３には、運転手を監視するドライバモニタ以外に、車両の近傍にいる歩行者を検出するためのカメラを、サイドミラーに設けた構成が記載されている。本文献では、ドライバモニタのカメラで運転手の顔の向きを検出し、右折または左折にあたって運転手が側方を十分に視認していないと判断される場合に、サイドミラーのカメラで歩行者が検出されれば、運転手に対して警報を発するようにしている。 Patent Document 3 discloses a configuration in which a camera for detecting a pedestrian in the vicinity of a vehicle is provided on a side mirror in addition to a driver monitor for monitoring a driver. In this document, the direction of the driver's face is detected by the camera of the driver monitor, and when it is determined that the driver does not visually recognize the side sufficiently when turning right or left, the pedestrian is detected by the camera of the side mirror. If detected, an alarm is issued to the driver.
　ところで、ドライバモニタで撮像した運転手の顔画像から得られる視線が、車両の進行方向からずれていても、運転手が脇見運転をしているとはいえない場合がある。たとえば、走行中の車両が車線を変更する際には、運転手はサイドミラーを見て、変更先の車線の車両状況を確認する。この場合、ドライバモニタが撮像した顔画像においては、運転手の視線がサイドミラーの方向を向いているが、車線変更にあたって運転手がサイドミラーを見る行為は、車両の安全走行のために必要な行為であって、脇見運転ではない。それにもかかわらず、ドライバモニタが脇見運転と認識すると、誤って警報が出力されることになる。 By the way, even if the line of sight obtained from the driver's face image captured by the driver monitor deviates from the traveling direction of the vehicle, it may not be said that the driver is driving aside. For example, when a running vehicle changes lanes, the driver looks at the side mirror and checks the vehicle status of the lane to be changed. In this case, in the face image captured by the driver monitor, the driver's line of sight is directed to the side mirror, but the act of looking at the side mirror when changing lanes is necessary for safe driving of the vehicle. It is an act, not an inattentive driving. Nevertheless, if the driver monitor recognizes that the driver is looking aside, an alarm is erroneously output.
　そこで、上述した特許文献１のように、運転手の視線がサイドミラーの方向を向いているときは、警報出力を禁止することで、ドライバモニタの誤認識に基づく警報出力を回避することができる。しかしながら、運転手の正面に配置されたドライバモニタのカメラからみて、運転手が脇見運転をしているときの視線方向と、サイドミラーを見ているときの視線方向とは似た方向となるため、脇見運転なのかサイドミラーの視認なのかを、顔画像から得られる視線方向だけで正確に判断するのは難しい。同様のことは、運転手が後方車両を確認するためにルームミラーを見たり、走行ルートを確認するためにカーナビゲーション装置（以下「カーナビ」という）のディスプレイを見たりする場合にも起こり得る。 Therefore, when the driver's line of sight is directed to the side mirror as in Patent Document 1 described above, by prohibiting the alarm output, it is possible to avoid the alarm output based on erroneous recognition of the driver monitor. . However, when viewed from the driver monitor camera arranged in front of the driver, the line of sight when the driver is looking aside and the line of sight when looking at the side mirror are similar. It is difficult to accurately judge whether the driver is looking aside or looking at the side mirror only based on the gaze direction obtained from the face image. A similar situation may occur when the driver looks at the rearview mirror to check the vehicle behind, or looks at the display of a car navigation device (hereinafter referred to as “car navigation”) to check the traveling route.
特開２００９－１８１２８６号公報JP 2009-181286 A 特開２００９－２８００１５号公報JP 2009-280015 A 特開２０１０－１１３６０１号公報JP 2010-113601 A
　本発明は、脇見運転とそうでない行為とを明確に区別して判定できる脇見運転検出装置を提供することを課題としている。 It is an object of the present invention to provide an inattentive driving detection device capable of clearly distinguishing an act of inattentive driving from an act of inattentive driving.
　本発明に係る脇見運転検出装置は、車両の運転席に設けられて運転手の顔を撮像する主カメラと、この主カメラが撮像した画像から得られる運転手の視線の方向に基づいて、脇見運転の有無を判定する脇見判定部と、この脇見判定部が脇見運転であると判定した場合に警報を出力する警報出力部と、車両の走行中に運転手が視認する特定の車載部品の設置箇所に設けられ、主カメラとは異なる場所から運転手の顔を撮像する補助カメラとを備えている。脇見判定部は、主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、補助カメラの撮像画像から得られる運転手の視線が、特定の車載部品の方向を向いている場合に、運転手の視線が特定の車載部品の方向を向いている時間が、あらかじめ定められた基準時間を超えれば、脇見運転であると判定し、基準時間を超えなければ、脇見運転でないと判定する。 An inattentive driving detection device according to the present invention includes an inattentive driving camera provided at a driver's seat of a vehicle, the main camera capturing an image of a driver's face, and an inattentive driving direction obtained from an image captured by the main camera. An inattentiveness determination unit that determines whether or not to drive, an alarm output unit that outputs an alarm when the inattentiveness determination unit determines that the driver is inattentive driving, and installation of a specific in-vehicle component that is visually recognized by a driver while the vehicle is traveling. And an auxiliary camera that is provided at a location and captures the driver's face from a location different from the main camera. The inattentiveness determination unit determines that the driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and that the driver's line of sight obtained from the captured image of the auxiliary camera indicates the direction of the specific vehicle-mounted component. When the driver is looking at the vehicle, the driver's gaze is directed toward a specific vehicle-mounted component.If the time exceeds a predetermined reference time, it is determined that the driver is looking aside. It is determined that it is not driving.
　このような本発明の脇見運転検出装置によると、主カメラの撮像画像から得られる運転手の視線が脇見の方向を向いていても、補助カメラの撮像画像から得られる運転手の視線が、基準時間内で特定の車載部品の方向を向いている場合は、脇見を運転に必要な行為とみなして、脇見運転とは判定しない。このため、運転手の正当行為に対して警報が出力されるのを回避することができる。一方、運転手の視線が、基準時間を超えて特定の車載部品の方向を向いている場合は、脇見を危険行為とみなして、脇見運転と判定する。このため、運転手の危険行為に対しては、警報を出力して安全を確保することができる。 According to such an inattentive driving detection device of the present invention, even if the driver's line of sight obtained from the captured image of the main camera is facing the inattentive direction, the driver's line of sight obtained from the captured image of the auxiliary camera is the reference line. If the vehicle is facing the direction of a specific vehicle-mounted component within the time, the inattentive operation is regarded as an action necessary for driving, and the inattentive operation is not determined. Therefore, it is possible to avoid outputting an alarm for a driver's legitimate action. On the other hand, if the driver's line of sight is facing the direction of a specific vehicle-mounted component beyond the reference time, it is determined that inattentive driving is a dangerous act, and that inattentive driving is determined. For this reason, it is possible to output a warning and ensure safety for dangerous behavior of the driver.
　また、本発明では、運転手の視線が特定の車載部品の方向を向いていることを、当該車載部品に付設した補助カメラの撮像画像から得られる視線に基づいて、直接的に検出する。このため、主カメラの撮像画像から得られる視線に基づいて間接的に検出する場合に比べて、検出精度が向上し、脇見運転とそうでない行為とを明確に区別して判定することができる。 According to the present invention, the fact that the driver's line of sight is directed to a specific vehicle-mounted component is directly detected based on the line of sight obtained from an image captured by an auxiliary camera attached to the vehicle-mounted component. For this reason, the detection accuracy is improved as compared with the case where the detection is indirectly performed based on the line of sight obtained from the image captured by the main camera, and it is possible to clearly distinguish between the inattentive driving and the action that is not performed.
　本発明において、脇見判定部は、主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、補助カメラの撮像画像から得られる運転手の視線が特定の車載部品の方向を向いていない場合に、運転手の視線が特定の車載部品の方向を向いていない時間が、前記基準時間より短い所定時間を超えれば、脇見運転であると判定し、前記所定時間を超えなければ、脇見運転でないと判定してもよい。 In the present invention, the driver's line of sight obtained from the captured image of the main camera is directed to the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is a specific vehicle-mounted component. When the driver does not face the direction, the time when the driver's line of sight does not face the direction of the specific vehicle-mounted component exceeds a predetermined time shorter than the reference time, it is determined that the driver is looking aside, and the predetermined time is determined. If not, it may be determined that the driver is not looking aside.
　本発明において、運転手が視認する特定の車載部品は、車体の左側に設けられた左サイドミラー、車体の右側に設けられた右サイドミラー、および車室内の前方に設けられたルームミラーの少なくとも１つから構成されるミラー系車載部品を含んでいてもよい。この場合、前記の基準時間は、第１閾値として設定された時間であり、脇見判定部は、主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、補助カメラの撮像画像から得られる運転手の視線が、ミラー系車載部品の方向を向いている場合に、運転手の視線がミラー系車載部品の方向を向いている時間が、第１閾値を超えれば脇見運転であると判定し、第１閾値を超えなければ脇見運転でないと判定する。また、脇見判定部は、主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、補助カメラの撮像画像から得られる運転手の視線が、ミラー系車載部品の方向を向いていない場合に、運転手の視線がミラー系車載部品の方向を向いていない時間が、第１閾値より小さい第２閾値を超えれば、脇見運転であると判定し、第２閾値を超えなければ、脇見運転でないと判定する。 In the present invention, the specific in-vehicle component visually recognized by the driver includes at least a left side mirror provided on the left side of the vehicle body, a right side mirror provided on the right side of the vehicle body, and a room mirror provided on the front side of the vehicle interior. It may include a mirror-based vehicle-mounted component composed of one. In this case, the reference time is the time set as the first threshold, and the inattentiveness determination unit determines that the driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and If the driver's line of sight obtained from the captured image of the camera is directed to the mirror-based vehicle-mounted component and the time during which the driver's line of sight is directed to the mirror-based vehicle-mounted component exceeds the first threshold, It is determined that the driving is inattentive driving, and if it does not exceed the first threshold value, it is determined that the driving is not inattentive driving. In addition, the inattentiveness determination unit determines that the driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is the mirror-based vehicle-mounted component. When the driver does not face the direction, if the time during which the driver's line of sight does not face the mirror system vehicle-mounted component exceeds a second threshold smaller than the first threshold, it is determined that the driver is looking aside and the second threshold is set. If not, it is determined that the driver is not looking aside.
　本発明において、運転手が視認する特定の車載部品は、上述したミラー系車載部品に加えて、あるいはミラー系車載部品に代えて、車両のインストルメントパネルに設けられた、ディスプレイおよびメータの少なくとも１つから構成される表示系車載部品を含んでいてもよい。この場合、脇見判定部は、主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、補助カメラの撮像画像から得られる運転手の視線が、表示系車載部品の方向を向いている場合に、運転手の視線が表示系車載部品の方向を向いている時間が、第３閾値を超えれば脇見運転であると判定し、第３閾値を超えなければ脇見運転でないと判定する。第３閾値は、前記の第１閾値より小さい値であってもよい。また、脇見判定部は、主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、補助カメラの撮像画像から得られる運転手の視線が表示系車載部品の方向を向いていない場合に、運転手の視線が表示系車載部品の方向を向いていない時間が、第３閾値より小さい第４閾値を超えれば脇見運転であると判定し、第４閾値を超えなければ脇見運転でないと判定する。 In the present invention, the specific in-vehicle component visually recognized by the driver is at least one of a display and a meter provided on the instrument panel of the vehicle in addition to or instead of the mirror-based in-vehicle component described above. And a display system in-vehicle component composed of two components. In this case, the inattentiveness determination unit determines that the driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and that the driver's line of sight obtained from the captured image of the auxiliary camera is a display system in-vehicle component. When the driver's gaze is in the direction of the display system in-vehicle component when the driver's gaze exceeds the third threshold, it is determined that the driver is looking aside, and when the driver's gaze does not exceed the third threshold, the driver looks aside. It is determined that it is not. The third threshold value may be a value smaller than the first threshold value. In addition, the inattentiveness determination unit determines that the driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is the direction of the display system onboard component. If the driver's gaze is not facing the direction of the in-vehicle display component when the driver's gaze exceeds the fourth threshold smaller than the third threshold, it is determined that the driver is looking aside, and the driver must not exceed the fourth threshold. It is determined that the driver is not looking aside.
　本発明において、主カメラ、脇見判定部、および警報出力部は、車両に搭載されて運転手の状態を監視するドライバモニタに備わっていてもよい。 In the present invention, the main camera, the inattentiveness determination unit, and the alarm output unit may be provided in a driver monitor that is mounted on the vehicle and monitors the state of the driver.
　本発明において、主カメラで撮像された画像を処理する第１画像処理部と、補助カメラで撮像された画像を処理する第２画像処理部と、第１画像処理部および第２画像処理部で処理された画像に基づいて、運転手の視線の方向を検出する視線検出部とを設けてもよい。 In the present invention, a first image processing unit that processes an image captured by a main camera, a second image processing unit that processes an image captured by an auxiliary camera, and a first image processing unit and a second image processing unit A gaze detection unit that detects the direction of the gaze of the driver based on the processed image may be provided.
　補助カメラが、複数の箇所に設置された特定の車載部品に対応して、複数設けられている場合、第２画像処理部は、複数の補助カメラで撮像された画像を処理する単一の画像処理部であってもよい。 In the case where a plurality of auxiliary cameras are provided corresponding to specific vehicle-mounted components installed at a plurality of locations, the second image processing unit processes a single image processed by the plurality of auxiliary cameras. It may be a processing unit.
　また、第１画像処理部と第２画像処理部に代えて、主カメラで撮像された画像および補助カメラで撮像された画像を処理する単一の画像処理部を設けてもよい。 In addition, instead of the first image processing unit and the second image processing unit, a single image processing unit that processes an image captured by the main camera and an image captured by the auxiliary camera may be provided.
　本発明の脇見運転検出装置によれば、脇見運転とそうでない行為とを明確に区別して判定することができる。 According to the inattentive driving detection device of the present invention, the act of inattentive driving can be clearly distinguished from an unattended driving.
図１は、本発明の実施形態に係る脇見運転検出装置のブロック図である。FIG. 1 is a block diagram of an inattentive driving detection device according to an embodiment of the present invention. 図２は、主カメラと補助カメラの構成を示したブロック図である。FIG. 2 is a block diagram showing the configuration of the main camera and the auxiliary camera. 図３は、運転手の顔の撮像を説明する図である。FIG. 3 is a diagram illustrating imaging of a driver's face. 図４は、主カメラと補助カメラの設置例を示した図である。FIG. 4 is a diagram showing an example of installation of a main camera and an auxiliary camera. 図５Ａは、車両走行中における運転手の視線の例を示した図である。FIG. 5A is a diagram illustrating an example of a driver's line of sight during traveling of the vehicle. 図５Ｂは、車両走行中における運転手の視線の他の例を示した図である。FIG. 5B is a diagram illustrating another example of the driver's line of sight while the vehicle is traveling. 図６は、脇見運転検出装置の動作を示したフローチャートである。FIG. 6 is a flowchart showing the operation of the inattentive driving detection device. 図７は、図６の続きのフローチャートである。FIG. 7 is a continuation of the flowchart of FIG. 図８は、本発明の他の実施形態に係る脇見運転検出装置のブロック図である。FIG. 8 is a block diagram of an inattentive driving detection device according to another embodiment of the present invention. 図９は、本発明の他の実施形態に係る脇見運転検出装置のブロック図である。FIG. 9 is a block diagram of an inattentive driving detection device according to another embodiment of the present invention. 図１０は、脇見運転検出装置の動作を示したフローチャートの他の例である。FIG. 10 is another example of a flowchart showing the operation of the inattentive driving detection device. 図１１は、脇見運転検出装置の動作を示したフローチャートの他の例である。FIG. 11 is another example of a flowchart showing the operation of the inattentive driving detection device.
　以下、本発明による脇見運転検出装置の実施形態を、図面に基づいて詳細に説明する。図１～図１１において、同一の部分または対応する部分には、同一の符号を付してある。 Hereinafter, an embodiment of the inattentive driving detection device according to the present invention will be described in detail with reference to the drawings. 1 to 11, the same portions or corresponding portions are denoted by the same reference numerals.
　まず、脇見運転検出装置の構成を、図１を参照しながら説明する。図１において、脇見運転検出装置１００には、ドライバモニタ５０と、補助カメラ２～６と、画像処理部１５～１９とが備わっている。 First, the configuration of the inattentive driving detection device will be described with reference to FIG. 1, the inattentive driving detection apparatus 100 includes a driver monitor 50, auxiliary cameras 2 to 6, and image processing units 15 to 19.
　ドライバモニタ５０は、車両に搭載されて運転手の状態を監視する装置であって、主カメラ１、画像処理部１１、視線検出部１２、脇見判定部１３、および警報出力部１４を有している。視線検出部１２と脇見判定部１３と警報出力部１４とは、制御部１０を構成している。画像処理部１１を制御部１０に含めてもよい。制御部１０には、視線検出部１２や脇見判定部１３のほかに、運転手の瞼の動きや居眠りなどを検出するためのブロックも備わっているが、それらは本発明と直接関係しないので、図示を省略してある。 The driver monitor 50 is a device that is mounted on a vehicle and monitors the state of the driver, and includes a main camera 1, an image processing unit 11, a line-of-sight detection unit 12, an inattentiveness determination unit 13, and an alarm output unit 14. I have. The line-of-sight detection unit 12, the inattentiveness determination unit 13, and the alarm output unit 14 constitute the control unit 10. The image processing unit 11 may be included in the control unit 10. The control unit 10 includes, in addition to the line-of-sight detection unit 12 and the inattentiveness determination unit 13, blocks for detecting the movement of the driver's eyelids, falling asleep, and the like, but since these are not directly related to the present invention, The illustration is omitted.
　主カメラ１は、後述するように、車両の運転席に設けられて運転手の顔を撮像する。画像処理部１１は、主カメラ１で撮像された画像を処理して、運転手の顔画像を生成するとともに、当該顔画像から目などの特徴点を抽出する。視線検出部１２は、画像処理部１１で得られた顔画像や特徴点に基づいて、運転手の視線の方向を検出する。脇見判定部１３は、視線検出部１２で検出された運転手の視線の方向に基づいて、脇見運転の有無を判定する。警報出力部１４は、脇見判定部１３が脇見運転であると判定した場合に警報を出力する。 The main camera 1 is provided in the driver's seat of the vehicle and captures a driver's face as described later. The image processing unit 11 processes an image captured by the main camera 1, generates a driver's face image, and extracts feature points such as eyes from the face image. The line-of-sight detection unit 12 detects the direction of the driver's line of sight based on the face image and the feature points obtained by the image processing unit 11. The inattentiveness determination unit 13 determines the presence or absence of inattentive driving based on the driver's gaze direction detected by the gaze detection unit 12. The alarm output unit 14 outputs an alarm when the inattentive driving determination unit 13 determines that the driver is looking aside.
　補助カメラ２～６は、後述するように、車両の走行中に運転手が視認するミラーやディスプレイなどの特定の車載部品の設置箇所に設けられる。画像処理部１５～１９は、補助カメラ２～６のそれぞれに対応して設けられており、画像処理部１１と同様に、補助カメラ２～６で撮像された運転手の顔から目などの特徴点を抽出する。画像処理部１５～１９の出力は、画像処理部１１の出力とともに、ドライバモニタ５０の視線検出部１２へ送られる。また、ドライバモニタ５０の脇見判定部１３には、図示しないＥＣＵ（電子制御ユニット）から、車両の走行速度（車速）などの車両情報が入力される。 (4) The auxiliary cameras 2 to 6 are provided at locations where specific in-vehicle components, such as mirrors and displays, which are visually recognized by a driver while the vehicle is running, as described later. The image processing units 15 to 19 are provided corresponding to the auxiliary cameras 2 to 6, respectively, and like the image processing unit 11, the characteristics such as the face to the eyes of the driver imaged by the auxiliary cameras 2 to 6 are provided. Extract points. The outputs of the image processing units 15 to 19 are sent to the visual line detection unit 12 of the driver monitor 50 together with the output of the image processing unit 11. In addition, vehicle information such as a running speed (vehicle speed) of the vehicle is input to an inattentiveness determination unit 13 of the driver monitor 50 from an ECU (electronic control unit) not shown.
　上記構成において、画像処理部１１は、本発明における「第１画像処理部」に相当し、画像処理部１５～１９は、本発明における「第２画像処理部」に相当する。 In the above configuration, the image processing unit 11 corresponds to the “first image processing unit” in the present invention, and the image processing units 15 to 19 correspond to the “second image processing unit” in the present invention.
　図２は、主カメラ１と補助カメラ２～６の構成を示している。カメラ１～６は、光学系を構成するレンズ２０と、このレンズ２０を通して被写体である運転手を撮像する撮像素子２１と、この撮像素子２１で光電変換されたアナログの電気信号を増幅する増幅回路２２と、この増幅回路２２で増幅された信号をデジタル信号に変換し画像信号として出力する信号出力回路２３とを備えている。撮像素子２１は、たとえばＣＭＯＳイメージセンサからなる。なお、主カメラ１には、必要に応じて、運転手の顔を赤外光などで照射するための光源が設けられる（図示省略）。 FIG. 2 shows the configuration of the main camera 1 and the auxiliary cameras 2 to 6. Each of the cameras 1 to 6 includes a lens 20 that forms an optical system, an imaging element 21 that captures an image of a driver as a subject through the lens 20, and an amplification circuit that amplifies an analog electric signal photoelectrically converted by the imaging element 21. And a signal output circuit 23 that converts the signal amplified by the amplifier circuit 22 into a digital signal and outputs the digital signal as an image signal. The image sensor 21 is formed of, for example, a CMOS image sensor. The main camera 1 is provided with a light source for irradiating the driver's face with infrared light or the like as necessary (not shown).
　図３は、車両３０の運転席３１において、シート３４に着座した運転手ＤＲの顔を、主カメラ１で撮像している様子を示した図である。破線は主カメラ１の撮像範囲を示している。車両３０は、たとえば自動四輪車である。 FIG. 3 is a diagram showing a state in which the main camera 1 captures an image of the face of the driver DR sitting on the seat 34 in the driver seat 31 of the vehicle 30. A broken line indicates an imaging range of the main camera 1. The vehicle 30 is, for example, an automobile.
　図４は、車室内から車両の進行方向に向って、フロントガラスＷおよびインストルメントパネル３２を見た図であり、主カメラ１と補助カメラ２～６の設置例を示している。 FIG. 4 is a view of the windshield W and the instrument panel 32 viewed from the interior of the vehicle in the traveling direction of the vehicle, and shows an example in which the main camera 1 and the auxiliary cameras 2 to 6 are installed.
　主カメラ１は、インストルメントパネル３２における運転手ＤＲと対面する箇所に設置されており、図４に示すように、ステアリング３３を通して運転手ＤＲの顔を撮像する。補助カメラ２～６は、運転手ＤＲが視認する特定の車載部品の設置箇所に設けられ、主カメラ１とは異なる場所から運転手ＤＲの顔を撮像する。詳しくは、補助カメラ２は、車体の左側に設けられた左サイドミラー３５に付設されており、補助カメラ３は、車体の右側に設けられた右サイドミラー３６に付設されている。補助カメラ４は、車室内の前方（フロントガラスＷの上部近傍）に設けられたルームミラー３７に付設されている。補助カメラ５は、インストルメントパネル３２に設けられたディスプレイ３８に付設されている。ディスプレイ３８には、カーナビやテレビなどの映像が表示される。補助カメラ６は、インストルメントパネル３２に設けられたメータ３９に付設されている。メータ３９には、走行速度やエンジン回転数などが表示される。 (4) The main camera 1 is installed at a location on the instrument panel 32 facing the driver DR, and captures an image of the driver DR's face through the steering 33 as shown in FIG. The auxiliary cameras 2 to 6 are provided at locations where specific in-vehicle components are visually recognized by the driver DR, and capture images of the driver DR's face from a location different from the main camera 1. Specifically, the auxiliary camera 2 is attached to a left side mirror 35 provided on the left side of the vehicle body, and the auxiliary camera 3 is attached to a right side mirror 36 provided on the right side of the vehicle body. The auxiliary camera 4 is attached to a room mirror 37 provided in front of the vehicle interior (near the upper portion of the windshield W). The auxiliary camera 5 is attached to a display 38 provided on the instrument panel 32. The display 38 displays an image such as a car navigation or a television. The auxiliary camera 6 is attached to a meter 39 provided on the instrument panel 32. The meter 39 displays a running speed, an engine speed, and the like.
　左サイドミラー３５、右サイドミラー３６、およびルームミラー３７は、本発明における「ミラー系車載部品」の一例であり、ディスプレイ３８およびメータ３９は、本発明における「表示系車載部品」の一例である。 The left side mirror 35, the right side mirror 36, and the room mirror 37 are examples of the "mirror-based vehicle-mounted parts" in the present invention, and the display 38 and the meter 39 are examples of the "display-based vehicle-mounted parts" in the present invention. .
　図５Ａと図５Ｂは、車両走行中における運転手ＤＲの視線の例を示している。これらの各場合において、主カメラ１により運転手ＤＲが撮像され、画像処理部１１（図１）で顔画像が生成され、視線検出部１２で視線（主カメラ１からみた視線）が検出される。また、補助カメラ２～６により運転手ＤＲが撮像され、画像処理部１５～１９で顔画像が生成され、視線検出部１２で視線（補助カメラ２～６からみた視線）が検出される。脇見判定部１３は、視線検出部１２が検出したそれぞれの視線に基づいて、運転手ＤＲが脇見運転をしているか否かを判定する。以下、この判定方法について具体的に説明する。 5A and 5B show an example of the line of sight of the driver DR while the vehicle is traveling. In each of these cases, the driver DR is imaged by the main camera 1, a face image is generated by the image processing unit 11 (FIG. 1), and the line of sight (the line of sight viewed from the main camera 1) is detected by the line of sight detection unit 12. . Further, the driver DR is imaged by the auxiliary cameras 2 to 6, a face image is generated by the image processing units 15 to 19, and the line of sight (the line of sight viewed from the auxiliary cameras 2 to 6) is detected by the line of sight detection unit 12. The inattentiveness determination unit 13 determines whether or not the driver DR is performing inattentive driving based on each line of sight detected by the line of sight detection unit 12. Hereinafter, this determination method will be specifically described.
　図５Ａに示した視線の例は、車両の運転に必要な運転手ＤＲの正当行為に基づくものであり、いずれの場合も基本的には脇見運転に該当しない。 例 The example of the line of sight shown in FIG. 5A is based on the legitimate action of the driver DR required for driving the vehicle, and in any case, it does not basically correspond to inattentive driving.
　図５Ａ（ａ）は、運転手ＤＲが正面を向いている状態であり、視線は進行方向と一致している。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは前方をまっすぐ見ている状態となるので、補助カメラ２～６が撮像した画像を参照するまでもなく、脇見運転に該当しないと判定される。 FIG. 5A (a) shows a state where the driver DR is facing the front, and the line of sight coincides with the traveling direction. In this case, in the image captured by the main camera 1, the driver DR is looking straight ahead, so it is not necessary to refer to the images captured by the auxiliary cameras 2 to 6 and the driver DR does not correspond to inattentive driving. Is determined.
　図５Ａ（ｂ）は、運転手ＤＲが左サイドミラー３５（図４）を見ている状態であり、視線は進行方向に対して斜め左下方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは左側に脇見をしている状態となるが、補助カメラ２が撮像した画像においては、運転手ＤＲは左サイドミラー３５の方向を向いている状態となる。そして、運転手ＤＲが左サイドミラー３５を見るのは、左折や左側車線への車線変更をする際に必要な行為であることから、この場合は脇見運転に該当しないと判定される。但し、一定時間を超えて左サイドミラー３５を見続けた場合は、前方不注意となって安全運転に支障を来たすため、脇見運転と判定される。 5A (b) shows a state in which the driver DR is looking at the left side mirror 35 (FIG. 4), and his line of sight is obliquely directed to the lower left with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR looks aside to the left, but in the image captured by the auxiliary camera 2, the driver DR points in the direction of the left side mirror 35. It is in a state of facing. Then, the driver DR looking at the left side mirror 35 is an action necessary for turning left or changing lanes to the left lane, so that it is determined that the driver DR does not correspond to inattentive driving in this case. However, if the driver keeps looking at the left side mirror 35 for more than a certain period of time, the driver becomes careless in the front and hinders safe driving.
　図５Ａ（ｃ）は、運転手ＤＲが右サイドミラー３６（図４）を見ている状態であり、視線は進行方向に対して斜め右下方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは右側に脇見をしている状態となるが、補助カメラ３が撮像した画像においては、運転手ＤＲは右サイドミラー３６の方向を向いている状態となる。そして、運転手ＤＲが右サイドミラー３６を見るのは、右折や右側車線への車線変更をする際に必要な行為であることから、この場合は脇見運転に該当しないと判定される。但し、一定時間を超えて右サイドミラー３６を見続けた場合は、前方不注意となって安全運転に支障を来たすため、脇見運転と判定される。 FIG. 5A (c) shows a state in which the driver DR is looking at the right side mirror 36 (FIG. 4), and the line of sight is obliquely directed to the lower right with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR looks sideways to the right, but in the image captured by the auxiliary camera 3, the driver DR points in the direction of the right side mirror 36. It is in a state of facing. Then, since the driver DR looks at the right side mirror 36 because it is necessary to turn right or change lanes to the right lane, it is determined that the driver DR does not correspond to inattentive driving in this case. However, if the driver keeps looking at the right side mirror 36 for more than a certain period of time, the driver becomes careless in the forward direction and hinders safe driving.
　図５Ａ（ｄ）は、運転手ＤＲがルームミラー３７（図４）を見ている状態であり、視線は進行方向に対して斜め左上方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは上側に脇見をしている状態となるが、補助カメラ４が撮像した画像においては、運転手ＤＲはルームミラー３７の方向を向いている状態となる。そして、運転手ＤＲがルームミラー３７を見るのは、後方を走行する車両を確認する際に必要な行為であることから、この場合は脇見運転に該当しないと判定される。但し、一定時間を超えてルームミラー３７を見続けた場合は、前方不注意となって安全運転に支障を来たすため、脇見運転と判定される。 FIG. 5A (d) shows a state in which the driver DR is looking at the rear-view mirror 37 (FIG. 4), and the line of sight is directed obliquely to the upper left with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR looks sideways upward, but in the image captured by the auxiliary camera 4, the driver DR faces the room mirror 37. State. Then, the driver DR looks at the rear-view mirror 37 because it is necessary to confirm the vehicle traveling behind, and in this case, it is determined that the driver DR does not correspond to inattentive driving. However, if the user continues to look at the rear-view mirror 37 for a certain period of time, the driver becomes careless in the forward direction and hinders safe driving.
　図５Ａ（ｅ）は、運転手ＤＲがディスプレイ３８（図４）を見ている状態であり、視線は進行方向に対して斜め左下方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは下側に脇見をしている状態となるが、補助カメラ５が撮像した画像においては、運転手ＤＲはディスプレイ３８の方向を向いている状態となる。そして、運転手ＤＲがディスプレイ３８を見るのは、たとえばカーナビにより走行ルートを確認する際に必要な行為であることから、この場合は脇見運転に該当しないと判定される。但し、一定時間を超えてディスプレイ３８を見続けた場合は、前方不注意となって安全運転に支障を来たすため、脇見運転と判定される。 FIG. 5A (e) shows a state in which the driver DR is looking at the display 38 (FIG. 4), and his line of sight is obliquely directed to the lower left with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR looks aside at the lower side, but in the image captured by the auxiliary camera 5, the driver DR faces the display 38. State. Then, since the driver DR looks at the display 38, for example, is an action necessary for confirming the traveling route by a car navigation system, it is determined that the driver DR does not correspond to inattentive driving in this case. However, if the user continues to look at the display 38 for a certain period of time, the driver becomes careless in the front and hinders safe driving.
　図５Ａ（ｆ）は、運転手ＤＲがメータ３９（図４）を見ている状態であり、視線は進行方向に対して斜め左下方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは下側に脇見をしている状態となるが、補助カメラ６が撮像した画像においては、運転手ＤＲはメータ３９の方向を向いている状態となる。そして、運転手ＤＲがメータ３９を見るのは、走行速度などを確認する際に必要な行為であることから、この場合は脇見運転に該当しないと判定される。但し、一定時間を超えてメータ３９を見続けた場合は、前方不注意となって安全運転に支障を来たすため、脇見運転と判定される。 FIG. 5A (f) shows a state in which the driver DR is looking at the meter 39 (FIG. 4), and the line of sight is obliquely directed to the lower left with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR looks aside at the lower side, but in the image captured by the auxiliary camera 6, the driver DR faces the direction of the meter 39. State. Then, since the driver DR looks at the meter 39 because it is necessary to check the traveling speed and the like, in this case, it is determined that the driver DR does not correspond to the inattentive driving. However, if the user continues to look at the meter 39 for more than a certain period of time, the driver becomes careless in the front and hinders safe driving.
　一方、図５Ｂに示した視線の例は、車両の運転に必要でない運転手ＤＲの危険行為に基づくもので、いずれの場合も基本的には脇見運転に該当する。 On the other hand, the example of the line of sight shown in FIG. 5B is based on the dangerous action of the driver DR that is not necessary for driving the vehicle, and in any case, basically corresponds to the inattentive driving.
　図５Ｂ（ｕ）は、運転手ＤＲが左側の車外風景などを見ている状態であり、視線は進行方向に対して左方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは左側に脇見をしている状態となり、補助カメラ２～６が撮像した画像においては、運転手ＤＲは車載部品３５～３９のいずれの方向も向いていない状態となる。したがって、運転手ＤＲの視線は運転に必要な行為に基づくものではない可能性が高いので、この場合は脇見運転に該当すると判定される。但し、運転手ＤＲが左方向を向いている時間がごく短時間（一瞬）であれば、実質的に安全運転に支障を来たさないことから、脇見運転に該当しないと判定される。 FIG. 5B (u) shows a state in which the driver DR is looking at a scene outside the vehicle on the left side, and the line of sight is directed leftward with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR looks aside to the left, and in the image captured by the auxiliary cameras 2 to 6, the driver DR is selected from any of the in-vehicle components 35 to 39. Is also not oriented. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is pointing to the left is extremely short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
　図５Ｂ（ｖ）は、運転手ＤＲが右側の車外風景などを見ている状態であり、視線は進行方向に対して右方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは右側に脇見をしている状態となり、補助カメラ２～６が撮像した画像においては、運転手ＤＲは車載部品３５～３９のいずれの方向も向いていない状態となる。したがって、運転手ＤＲの視線は運転に必要な行為に基づくものではない可能性が高いので、この場合は脇見運転に該当すると判定される。但し、運転手ＤＲが右方向を向いている時間がごく短時間（一瞬）であれば、実質的に安全運転に支障を来たさないことから、脇見運転に該当しないと判定される。 FIG. 5B (v) shows a state in which the driver DR is looking at the scenery outside the vehicle on the right side, and the line of sight is to the right with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR looks aside to the right, and in the image captured by the auxiliary cameras 2 to 6, the driver DR is selected from any of the in-vehicle components 35 to 39. Is also not oriented. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is pointing to the right is very short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
　図５Ｂ（ｗ）は、運転手ＤＲが空を見上げているような状態であり、視線は進行方向に対して上方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは上側に脇見をしている状態となり、補助カメラ２～６が撮像した画像においては、運転手ＤＲは車載部品３５～３９のいずれの方向も向いていない状態となる。したがって、運転手ＤＲの視線は運転に必要な行為に基づくものではない可能性が高いので、この場合は脇見運転に該当すると判定される。但し、運転手ＤＲが上方向を向いている時間がごく短時間（一瞬）であれば、実質的に安全運転に支障を来たさないことから、脇見運転に該当しないと判定される。 FIG. 5B (w) shows a state where the driver DR is looking up at the sky, and the line of sight is directed upward with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR is looking aside at the upper side, and in the images captured by the auxiliary cameras 2 to 6, the driver DR is selected from any of the in-vehicle components 35 to 39. Is also not oriented. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is facing upward is very short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
　図５Ｂ（ｘ）は、運転手ＤＲが足元などを見ている状態であり、視線は進行方向に対して下方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは下側に脇見をしている状態となり、補助カメラ２～６が撮像した画像においては、運転手ＤＲは車載部品３５～３９のいずれの方向も向いていない状態となる。したがって、運転手ＤＲの視線は運転に必要な行為に基づくものではない可能性が高いので、この場合は脇見運転に該当すると判定される。但し、運転手ＤＲが下方向を向いている時間がごく短時間（一瞬）であれば、実質的に安全運転に支障を来たさないことから、脇見運転に該当しないと判定される。 FIG. 5B (x) shows a state in which the driver DR is looking at the feet and the like, and the line of sight is directed downward with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR is looking aside at the lower side, and in the images captured by the auxiliary cameras 2 to 6, the driver DR is connected to the in-vehicle components 35 to 39. It is in a state where it is not oriented in any direction. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is facing downward is extremely short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
　図５Ｂ（ｙ）は、運転手ＤＲが左側の斜め上の車外風景などを見ている状態であり、視線は進行方向に対して斜め左上方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは上側に脇見をしている状態となり、補助カメラ２～６が撮像した画像においては、運転手ＤＲは車載部品３５～３９のいずれの方向も向いていない状態となる。したがって、運転手ＤＲの視線は運転に必要な行為に基づくものではない可能性が高いので、この場合は脇見運転に該当すると判定される。但し、運転手ＤＲが斜め左上を向いている時間がごく短時間（一瞬）であれば、実質的に安全運転に支障を来たさないことから、脇見運転に該当しないと判定される。 FIG. 5B (y) shows a state in which the driver DR is looking at an obliquely upper left outside scenery of the vehicle, and the line of sight is obliquely directed to the upper left with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR is looking aside at the upper side, and in the images captured by the auxiliary cameras 2 to 6, the driver DR is selected from any of the in-vehicle components 35 to 39. Is also not oriented. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is pointing diagonally to the upper left is very short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
　図５Ｂ（ｚ）は、運転手ＤＲがディスプレイ３８よりも下の部分を見ている状態であり、視線は進行方向に対して斜め左下方向を向いている。この場合、主カメラ１が撮像した画像においては、運転手ＤＲは下側に脇見をしている状態となり、補助カメラ２～６が撮像した画像においては、運転手ＤＲは車載部品３５～３９のいずれの方向も向いていない状態となる。したがって、運転手ＤＲの視線は運転に必要な行為に基づくものではない可能性が高いので、この場合は脇見運転に該当すると判定される。但し、運転手ＤＲが斜め左下を向いている時間がごく短時間（一瞬）であれば、実質的に安全運転に支障を来たさないことから、脇見運転に該当しないと判定される。 5B (z) shows a state in which the driver DR is looking at a portion below the display 38, and the line of sight is obliquely directed to the lower left with respect to the traveling direction. In this case, in the image captured by the main camera 1, the driver DR is looking aside at the lower side, and in the images captured by the auxiliary cameras 2 to 6, the driver DR is connected to the in-vehicle components 35 to 39. It is in a state where it is not oriented in any direction. Therefore, it is highly possible that the driver's line of sight is not based on an action necessary for driving, and in this case, it is determined that the driver DR corresponds to inattentive driving. However, if the time during which the driver DR is pointing diagonally to the lower left is very short (for a moment), it does not substantially hinder safe driving, and thus it is determined that the driving does not correspond to inattentive driving.
　次に、図６および図７のフローチャートに従って、脇見運転の判定手順をさらに詳細に説明する。 Next, the procedure for determining inattentive driving will be described in further detail with reference to the flowcharts of FIGS. 6 and 7.
　図６において、ステップＳ１では、主カメラ１と補助カメラ２～６が運転手ＤＲの顔を撮像する。ステップＳ２では、画像処理部１１、１５～１９が運転手ＤＲの顔画像を生成する。ステップＳ３では、視線検出部１２が顔画像から運転手ＤＲの視線を検出する。 In FIG. 6, in step S1, the main camera 1 and the auxiliary cameras 2 to 6 image the driver DR's face. In step S2, the image processing units 11, 15 to 19 generate a face image of the driver DR. In step S3, the gaze detecting unit 12 detects the gaze of the driver DR from the face image.
　ステップＳ４では、脇見判定部１３が、ステップＳ３で検出された視線のうち、主カメラ１で撮像された顔画像における視線と、ＥＣＵから取得した車両情報とに基づいて、運転手ＤＲが進行方向に対して左側に脇見をしているか否かを判定する。詳しくは、車速が一定以上（たとえば１０ｋｍ／ｈ以上）で、かつ運転手ＤＲの視線が進行方向に対して左側を向いている場合に、脇見判定部１３は左側の脇見であると判定する。左側に脇見をしている場合はステップＳ５へ進み、左側に脇見をしていない場合はステップＳ９へ進む。 In step S4, the inattentiveness determination unit 13 determines the driving direction of the driver DR based on the line of sight in the face image captured by the main camera 1 and the vehicle information acquired from the ECU out of the line of sight detected in step S3. It is determined whether the player looks aside to the left. More specifically, when the vehicle speed is equal to or higher than a predetermined value (for example, equal to or higher than 10 km / h) and the line of sight of the driver DR is directed leftward with respect to the traveling direction, the inattentiveness determination unit 13 determines that the driver is looking leftward. If the driver is looking aside to the left, the process proceeds to step S5. If the driver is not looking aside to the left, the process proceeds to step S9.
　ステップＳ５では、脇見判定部１３が、運転手ＤＲが左サイドミラー３５を見ているか否かを判定する。この判定は、補助カメラ２が撮像し画像処理部１５で生成された運転手ＤＲの顔画像から視線検出部１２が検出した、運転手ＤＲの視線の方向に基づいて行われる。判定の結果、図５Ａ（ｂ）のように、運転手ＤＲが左サイドミラー３５を見ている場合は、ステップＳ６ａへ進む。また、図５Ｂ（ｕ）のように、運転手ＤＲが左サイドミラー３５を見ていない場合は、ステップＳ６ｂへ進む。 In step S5, the inattentiveness determination unit 13 determines whether the driver DR is looking at the left side mirror 35. This determination is performed based on the direction of the driver's line of sight detected by the line-of-sight detection unit 12 from the driver's DR face image generated by the image processing unit 15 captured by the auxiliary camera 2. As a result of the determination, when the driver DR is looking at the left side mirror 35 as shown in FIG. 5A (b), the process proceeds to step S6a. When the driver DR does not look at the left side mirror 35 as shown in FIG. 5B (u), the process proceeds to step S6b.
　ステップＳ６ａでは、脇見運転判定のための閾値Ｔ１（第１閾値）が設定される。この閾値Ｔ１は、制御部１０の図示しないメモリにあらかじめ記憶されており、本発明の「基準時間」に相当する。続くステップＳ７では、脇見判定部１３が、脇見時間が閾値Ｔ１を超えたか否かを判定する。この場合の脇見時間は、運転手ＤＲが左サイドミラー３５を見ている時間であり、補助カメラ２の撮像画像に基づいて計測される。 で は In step S6a, a threshold T1 (first threshold) for inattentive driving determination is set. This threshold T1 is stored in advance in a memory (not shown) of the control unit 10, and corresponds to the “reference time” of the present invention. In the following step S7, the inattentiveness determination unit 13 determines whether the inattentiveness time has exceeded the threshold T1. The inattentive time in this case is the time during which the driver DR is looking at the left side mirror 35 and is measured based on the image captured by the auxiliary camera 2.
　一方、ステップＳ６ｂでは、脇見運転判定のための閾値Ｔ２（第２閾値）が設定される。この閾値Ｔ２も、制御部１０の図示しないメモリにあらかじめ記憶されている。閾値Ｔ２は、閾値Ｔ１よりも小さく（Ｔ１＞Ｔ２）、本発明の「基準時間より短い所定時間」に相当する。続くステップＳ７では、脇見判定部１３が、脇見時間が閾値Ｔ２を超えたか否かを判定する。この場合の脇見時間は、運転手ＤＲが左サイドミラー３５を見ずに左側に脇見をしている時間であって、主カメラ１の撮像画像に基づいて計測される。 On the other hand, in step S6b, a threshold T2 (second threshold) for inattentive driving determination is set. This threshold value T2 is also stored in the memory (not shown) of the control unit 10 in advance. The threshold value T2 is smaller than the threshold value T1 (T1> T2), and corresponds to the "predetermined time shorter than the reference time" of the present invention. In the following step S7, the inattentiveness determination unit 13 determines whether the inattentiveness time has exceeded the threshold T2. The inattentive time in this case is the time during which the driver DR looks aside to the left without looking at the left side mirror 35, and is measured based on the image captured by the main camera 1.
　ステップＳ７の判定の結果、脇見時間が閾値Ｔ１またはＴ２を超えている場合は、ステップＳ８へ進み、超えていない場合は、最初のステップＳ１へ戻る。ステップＳ８では、脇見判定部１３が、左方向の脇見運転と判定するとともに、警報出力部１４が警報を出力する。この警報は、音声や表示などの形態で出力される（ステップＳ１３、Ｓ１８、Ｓ２３、Ｓ２８においても同様）。 結果 If the result of the determination in step S7 is that the inattentive time exceeds the threshold value T1 or T2, the process proceeds to step S8, and if not, the process returns to the first step S1. In step S8, the inattentiveness determination unit 13 determines that the driver is looking aside in the left direction, and the alarm output unit 14 outputs an alarm. This warning is output in a form such as voice or display (the same applies to steps S13, S18, S23, and S28).
　ステップＳ４において、運転手ＤＲが左側に脇見をしていない場合、脇見判定部１３は、ステップＳ９で運転手ＤＲが右側に脇見をしているか否かを判定する。この判定は、主カメラ１で撮像された顔画像における視線と、ＥＣＵから取得した車両情報とに基づいて行われる。詳しくは、車速が一定以上で、かつ運転手ＤＲの視線が進行方向に対して右側を向いている場合に、脇見判定部１３は右側の脇見であると判定する。右側に脇見をしている場合はステップＳ１０へ進み、右側に脇見をしていない場合はステップＳ１４へ進む。 場合 If the driver DR does not look aside to the left in step S4, the inattentiveness determination unit 13 determines whether the driver DR looks aside to the right in step S9. This determination is made based on the line of sight in the face image captured by the main camera 1 and the vehicle information acquired from the ECU. Specifically, when the vehicle speed is equal to or higher than a certain value and the driver DR's line of sight is directed to the right with respect to the traveling direction, the inattentiveness determination unit 13 determines that the driver is looking rightward. If the driver is looking aside to the right, the process proceeds to step S10. If the driver is not looking aside to the right, the process proceeds to step S14.
　ステップＳ１０では、脇見判定部１３が、運転手ＤＲが右サイドミラー３６を見ているか否かを判定する。この判定は、補助カメラ３が撮像し画像処理部１６で生成された運転手ＤＲの顔画像から視線検出部１２が検出した、運転手ＤＲの視線の方向に基づいて行われる。判定の結果、図５Ａ（ｃ）のように、運転手ＤＲが右サイドミラー３６を見ている場合は、ステップＳ１１ａへ進む。また、図５Ｂ（ｖ）のように、運転手ＤＲが右サイドミラー３６を見ていない場合は、ステップＳ１１ｂへ進む。 In step S10, the inattentiveness determination unit 13 determines whether the driver DR is looking at the right side mirror 36. This determination is made based on the direction of the driver's line of sight detected by the line of sight detection unit 12 from the driver DR's face image captured by the auxiliary camera 3 and generated by the image processing unit 16. As a result of the determination, when the driver DR is looking at the right side mirror 36 as shown in FIG. 5C, the process proceeds to step S11a. In addition, as shown in FIG. 5B (v), when the driver DR does not look at the right side mirror 36, the process proceeds to step S11b.
　ステップＳ１１ａでは、前述した閾値Ｔ１が設定される。続くステップＳ１２では、脇見判定部１３が、脇見時間が閾値Ｔ１を超えたか否かを判定する。この場合の脇見時間は、運転手ＤＲが右サイドミラー３６を見ている時間であり、補助カメラ３の撮像画像に基づいて計測される。 で は In step S11a, the above-described threshold value T1 is set. In the following step S12, the inattentiveness determination unit 13 determines whether or not the inattentive time has exceeded the threshold value T1. The inattentive time in this case is a time during which the driver DR is looking at the right side mirror 36, and is measured based on an image captured by the auxiliary camera 3.
　一方、ステップＳ１１ｂでは、前述した閾値Ｔ２が設定される。続くステップＳ１２では、脇見判定部１３が、脇見時間が閾値Ｔ２を超えたか否かを判定する。この場合の脇見時間は、運転手ＤＲが右サイドミラー３６を見ずに右側に脇見をしている時間であって、主カメラ１の撮像画像に基づいて計測される。 On the other hand, in step S11b, the above-described threshold value T2 is set. In the following step S12, the inattentiveness determination unit 13 determines whether the inattentiveness time has exceeded the threshold T2. The inattentive time in this case is a time during which the driver DR looks aside to the right without looking at the right side mirror 36, and is measured based on the captured image of the main camera 1.
　ステップＳ１２の判定の結果、脇見時間が閾値Ｔ１またはＴ２を超えている場合は、ステップＳ１３へ進み、超えていない場合は、最初のステップＳ１へ戻る。ステップＳ１３では、脇見判定部１３が、右方向の脇見運転と判定するとともに、警報出力部１４が警報を出力する。 結果 If the result of determination in step S12 is that the inattentive time exceeds the threshold value T1 or T2, the process proceeds to step S13, and if not, the process returns to the first step S1. In step S13, the inattentiveness determination unit 13 determines that the driver is looking aside in the right direction, and the alarm output unit 14 outputs an alarm.
　ステップＳ９において、運転手ＤＲが右側に脇見をしていない場合、脇見判定部１３は、ステップＳ１４で運転手ＤＲが上側に脇見をしているか否かを判定する。この判定は、主カメラ１で撮像された顔画像における視線と、ＥＣＵから取得した車両情報とに基づいて行われる。詳しくは、車速が一定以上で、かつ運転手ＤＲの視線が進行方向に対して上側を向いている場合に、脇見判定部１３は上側の脇見であると判定する。上側に脇見をしている場合はステップＳ１５へ進み、上側に脇見をしていない場合はステップＳ１９（図７）へ進む。 In step S9, when the driver DR is not looking aside to the right, the inattentiveness determination unit 13 determines whether the driver DR is looking aside upward at step S14. This determination is made based on the line of sight in the face image captured by the main camera 1 and the vehicle information acquired from the ECU. More specifically, when the vehicle speed is equal to or higher than a certain value and the driver DR's line of sight is facing upward with respect to the traveling direction, the inattentiveness determination unit 13 determines that the driver is looking at the upper side. If the driver looks aside at the upper side, the process proceeds to step S15. If the driver does not look aside at the upper side, the process proceeds to step S19 (FIG. 7).
　ステップＳ１５では、脇見判定部１３が、運転手ＤＲがルームミラー３７を見ているか否かを判定する。この判定は、補助カメラ４が撮像し画像処理部１７で生成された運転手ＤＲの顔画像から視線検出部１２が検出した、運転手ＤＲの視線の方向に基づいて行われる。判定の結果、図５Ａ（ｄ）のように、運転手ＤＲがルームミラー３７を見ている場合は、ステップＳ１６ａへ進む。また、図５Ｂ（ｗ）（ｙ）のように、運転手ＤＲがルームミラー３７を見ていない場合は、ステップＳ１６ｂへ進む。 In step S15, the inattentiveness determination unit 13 determines whether the driver DR is looking at the rear-view mirror 37. This determination is performed based on the direction of the driver DR's line of sight detected by the line of sight detection unit 12 from the driver's DR face image captured by the auxiliary camera 4 and generated by the image processing unit 17. As a result of the determination, when the driver DR is looking at the rear-view mirror 37 as shown in FIG. 5D, the process proceeds to step S16a. If the driver DR is not looking at the room mirror 37 as shown in FIG. 5B (w) (y), the process proceeds to step S16b.
　ステップＳ１６ａでは、前述した閾値Ｔ１が設定される。続くステップＳ１７では、脇見判定部１３が、脇見時間が閾値Ｔ１を超えたか否かを判定する。この場合の脇見時間は、運転手ＤＲがルームミラー３７を見ている時間であり、補助カメラ４の撮像画像に基づいて計測される。 で は In step S16a, the above-described threshold value T1 is set. In the following step S17, the inattentiveness determination unit 13 determines whether the inattentiveness time has exceeded the threshold T1. The inattentive time in this case is a time during which the driver DR is looking at the rear-view mirror 37, and is measured based on an image captured by the auxiliary camera 4.
　一方、ステップＳ１６ｂでは、前述した閾値Ｔ２が設定される。続くステップＳ１７では、脇見判定部１３が、脇見時間が閾値Ｔ２を超えたか否かを判定する。この場合の脇見時間は、運転手ＤＲがルームミラー３７を見ずに上側に脇見をしている時間であって、主カメラ１の撮像画像に基づいて計測される。 On the other hand, in step S16b, the above-described threshold value T2 is set. In the following step S17, the inattentiveness determination unit 13 determines whether or not the inattentiveness time has exceeded the threshold T2. In this case, the inattentive time is a time during which the driver DR looks aside upward without looking at the room mirror 37, and is measured based on the image captured by the main camera 1.
　ステップＳ１７の判定の結果、脇見時間が閾値Ｔ１またはＴ２を超えている場合は、ステップＳ１８へ進み、超えていない場合は、最初のステップＳ１へ戻る。ステップＳ１８では、脇見判定部１３が、上方向の脇見運転と判定するとともに、警報出力部１４が警報を出力する。 結果 If the result of determination in step S17 is that the inattentive time exceeds the threshold value T1 or T2, the process proceeds to step S18, and if not, the process returns to the first step S1. In step S18, the inattentiveness determination unit 13 determines that the driver is looking aside in an upward direction, and the alarm output unit 14 outputs an alarm.
　ステップＳ１４において、運転手ＤＲが上側に脇見をしていない場合、脇見判定部１３は、図７のステップＳ１９で運転手ＤＲが下側に脇見をしているか否かを判定する。この判定は、主カメラ１で撮像された顔画像における視線と、ＥＣＵから取得した車両情報とに基づいて行われる。詳しくは、車速が一定以上で、かつ運転手ＤＲの視線が進行方向に対して下側を向いている場合に、脇見判定部１３は下側の脇見であると判定する。下側に脇見をしている場合は、ステップＳ２０およびステップＳ２４へ進む。ステップＳ２０～Ｓ２３と、ステップＳ２４～Ｓ２７とは、並行して実行される。ステップＳ１９で運転手ＤＲが下側に脇見をしていない場合は、図６のステップＳ１へ戻る。 In step S14, when the driver DR does not look aside at the upper side, the inattentiveness determination unit 13 determines whether the driver DR looks down at the lower side at step S19 in FIG. This determination is made based on the line of sight in the face image captured by the main camera 1 and the vehicle information acquired from the ECU. More specifically, when the vehicle speed is equal to or higher than a certain value and the driver DR's line of sight is directed downward with respect to the traveling direction, the inattentiveness determination unit 13 determines that the driver is looking at the lower side. If the driver looks aside at the lower side, the process proceeds to step S20 and step S24. Steps S20 to S23 and steps S24 to S27 are executed in parallel. If the driver DR does not look down at the side in step S19, the process returns to step S1 in FIG.
　ステップＳ２０では、脇見判定部１３が、運転手ＤＲがディスプレイ３８を見ているか否かを判定する。この判定は、補助カメラ５が撮像し画像処理部１８で生成された運転手ＤＲの顔画像から視線検出部１２が検出した、運転手ＤＲの視線の方向に基づいて行われる。判定の結果、図５Ａ（ｅ）のように、運転手ＤＲがディスプレイ３８を見ている場合は、ステップＳ２１ａへ進む。また、図５Ｂ（ｚ）のように、運転手ＤＲがディスプレイ３８を見ていない場合は、ステップＳ２１ｂへ進む。 In step S20, the inattentiveness determination unit 13 determines whether the driver DR is looking at the display 38. This determination is performed based on the direction of the driver DR's line of sight detected by the line-of-sight detection unit 12 from the driver DR's face image captured by the auxiliary camera 5 and generated by the image processing unit 18. As a result of the determination, when the driver DR is looking at the display 38 as shown in FIG. 5A (e), the process proceeds to step S21a. When the driver DR does not look at the display 38 as shown in FIG. 5B (z), the process proceeds to step S21b.
　ステップＳ２１ａでは、前述した閾値Ｔ１より小さい閾値Ｔ３が設定される（Ｔ１＞Ｔ３）。続くステップＳ２２では、脇見判定部１３が、脇見時間が閾値Ｔ３を超えたか否かを判定する。この場合の脇見時間は、運転手ＤＲがディスプレイ３８を見ている時間であり、補助カメラ５の撮像画像に基づいて計測される。 In step S21a, a threshold value T3 smaller than the above-described threshold value T1 is set (T1> T3). In the following step S22, the inattentiveness determination unit 13 determines whether or not the inattentive time has exceeded the threshold T3. The inattentive time in this case is the time during which the driver DR is looking at the display 38, and is measured based on the image captured by the auxiliary camera 5.
　一方、ステップＳ２１ｂでは、閾値Ｔ３より小さい閾値Ｔ４が設定される。続くステップＳ２２では、脇見判定部１３が、脇見時間が閾値Ｔ４を超えたか否かを判定する。この場合の脇見時間は、運転手ＤＲがディスプレイ３８を見ずに下側に脇見をしている時間であって、主カメラ１の撮像画像に基づいて計測される。 On the other hand, in step S21b, a threshold T4 smaller than the threshold T3 is set. In the following step S22, the inattentiveness determination unit 13 determines whether or not the inattentiveness time has exceeded the threshold T4. The inattentive time in this case is the time during which the driver DR looks aside downward without looking at the display 38, and is measured based on the image captured by the main camera 1.
　ステップＳ２２の判定の結果、脇見時間が閾値Ｔ３またはＴ４を超えている場合は、ステップＳ２３へ進み、超えていない場合は、図６のステップＳ１へ戻る。ステップＳ２３では、脇見判定部１３が、下方向の脇見運転と判定するとともに、警報出力部１４が警報を出力する。 If the inattentive time exceeds the threshold T3 or T4 as a result of the determination in step S22, the process proceeds to step S23, and if not, the process returns to step S1 in FIG. In step S23, the inattentiveness determination unit 13 determines that the vehicle is looking aside in the downward direction, and the alarm output unit 14 outputs an alarm.
　一方、ステップＳ２４では、脇見判定部１３が、運転手ＤＲがメータ３９を見ているか否かを判定する。この判定は、補助カメラ６が撮像し画像処理部１９で生成された運転手ＤＲの顔画像から視線検出部１２が検出した、運転手ＤＲの視線の方向に基づいて行われる。判定の結果、図５Ａ（ｆ）のように、運転手ＤＲがメータ３９を見ている場合は、ステップＳ２５ａへ進む。また、図５Ｂ（ｚ）のように、運転手ＤＲがメータ３９を見ていない場合は、ステップＳ２５ｂへ進む。 On the other hand, in step S24, the inattentiveness determination unit 13 determines whether the driver DR is looking at the meter 39. This determination is made based on the direction of the driver DR's line of sight detected by the line of sight detection unit 12 from the driver DR's face image captured by the auxiliary camera 6 and generated by the image processing unit 19. As a result of the determination, when the driver DR is looking at the meter 39 as shown in FIG. 5A (f), the process proceeds to step S25a. When the driver DR does not look at the meter 39 as shown in FIG. 5B (z), the process proceeds to step S25b.
　ステップＳ２５ａでは、前述した閾値Ｔ３が設定される。続くステップＳ２６では、脇見判定部１３が、脇見時間が閾値Ｔ３を超えたか否かを判定する。この場合の脇見時間は、運転手ＤＲがメータ３９を見ている時間であり、補助カメラ６の撮像画像に基づいて計測される。 で は In step S25a, the above-described threshold value T3 is set. In the following step S26, the inattentiveness determination unit 13 determines whether or not the inattentiveness time has exceeded the threshold value T3. The inattentive time in this case is the time during which the driver DR is looking at the meter 39, and is measured based on the image captured by the auxiliary camera 6.
　一方、ステップＳ２５ｂでは、前述した閾値Ｔ４が設定される。続くステップＳ２６では、脇見判定部１３が、脇見時間が閾値Ｔ４を超えたか否かを判定する。この場合の脇見時間は、運転手ＤＲがメータ３９を見ずに下側に脇見をしている時間であって、主カメラ１の撮像画像に基づいて計測される。 On the other hand, in step S25b, the above-described threshold value T4 is set. In the following step S26, the inattentiveness determination unit 13 determines whether or not the inattentive time has exceeded the threshold T4. The inattentive time in this case is the time during which the driver DR looks aside downward without looking at the meter 39, and is measured based on the image captured by the main camera 1.
　ステップＳ２６の判定の結果、脇見時間が閾値Ｔ３またはＴ４を超えている場合は、ステップＳ２７へ進み、超えていない場合は、図６のステップＳ１へ戻る。ステップＳ２７では、脇見判定部１３が、下方向の脇見運転と判定するとともに、警報出力部１４が警報を出力する。 結果 If the result of determination in step S26 is that the inattentive time has exceeded the threshold value T3 or T4, the process proceeds to step S27; otherwise, the process returns to step S1 in FIG. In step S27, the inattentiveness determination unit 13 determines that the driver is looking aside in the downward direction, and the alarm output unit 14 outputs an alarm.
　以上のように、本実施形態においては、ドライバモニタ５０に備わる主カメラ１に加えて、運転手ＤＲが視認する箇所に補助カメラ２～６を設けている。そして、主カメラ１の撮像画像から得られる運転手ＤＲの視線が脇見の方向を向いていても、補助カメラ２～６の撮像画像から得られる運転手ＤＲの視線が、基準時間（閾値Ｔ１、Ｔ３）内で、ミラー３５～３７、ディスプレイ３８、メータ３９などの特定の車載部品の方向を向いている場合は、脇見を運転に必要な行為とみなして、脇見運転とは判定しない。このため、運転手ＤＲの正当行為に対して警報が出力されるのを回避することができる。一方、運転手ＤＲの視線が、基準時間を超えて特定の車載部品の方向を向いている場合は、脇見を運転に必要でない危険行為とみなして、脇見運転と判定する。このため、運転手ＤＲの危険行為に対しては、警報を出力して安全を確保することができる。 As described above, in the present embodiment, in addition to the main camera 1 provided in the driver monitor 50, the auxiliary cameras 2 to 6 are provided at places where the driver DR can view. Then, even if the line of sight of the driver DR obtained from the captured image of the main camera 1 is looking in the inattentive direction, the line of sight of the driver DR obtained from the captured images of the auxiliary cameras 2 to 6 remains at the reference time (threshold T1, In T3), when the vehicle is facing a specific vehicle-mounted component such as the mirrors 35 to 37, the display 38, and the meter 39, the inattentive driving is regarded as an action necessary for driving, and the inattentive driving is not determined. For this reason, it is possible to avoid outputting an alarm for a legitimate action of the driver DR. On the other hand, if the driver DR's line of sight is facing the direction of a specific vehicle-mounted component beyond the reference time, it is determined that the inattentive driving is not necessary for driving, and is determined to be inattentive driving. For this reason, a warning can be output for the dangerous action of the driver DR to ensure safety.
　また、本実施形態においては、運転手ＤＲの視線が特定の車載部品の方向を向いていることを、当該車載部品に付設した補助カメラ２～６の撮像画像から得られる視線に基づいて、直接的に検出している。このため、特許文献１のような、主カメラ１の撮像画像から得られる視線に基づいて間接的に検出する方法に比べて、検出精度が向上し、脇見運転とそうでない行為とを明確に区別して判定することができる。 Further, in the present embodiment, the fact that the line of sight of the driver DR faces the direction of a specific vehicle-mounted component is directly determined based on the line of sight obtained from the captured images of the auxiliary cameras 2 to 6 attached to the vehicle-mounted component. Detected. For this reason, compared with the method of indirectly detecting based on the line of sight obtained from the image captured by the main camera 1 as in Patent Document 1, the detection accuracy is improved, and the act of looking aside and the act that is not so are clearly distinguished. It can be determined separately.
　また、本実施形態においては、主カメラ１の撮像画像から得られる運転手ＤＲの視線が脇見の方向を向いていて、補助カメラ２～６の撮像画像から得られる運転手ＤＲの視線が、特定の車載部品の方向を向いていない場合は、当該車載部品の方向を向いていない時間に応じて、脇見運転であるかそうでないかを判定している。このため、脇見をしている時間が長ければ、警報を出力して安全を確保することができ、脇見をしている時間が安全運転に支障を来たさない瞬間的なものであれば、警報を出力するのを抑制することができる。 Further, in the present embodiment, the line of sight of the driver DR obtained from the image captured by the main camera 1 faces in the inattentive direction, and the line of sight of the driver DR obtained from the images captured by the auxiliary cameras 2 to 6 is specified. If the vehicle component is not facing the direction of the vehicle-mounted component, it is determined whether the driver is looking aside or not depending on the time during which the vehicle component is not facing the direction of the vehicle-mounted component. For this reason, if the inattentive time is long, an alarm can be output to ensure safety.If the inattentive time is momentary that does not hinder safe driving, It is possible to suppress output of an alarm.
　また、本実施形態においては、主カメラ１の撮像画像から得られる運転手ＤＲの視線が脇見の方向を向いていて、補助カメラ２～６の撮像画像から得られる運転手ＤＲの視線が、サイドミラー３５、３６やルームミラー３７などのミラー系車載部品の方向を向いている場合は、当該ミラー系車載部品の方向を向いている時間が、第１閾値Ｔ１を超えれば脇見運転であると判定し、第１閾値Ｔ１を超えなければ脇見運転でないと判定している。一方、補助カメラ２～６の撮像画像から得られる運転手ＤＲの視線が、ミラー系車載部品の方向を向いていない場合は、当該ミラー系車載部品の方向を向いていない時間が、第１閾値Ｔ１より小さい第２閾値Ｔ２を超えれば脇見運転であると判定し、第２閾値Ｔ２を超えなければ脇見運転でないと判定している。このため、運転手ＤＲがミラー系車載部品を見ている場合は、閾値を大きくしてサイドミラー３５、３６やルームミラー３７を確認するための時間を確保することができ、運転手ＤＲがミラー系車載部品を見ていない場合は、閾値を小さくして警報を短時間で出力することで、安全を確保することができる。 In the present embodiment, the line of sight of the driver DR obtained from the captured image of the main camera 1 is directed in the inattentive direction, and the line of sight of the driver DR obtained from the captured images of the auxiliary cameras 2 to 6 is When the vehicle faces the mirror-based vehicle-mounted components such as the mirrors 35 and 36 and the room mirror 37, if the time in which the mirror-based vehicle-mounted components are directed exceeds the first threshold T1, it is determined that the driver is looking aside. If it does not exceed the first threshold value T1, it is determined that the driver is not looking aside. On the other hand, when the driver DR's line of sight obtained from the captured images of the auxiliary cameras 2 to 6 does not face the mirror-based vehicle-mounted component, the time during which the driver DR does not face the mirror-based vehicle-mounted component is set to the first threshold. If it exceeds the second threshold value T2 smaller than T1, it is determined that the driving is inattentive driving, and if it does not exceed the second threshold value T2, it is determined that it is not inattentive driving. For this reason, when the driver DR is looking at the mirror-based in-vehicle components, the threshold value can be increased to secure time for checking the side mirrors 35 and 36 and the room mirror 37. When the in-vehicle component is not viewed, safety can be ensured by reducing the threshold value and outputting an alarm in a short time.
　また、本実施形態においては、主カメラ１の撮像画像から得られる運転手ＤＲの視線が脇見の方向を向いていて、補助カメラ２～６の撮像画像から得られる運転手ＤＲの視線が、ディスプレイ３８やメータ３９などの表示系車載部品の方向を向いている場合は、当該表示系車載部品の方向を向いている時間が、第１閾値Ｔ１より小さい第３閾値Ｔ３を超えれば脇見運転であると判定し、第３閾値Ｔ３を超えなければ脇見運転でないと判定している。一方、補助カメラ２～６の撮像画像から得られる運転手ＤＲの視線が、表示系車載部品の方向を向いていない場合は、当該表示系車載部品の方向を向いていない時間が、第３閾値Ｔ３より小さい第４閾値Ｔ４を超えれば脇見運転であると判定し、第４閾値Ｔ４を超えなければ脇見運転でないと判定している。このため、運転手ＤＲが表示系車載部品を見ている場合は、閾値をある程度大きくしてディスプレイ３８やメータ３９を確認するための時間を確保することができ、運転手ＤＲが表示系車載部品を見ていない場合は、閾値を極力小さくして警報をすぐに出力することで、安全を確保することができる。 In the present embodiment, the line of sight of the driver DR obtained from the image captured by the main camera 1 is directed in the inattentive direction, and the line of sight of the driver DR obtained from the images captured by the auxiliary cameras 2 to 6 is displayed on the display. In the case where the vehicle faces the direction of the display system in-vehicle component such as 38 or the meter 39, if the time in which the direction of the display system in-vehicle component is directed exceeds the third threshold value T3 smaller than the first threshold value T1, it is the inattentive driving. If it does not exceed the third threshold value T3, it is determined that the driver is not looking aside. On the other hand, when the line of sight of the driver DR obtained from the captured images of the auxiliary cameras 2 to 6 does not face the direction of the display system vehicle component, the time during which the driver DR does not face the display system vehicle component is the third threshold value. If it exceeds the fourth threshold T4 smaller than T3, it is determined that the driver is looking aside, and if the driver does not exceed the fourth threshold T4, it is determined that the driver is not looking aside. For this reason, when the driver DR is looking at the display vehicle components, the threshold value can be increased to some extent to secure time for checking the display 38 and the meter 39. If the user does not see the warning, the threshold can be made as small as possible and an alarm can be output immediately to ensure safety.
　また、本実施形態においては、主カメラ１は、ドライバモニタ５０の専用カメラである。ドライバモニタ５０は、視線に基づく脇見の検出だけでなく、前述したように、瞼の動きや居眠りなどの検出も行うため、主カメラ１としては高性能のカメラを用いる必要がある。これに対して、補助カメラ２～６は、それらが付設されている車載部品３５～３９を運転手ＤＲが見ていることを検出するものであり、運転手ＤＲの視線を検出できれば足りるので、主カメラ１のような高性能のカメラである必要はない。このため、補助カメラ２～６のコストを抑制して、脇見運転検出装置１００を安価に構成することができる。 In the present embodiment, the main camera 1 is a dedicated camera for the driver monitor 50. The driver monitor 50 not only detects inattentiveness based on the line of sight, but also detects eyelid movements and falling asleep, as described above. Therefore, it is necessary to use a high-performance camera as the main camera 1. On the other hand, the auxiliary cameras 2 to 6 detect that the driver DR is looking at the vehicle-mounted components 35 to 39 to which the auxiliary cameras 2 to 6 are attached, and it is sufficient if the driver DR can detect the line of sight of the driver DR. It does not need to be a high-performance camera like the main camera 1. Therefore, the cost of the auxiliary cameras 2 to 6 can be suppressed, and the inattentive driving detection device 100 can be configured at a low cost.
　本発明では、上述した実施形態以外にも、以下のような種々の実施形態を採用することができる。 で は In the present invention, in addition to the above-described embodiments, the following various embodiments can be adopted.
　図１の脇見運転検出装置１００においては、複数の補助カメラ２～６のそれぞれに対応して画像処理部１５～１９を設けたが、図８に示す脇見運転検出装置２００のように、補助カメラ２～６のそれぞれで撮像された画像を処理する単一の画像処理部４０を設けてもよい。この画像処理部４０は、本発明における「第２画像処理部」に相当する。このようにすれば、補助カメラ２～６ごとに画像処理部を設ける必要がないので、脇見運転検出装置２００をより安価に構成することができる。 In the inattentive driving detection device 100 of FIG. 1, the image processing units 15 to 19 are provided corresponding to the plurality of auxiliary cameras 2 to 6, respectively. However, as in the inattentive driving detecting device 200 shown in FIG. A single image processing unit 40 for processing images captured by each of 2 to 6 may be provided. This image processing unit 40 corresponds to the “second image processing unit” in the present invention. By doing so, it is not necessary to provide an image processing unit for each of the auxiliary cameras 2 to 6, so that the inattentive driving detection device 200 can be configured at lower cost.
　また、図９に示す脇見運転検出装置３００のように、補助カメラ２～６のそれぞれで撮像された画像を、ドライバモニタ５０の画像処理部１１で処理するようにしてもよい。このようにすれば、補助カメラ２～６に画像処理部を設ける必要がないので、脇見運転検出装置３００をさらに安価に構成することができる。 Also, as in the case of the inattentive driving detection device 300 shown in FIG. 9, images taken by the auxiliary cameras 2 to 6 may be processed by the image processing unit 11 of the driver monitor 50. In this way, since it is not necessary to provide an image processing unit in the auxiliary cameras 2 to 6, the inattentive driving detection device 300 can be configured at a lower cost.
　図６および図７においては、ミラー系車載部品についての脇見運転判定（図６）に続いて、表示系車載部品についての脇見運転判定（図７）が実行される例を挙げたが、両者を切り離して実行するようにしてもよい。図１０および図１１は、その場合のフローチャートを示している。図１０では、ステップＳ１４で上側の脇見がない場合、最初のステップＳ１に戻って、以降のミラー系車載部品についての脇見運転判定が行われる。なお、図１０のステップＳ１で運転手ＤＲの顔を撮像するのは、主カメラ１と補助カメラ２～４である。 FIGS. 6 and 7 show an example in which an inattentive driving determination (FIG. 7) for the display-system in-vehicle component is performed following the inattentive driving determination for the mirror-based in-vehicle component (FIG. 6). It may be executed separately. FIG. 10 and FIG. 11 show flowcharts in that case. In FIG. 10, when there is no inattentive upper side in step S14, the process returns to the first step S1, and the inattentive driving determination is performed for the subsequent mirror system in-vehicle components. It is to be noted that the main camera 1 and the auxiliary cameras 2 to 4 image the face of the driver DR in step S1 in FIG.
　図１１のフローチャートは、図１０のフローチャートと並行して実行され、図７のフローチャートにステップＳ３１～Ｓ３３が追加されている。ステップＳ３１で運転手ＤＲの顔を撮像するのは、主カメラ１と補助カメラ５、６である。ステップＳ３２～Ｓ３３の内容は、図１０のステップＳ２～Ｓ３の内容と同じである。 The flowchart of FIG. 11 is executed in parallel with the flowchart of FIG. 10, and steps S31 to S33 are added to the flowchart of FIG. It is the main camera 1 and the auxiliary cameras 5 and 6 that image the face of the driver DR in step S31. The contents of steps S32 to S33 are the same as the contents of steps S2 to S3 in FIG.
　図６と図７、および図１０と図１１においては、ミラー系車載部品についての脇見運転判定と、表示系車載部品についての脇見運転判定の両方を行っているが、これらの脇見運転判定のうち、いずれか一方のみを行うようにしてもよい。 In FIGS. 6 and 7 and FIGS. 10 and 11, both the inattentive driving determination for the mirror-based in-vehicle component and the inattentive driving determination for the display-based in-vehicle component are performed. Alternatively, only one of them may be performed.
　図６と図１０においては、サイドミラー３５、３６についての閾値Ｔ１、Ｔ２を、ルームミラー３７についても適用したが、サイドミラー３５、３６とルームミラー３７とで閾値を異ならせてもよい。同様に、図７と図１１においては、ディスプレイ３８についての閾値Ｔ３、Ｔ４を、メータ３９についても適用したが、ディスプレイ３８とメータ３９とで閾値を異ならせてもよい。また、図７と図１１においては、閾値Ｔ３が閾値Ｔ１より小さく設定されているが（Ｔ１＞Ｔ３）、閾値Ｔ３は閾値Ｔ１と同じ値であってもよい（Ｔ１＝Ｔ３）。 6 and 10, the thresholds T1 and T2 for the side mirrors 35 and 36 are applied to the room mirror 37, but the thresholds may be different between the side mirrors 35 and 36 and the room mirror 37. Similarly, in FIGS. 7 and 11, the thresholds T3 and T4 for the display 38 are applied to the meter 39, but the thresholds may be different between the display 38 and the meter 39. In FIGS. 7 and 11, the threshold value T3 is set smaller than the threshold value T1 (T1> T3), but the threshold value T3 may be the same value as the threshold value T1 (T1 = T3).
　図７と図１１においては、運転手ＤＲがディスプレイ３８を見ているか否かだけを判定し（ステップＳ２０）、見ている場合は一律に閾値をＴ３に設定したが（ステップＳ２１ａ）、ディスプレイ３８の表示内容に応じて閾値を異ならせてもよい。たとえば、ディスプレイ３８に表示されているのがカーナビの画面である場合は、走行中に運転手ＤＲがディスプレイ３８を見る行為は、走行ルートなどを確認するために必要な行為であるから、閾値はＴ３に設定する。一方、ディスプレイ３８に表示されているのがテレビの画面である場合は、走行中に運転手ＤＲがディスプレイ３８を見る行為は、運転と関係のない危険行為であるから、閾値はＴ３より小さい値（Ｔ４でもよい）に設定する。 7 and 11, it is determined only whether the driver DR is looking at the display 38 (step S20). If the driver DR is looking at the display 38, the threshold is set to T3 (step S21a). The threshold value may be changed according to the display content of. For example, if the display 38 is a car navigation screen, the act of the driver DR looking at the display 38 during traveling is an act necessary for confirming a traveling route or the like. Set to T3. On the other hand, if the display 38 is a television screen, the act of the driver DR watching the display 38 while driving is a dangerous act unrelated to driving, and the threshold value is smaller than T3. (May be T4).
　前記の実施形態では、表示系車載部品としてディスプレイ３８とメータ３９を例に挙げたが、これら以外に、時計やランプなどを含めてもよい。また、車両の走行中に運転手ＤＲが視認する車載部品として、ミラー系車載部品と表示系車載部品以外に、操作系車載部品を含めてもよい。操作系車載部品としては、たとえば、ドア開閉用のスイッチ、窓開閉用のスイッチ、エアコンやオーディオ用のスイッチ、チェンジレバーなどがある。 In the above-described embodiment, the display 38 and the meter 39 have been described as examples of the display system in-vehicle components. Further, as the in-vehicle components visually recognized by the driver DR while the vehicle is traveling, operation-related in-vehicle components may be included in addition to the mirror-based in-vehicle components and the display-based in-vehicle components. The on-board operation system components include, for example, a switch for opening and closing a door, a switch for opening and closing a window, a switch for an air conditioner and audio, a change lever, and the like.
　前記の実施形態では、本発明の脇見運転検出装置を自動四輪車に搭載した例を挙げたが、本発明は、トラックやバスなどの他の車両に搭載される脇見運転検出装置にも適用することができる。 In the above-described embodiment, an example is described in which the inattentive driving detection device of the present invention is mounted on a four-wheeled vehicle. However, the present invention is also applied to an inattentive driving detecting device mounted on another vehicle such as a truck or a bus. can do.
　　１　　主カメラ
　　２～６　補助カメラ
　　１１　画像処理部（第１画像処理部）
　　１２　視線検出部
　　１３　脇見判定部
　　１４　警報出力部
　　１５～１９　画像処理部（第２画像処理部）
　　３２　インストルメントパネル
　　３５　左サイドミラー（ミラー系車載部品）
　　３６　右サイドミラー（ミラー系車載部品）
　　３７　ルームミラー（ミラー系車載部品）
　　３８　ディスプレイ（表示系車載部品）
　　３９　メータ（表示系車載部品）
　　４０　画像処理部（第２画像処理部）
　　５０　ドライバモニタ
　　１００、２００、３００　脇見運転検出装置
　　ＤＲ　運転手
　　Ｔ１　第１閾値（基準時間）
　　Ｔ２　第２閾値（基準時間より短い所定時間）
　　Ｔ３　第３閾値
　　Ｔ４　第４閾値 Reference Signs List 1 main camera 2 to 6 auxiliary camera 11 image processing unit (first image processing unit)
12 gaze detection unit 13 inattentive judgment unit 14 alarm output unit 15-19 image processing unit (second image processing unit)
32 Instrument panel 35 Left side mirror (mirror system vehicle parts)
36 Right side mirror (mirror system in-vehicle parts)
37 room mirror (mirror system vehicle parts)
38 Display (in-vehicle parts for display system)
39 meters (display system in-vehicle parts)
40 Image processing unit (second image processing unit)
50 Driver monitor 100, 200, 300 Inattentive driving detection device DR Driver T1 First threshold (reference time)
T2 Second threshold (predetermined time shorter than reference time)
T3 Third threshold T4 Fourth threshold

Claims (9)

  1. 　車両の運転席に設けられて運転手の顔を撮像する主カメラと、
    　前記主カメラが撮像した画像から得られる運転手の視線の方向に基づいて、脇見運転の有無を判定する脇見判定部と、
    　前記脇見判定部が脇見運転であると判定した場合に警報を出力する警報出力部と、を備えた脇見運転検出装置において、
    　車両の走行中に運転手が視認する特定の車載部品の設置箇所に設けられ、前記主カメラとは異なる場所から運転手の顔を撮像する補助カメラをさらに備え、
    　前記脇見判定部は、
    　前記主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、前記補助カメラの撮像画像から得られる運転手の視線が、前記特定の車載部品の方向を向いている場合に、
    　前記運転手の視線が前記特定の車載部品の方向を向いている時間が、あらかじめ定められた基準時間を超えれば、脇見運転であると判定し、
    　前記運転手の視線が前記特定の車載部品の方向を向いている時間が、前記基準時間を超えなければ、脇見運転でないと判定する、ことを特徴とする脇見運転検出装置。     A main camera provided in the driver's seat of the vehicle and imaging a driver's face;
    An inattentiveness determination unit that determines the presence or absence of inattentive driving based on the direction of the driver's line of sight obtained from the image captured by the main camera,
    An alarm output unit that outputs an alarm when the inattentive driving determination unit determines that inattentive driving is performed,
    It is further provided with an auxiliary camera that is provided at an installation location of a specific in-vehicle component to be visually recognized by a driver during traveling of the vehicle, and captures an image of the driver's face from a place different from the main camera,
    The inattentiveness determination unit,
    The line of sight of the driver obtained from the captured image of the main camera faces in the inattentive direction, and the line of sight of the driver obtained from the captured image of the auxiliary camera faces the direction of the specific vehicle-mounted component. If you have
    If the time when the driver's line of sight is facing the direction of the specific in-vehicle component exceeds a predetermined reference time, it is determined that the driver is looking aside,
    If the time during which the driver's line of sight is directed to the specific vehicle-mounted component does not exceed the reference time, it is determined that the driver is not looking aside.
  2. 　請求項１に記載の脇見運転検出装置において、
    　前記脇見判定部は、
    　前記主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、前記補助カメラの撮像画像から得られる運転手の視線が前記特定の車載部品の方向を向いていない場合に、
    　前記運転手の視線が前記特定の車載部品の方向を向いていない時間が、前記基準時間より短い所定時間を超えれば、脇見運転であると判定し、
    　前記運転手の視線が前記特定の車載部品の方向を向いていない時間が、前記所定時間を超えなければ、脇見運転でないと判定する、ことを特徴とする脇見運転検出装置。     The inattentive driving detection device according to claim 1,
    The inattentiveness determination unit,
    The driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is not in the direction of the specific vehicle-mounted component. In case,
    If the time when the driver's line of sight does not face the direction of the specific vehicle-mounted component exceeds a predetermined time shorter than the reference time, it is determined that the driver is looking aside.
    If the time during which the driver's line of sight does not face the direction of the specific in-vehicle component does not exceed the predetermined time, it is determined that the driver is not looking aside.
  3. 　請求項１に記載の脇見運転検出装置において、
    　前記特定の車載部品は、車体の左側に設けられた左サイドミラー、車体の右側に設けられた右サイドミラー、および車室内の前方に設けられたルームミラーの少なくとも１つから構成されるミラー系車載部品を含み、
    　前記基準時間は、第１閾値として設定された時間であり、
    　前記脇見判定部は、
    　前記主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、前記補助カメラの撮像画像から得られる運転手の視線が、前記ミラー系車載部品の方向を向いている場合に、
    　前記運転手の視線が前記ミラー系車載部品の方向を向いている時間が、前記第１閾値を超えれば、脇見運転であると判定し、
    　前記運転手の視線が前記ミラー系車載部品の方向を向いている時間が、前記第１閾値を超えなければ、脇見運転でないと判定し、
    　前記主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、前記補助カメラの撮像画像から得られる運転手の視線が、前記ミラー系車載部品の方向を向いていない場合に、
    　前記運転手の視線が前記ミラー系車載部品の方向を向いていない時間が、前記第１閾値より小さい第２閾値を超えれば、脇見運転であると判定し、
    　前記運転手の視線が前記ミラー系車載部品の方向を向いていない時間が、前記第２閾値を超えなければ、脇見運転でないと判定する、ことを特徴とする脇見運転検出装置。     The inattentive driving detection device according to claim 1,
    The specific in-vehicle component is a mirror system including at least one of a left side mirror provided on the left side of the vehicle body, a right side mirror provided on the right side of the vehicle body, and a room mirror provided on the front side of the vehicle interior. Including in-vehicle components,
    The reference time is a time set as a first threshold,
    The inattentiveness determination unit,
    The driver's line of sight obtained from the captured image of the main camera is directed in the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is directed to the mirror-based vehicle-mounted component. If you have
    If the time when the driver's line of sight is facing the mirror-based vehicle-mounted component exceeds the first threshold, it is determined that the driver is looking aside,
    If the time when the driver's line of sight is facing the mirror-based vehicle-mounted component does not exceed the first threshold, it is determined that the driver is not looking aside,
    The driver's line of sight obtained from the captured image of the main camera is directed in the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is directed toward the mirror system vehicle-mounted component. If not,
    If the time when the driver's line of sight does not face the mirror-based vehicle-mounted component exceeds a second threshold smaller than the first threshold, it is determined that the driver is looking aside.
    If the time during which the driver's line of sight does not face the mirror-based in-vehicle component does not exceed the second threshold, it is determined that the driver is not looking aside, and the inattentive driving detection device is characterized in that the driver is not looking aside.
  4. 　請求項３に記載の脇見運転検出装置において、
    　前記特定の車載部品は、車両のインストルメントパネルに設けられた、ディスプレイおよびメータの少なくとも１つから構成される表示系車載部品をさらに含み、
    　前記脇見判定部は、
    　前記主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、前記補助カメラの撮像画像から得られる運転手の視線が、前記表示系車載部品の方向を向いている場合に、
    　前記運転手の視線が前記表示系車載部品の方向を向いている時間が、前記第１閾値より小さい第３閾値を超えれば、脇見運転であると判定し、
    　前記運転手の視線が前記表示系車載部品の方向を向いている時間が、前記第３閾値を超えなければ、脇見運転でないと判定し、
    　前記主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、前記補助カメラの撮像画像から得られる運転手の視線が前記表示系車載部品の方向を向いていない場合に、
    　前記運転手の視線が前記表示系車載部品の方向を向いていない時間が、前記第３閾値より小さい第４閾値を超えれば、脇見運転であると判定し、
    　前記運転手の視線が前記表示系車載部品の方向を向いていない時間が、前記第４閾値を超えなければ、脇見運転でないと判定する、ことを特徴とする脇見運転検出装置。     The inattentive driving detection device according to claim 3,
    The specific in-vehicle component further includes a display system in-vehicle component provided on an instrument panel of the vehicle, the display system component including at least one of a display and a meter.
    The inattentiveness determination unit,
    The driver's line of sight obtained from the captured image of the main camera is oriented in the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is oriented in the direction of the display system in-vehicle component. If you have
    If the time during which the driver's line of sight is facing the direction of the display system in-vehicle component exceeds a third threshold smaller than the first threshold, it is determined that the driver is looking aside.
    If the time at which the driver's line of sight is facing the direction of the display system vehicle-mounted component does not exceed the third threshold, it is determined that the driver is not looking aside,
    The driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is not in the direction of the display system on-board component. In case,
    If the time during which the driver's line of sight does not face the direction of the display system in-vehicle component exceeds a fourth threshold smaller than the third threshold, it is determined that the driver is looking aside.
    If the time during which the driver's line of sight does not face the display system in-vehicle component does not exceed the fourth threshold value, it is determined that the driver is not looking aside, the inattentive driving detection device.
  5. 　請求項１に記載の脇見運転検出装置において、
    　前記特定の車載部品は、車両のインストルメントパネルに設けられた、ディスプレイおよびメータの少なくとも１つから構成される表示系車載部品を含み、
    　前記基準時間は、第３閾値として設定された時間であり、
    　前記脇見判定部は、
    　前記主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、前記補助カメラの撮像画像から得られる運転手の視線が前記表示系車載部品の方向を向いている場合に、
    　前記運転手の視線が前記表示系車載部品の方向を向いている時間が、前記第３閾値を超えれば、脇見運転であると判定し、
    　前記運転手の視線が前記表示系車載部品の方向を向いている時間が、前記第３閾値を超えなければ、脇見運転でないと判定し、
    　前記主カメラの撮像画像から得られる運転手の視線が、脇見の方向を向いていて、かつ、前記補助カメラの撮像画像から得られる運転手の視線が前記表示系車載部品の方向を向いていない場合に、
    　前記運転手の視線が前記表示系車載部品の方向を向いていない時間が、前記第３閾値より小さい第４閾値を超えれば、脇見運転であると判定し、
    　前記運転手の視線が前記表示系車載部品の方向を向いていない時間が、前記第４閾値を超えなければ、脇見運転でないと判定する、ことを特徴とする脇見運転検出装置。     The inattentive driving detection device according to claim 1,
    The specific in-vehicle component includes a display system in-vehicle component that is provided on an instrument panel of a vehicle and includes at least one of a display and a meter.
    The reference time is a time set as a third threshold,
    The inattentiveness determination unit,
    The line of sight of the driver obtained from the captured image of the main camera is in the inattentive direction, and the line of sight of the driver obtained from the captured image of the auxiliary camera is in the direction of the display system in-vehicle component. In case,
    When the time when the driver's line of sight is facing the direction of the display system in-vehicle component exceeds the third threshold value, it is determined that the driver is looking aside,
    If the time when the driver's line of sight is facing the direction of the display system in-vehicle component does not exceed the third threshold, it is determined that the driver is not looking aside,
    The driver's line of sight obtained from the captured image of the main camera is in the inattentive direction, and the driver's line of sight obtained from the captured image of the auxiliary camera is not in the direction of the display system on-board component. In case,
    If the time when the driver's line of sight does not face the direction of the display system in-vehicle component exceeds a fourth threshold smaller than the third threshold, it is determined that the driver is looking aside.
    If the time during which the driver's line of sight does not face the display system in-vehicle component does not exceed the fourth threshold value, it is determined that the driver is not looking aside, the inattentive driving detection device.
  6. 　請求項１ないし請求項５のいずれかに記載の脇見運転検出装置において、
    　前記主カメラ、前記脇見判定部、および前記警報出力部は、車両に搭載されて運転手の状態を監視するドライバモニタに備わっている、ことを特徴とする脇見運転検出装置。     An inattentive driving detection device according to any one of claims 1 to 5,
    The inattentive driving detection device, wherein the main camera, the inattentive determination unit, and the alarm output unit are provided in a driver monitor that is mounted on a vehicle and monitors a driver's state.
  7. 　請求項１ないし請求項６のいずれかに記載の脇見運転検出装置において、
    　前記主カメラで撮像された画像を処理する第１画像処理部と、
    　前記補助カメラで撮像された画像を処理する第２画像処理部と、
    　前記第１画像処理部および前記第２画像処理部で処理された画像に基づいて、運転手の視線の方向を検出する視線検出部と、をさらに備えたことを特徴とする脇見運転検出装置。     The inattentive driving detection device according to any one of claims 1 to 6,
    A first image processing unit that processes an image captured by the main camera;
    A second image processing unit that processes an image captured by the auxiliary camera;
    An inattentive driving detection device, further comprising: a line-of-sight detection unit that detects a direction of a driver's line of sight based on images processed by the first image processing unit and the second image processing unit.
  8. 　請求項７に記載の脇見運転検出装置において、
    　前記補助カメラは、複数の箇所に設置された前記特定の車載部品に対応して、複数設けられており、
    　前記第２画像処理部は、前記複数の補助カメラで撮像された画像を処理する単一の画像処理部からなる、ことを特徴とする脇見運転検出装置。     The inattentive driving detection device according to claim 7,
    The auxiliary camera is provided in plurality corresponding to the specific vehicle-mounted component installed in a plurality of locations,
    The inattentive driving detection device, wherein the second image processing unit includes a single image processing unit that processes images captured by the plurality of auxiliary cameras.
  9. 　請求項１ないし請求項６のいずれかに記載の脇見運転検出装置において、
    　前記主カメラで撮像された画像および前記補助カメラで撮像された画像を処理する単一の画像処理部と、
    　前記画像処理部で処理された画像に基づいて、運転手の視線の方向を検出する視線検出部と、をさらに備えたことを特徴とする脇見運転検出装置。     The inattentive driving detection device according to any one of claims 1 to 6,
    A single image processing unit that processes the image captured by the main camera and the image captured by the auxiliary camera,
    An inattentive driving detection device, further comprising: a line-of-sight detection unit that detects the direction of the driver's line of sight based on the image processed by the image processing unit.
PCT/JP2019/030415 2018-08-07 2019-08-02 Visually-distracted driving detection device WO2020031873A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018-148124 2018-08-07
JP2018148124A JP2020024532A (en) 2018-08-07 2018-08-07 Inattentive driving detection device

Publications (1)

Publication Number Publication Date
WO2020031873A1 true WO2020031873A1 (en) 2020-02-13

Family

ID=69415259

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/030415 WO2020031873A1 (en) 2018-08-07 2019-08-02 Visually-distracted driving detection device

Country Status (2)

Country Link
JP (1) JP2020024532A (en)
WO (1) WO2020031873A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113442884A (en) * 2021-07-08 2021-09-28 的卢技术有限公司 Intelligent brake assisting method and carrier
CN114475426A (en) * 2022-01-29 2022-05-13 深圳智慧车联科技有限公司 Muck vehicle right-turning early warning reminding system and reminding method
CN115136220A (en) * 2020-02-18 2022-09-30 五十铃自动车株式会社 Warning control device and warning control system
CN115398505A (en) * 2020-03-11 2022-11-25 五十铃自动车株式会社 Safe driving determination device

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7294194B2 (en) 2020-03-11 2023-06-20 いすゞ自動車株式会社 Safe driving judgment device
JP2021196941A (en) 2020-06-16 2021-12-27 トヨタ自動車株式会社 Driver monitoring system
JP7378681B2 (en) 2021-08-12 2023-11-13 三菱電機株式会社 Occupant condition determination device, occupant condition determination method, and occupant condition determination system
WO2023243069A1 (en) * 2022-06-17 2023-12-21 三菱電機株式会社 Inattentiveness determination device and inattentiveness determination method

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008174092A (en) * 2007-01-18 2008-07-31 Aisin Seiki Co Ltd Speed control apparatus
JP2012022646A (en) * 2010-07-16 2012-02-02 Fujitsu Ltd Visual line direction detector, visual line direction detection method and safe driving evaluation system
JP2017039461A (en) * 2015-08-21 2017-02-23 株式会社今仙電機製作所 Vehicle display and control method of the same

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008174092A (en) * 2007-01-18 2008-07-31 Aisin Seiki Co Ltd Speed control apparatus
JP2012022646A (en) * 2010-07-16 2012-02-02 Fujitsu Ltd Visual line direction detector, visual line direction detection method and safe driving evaluation system
JP2017039461A (en) * 2015-08-21 2017-02-23 株式会社今仙電機製作所 Vehicle display and control method of the same

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115136220A (en) * 2020-02-18 2022-09-30 五十铃自动车株式会社 Warning control device and warning control system
CN115398505A (en) * 2020-03-11 2022-11-25 五十铃自动车株式会社 Safe driving determination device
CN115398505B (en) * 2020-03-11 2023-09-29 五十铃自动车株式会社 Safe driving determination device
CN113442884A (en) * 2021-07-08 2021-09-28 的卢技术有限公司 Intelligent brake assisting method and carrier
CN114475426A (en) * 2022-01-29 2022-05-13 深圳智慧车联科技有限公司 Muck vehicle right-turning early warning reminding system and reminding method

Also Published As

Publication number Publication date
JP2020024532A (en) 2020-02-13

Similar Documents

Publication Publication Date Title
WO2020031873A1 (en) Visually-distracted driving detection device
JP5099451B2 (en) Vehicle periphery confirmation device
US10166922B2 (en) On-vehicle image display device, on-vehicle image display method for vehicle, and on-vehicle image setting device
US11040660B2 (en) Monitor display system and display method of the same
US11518401B2 (en) Vehicular driving assist with driver monitoring
JP4720386B2 (en) Driving assistance device
EP3466763B1 (en) Vehicle monitor system
US11223775B2 (en) Method and apparatus for the spatially resolved detection of an object outside a transportation vehicle with the aid of a sensor installed in a transportation vehicle
JP6205640B2 (en) Warning device for vehicle
WO2017145549A1 (en) Looking aside and oversight warning system and computer program
JP2008545188A (en) Night vision system
WO2015086201A1 (en) Method for operating a rearview camera system of a motor vehicle after detection of a headlight flasher, rearview camera system and motor vehicle
JP5853897B2 (en) Operation control device and operation control program
JP2008120142A (en) Information display system for automobile
JP4561627B2 (en) Vehicle periphery monitoring device
US20220319200A1 (en) Device and method for determining image data of the eyes, eye positions and/or a viewing direction of a vehicle user in a vehicle
CN109690559A (en) For providing the device of vehicle environmental information
JP2003331397A (en) Start annunciator arrangement
KR102010407B1 (en) Smart Rear-view System
US20220219610A1 (en) Vehicular cabin monitoring camera system with dual function
JP2005309660A (en) Device supporting vehicle in turning right or left
JP2008269399A (en) Traffic lane departure alarm device for vehicle
KR100882689B1 (en) Monitering system of side-rear area of car and the control method
WO2019138874A1 (en) Driving state determination device and driving state determination method
JP2017061216A (en) On-board imaging system, vehicle and imaging method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19847848

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19847848

Country of ref document: EP

Kind code of ref document: A1