JP2019087018A - Driver monitor system - Google Patents

Abstract

To suppress erroneous determination of abnormality of a driver when diagnosing the abnormality of the driver.SOLUTION: A driver monitor system 1 comprises: an imaging apparatus 10 which picks up a face image of a driver; and a control device 20 to which the face image is transmitted. The control device includes: a face direction angle detection part 21 for detecting a roll angle of a face direction of the driver on the basis of the face image that is picked up by the imaging apparatus; an abnormality diagnosis part 22 which determines that there is abnormality in the driver in a case where an absolute value of the roll angle of the face direction is maintained equal to or more than a predetermined angle for a predetermined time or longer; and a change degree detection part 23 which detects a change degree of a face of the driver on the basis of the face image. Even while the absolute value of the roll angle of the face direction is maintained equal to or more than the predetermined angle for the predetermined time and longer, in a case where the change degree of the face of the driver that is detected by the change degree detection part is equal to or more than a predetermined reference, the abnormality diagnosis part determines that there is no abnormality in the driver.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a driver monitor system.

  2. Description of the Related Art A driver monitor system including a camera for capturing a face image of a driver has been known. In such a driver monitor system, for example, a blink of the driver is detected based on a face image captured by a camera, and it is determined whether the driver is in a drowsy driving based on the interval of the detected blink. (For example, patent document 1).

Japanese Patent Laid-Open No. 2000-326757

  By the way, when an abnormality occurs in the driver due to a sudden illness or the like, the driver can not appropriately drive or monitor the vehicle. Therefore, in order to ensure the safety of the vehicle, it is necessary to diagnose the abnormality of the driver.

  When performing such a driver's abnormality diagnosis, the roll angle of the driver's face (the angle at which the driver's face is inclined with respect to the upright state) is detected based on the driver's face image, and the detected roll angle It is considered to determine that the driver has an abnormality when the angle is equal to or more than a certain angle.

  However, the driver may drive or monitor the vehicle with a cane, in which case the roll angle of the driver's face will be maintained at a relatively large angle. Therefore, when the abnormality diagnosis of the driver is performed as described above, the driver may be erroneously determined as having an abnormality although the driver does not actually have an abnormality.

  The present invention has been made in view of the above problems, and it is an object of the present invention to diagnose an abnormality of a driver, but the driver has an abnormality although the abnormality is not actually generated. It is to suppress that it makes a false decision.

  The present invention was made in order to solve the above-mentioned subject, and the gist is as follows.

  (1) A driver monitor system comprising: an imaging device for capturing a face image of a driver; and a control device to which a face image captured by the imaging device is transmitted, the control device capturing an image by the imaging device A face direction angle detection unit that detects a face direction roll angle of the driver based on the selected face image, and an absolute value of the face direction roll angle detected by the face direction angle detection unit over a predetermined time or more And a change degree detection unit that detects a change degree of a face of a driver based on a face image captured by the imaging device. The abnormality diagnosis unit detects the change degree even when the absolute value of the roll angle in the face direction is maintained at the predetermined angle or more for the predetermined time or more. The degree of change in the face of the detected driver by the if is greater than or equal to a predetermined criteria, determines that there is no abnormality to the driver, the driver monitor system.

  According to the present invention, in carrying out an abnormality diagnosis of a driver, it is possible to suppress an erroneous determination that the driver has an abnormality although the driver does not actually have an abnormality.

FIG. 1 is a schematic block diagram showing the configuration of a driver monitor system. FIG. 2 is a schematic side view partially showing the inside of a vehicle equipped with a driver monitor system. FIG. 3 schematically shows the head of the driver. FIG. 4 is a diagram showing feature points used to detect the face orientation angle of the driver. FIG. 5 is a flowchart showing a control routine of abnormality diagnosis control performed by the abnormality diagnosis unit.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description, similar components are denoted by the same reference numerals.

<Device configuration>
The configuration of the driver monitor system 1 according to one embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a schematic block diagram showing the configuration of a driver monitor system 1 according to the present embodiment. FIG. 2 is a schematic side view partially showing the inside of a vehicle 50 equipped with the driver monitor system 1 according to the present embodiment.

  The driver monitor system 1 is mounted on the vehicle 50 and monitors the state of the driver of the vehicle 50. As shown in FIG. 1, the driver monitor system 1 includes a driver monitor camera (imaging device) 10 for capturing a face image of a driver, an electronic control unit (ECU) 20 functioning as a control device for performing various controls, and a human -A machine interface (Human Machine Interface (HMI)) 30 is provided.

  First, the driver monitor camera 10 will be described with reference to FIG. As shown in FIG. 2, the vehicle 50 includes a steering 52 attached via a steering column 51 and a rearview mirror 53 disposed on the front upper side of the driver.

  In the present embodiment, the driver monitor camera 10 is provided on the top of the steering column 51, and is disposed toward the driver so as to be able to specifically capture a part of the driver's face and upper body. . The driver monitor camera 10 may be provided at a position different from the upper portion of the steering column 51 as long as the driver of the vehicle 50 can be imaged. For example, the driver monitor camera 10 may be provided on the steering 52 of the vehicle 50, the rearview mirror 53, the meter panel, the meter hood, and the like.

  The driver monitor camera 10 comprises a camera and a projector. For example, the camera is a CMOS (complementary metal oxide semiconductor) camera or a CCD (charge coupled device) camera, and the light projector is an LED (light emitting diode). In addition, it is preferable that the light projector be a near infrared LED so that the face of the driver can be photographed without giving a sense of discomfort to the driver even at low illuminance such as nighttime, and the camera also detects near infrared light Preferably it can. For example, the light projectors are two near infrared LEDs arranged on both sides of the camera. In addition, the camera may be provided with a filter such as a visible light cut filter.

  The driver monitor camera 10 is connected to an ECU 20 described later by wire or wirelessly. Therefore, the image captured by the driver monitor camera 10 is transmitted to the ECU 20 as image data.

  The ECU 20 functions as a control device that performs various controls, and includes a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), an input port, and an output port mutually connected by a bidirectional bus. It is a equipped microcomputer. Although one ECU 20 is provided in the present embodiment, a plurality of ECUs connected to one another via a bus or the like conforming to CAN or the like may be provided for each function. As shown in FIG. 1, in the present embodiment, the ECU 20 includes a face direction angle detection unit 21, an abnormality diagnosis unit 22, and a change degree detection unit 23.

  The face direction angle detection unit 21 detects the face direction angle of the driver based on the face image of the driver captured by the driver monitor camera 10. The face orientation angle of the driver includes a pitch angle θx, a yaw angle θy, and a roll angle θz (these are collectively referred to as a “face orientation angle”). Hereinafter, the pitch angle θx, the yaw angle θy and the roll angle θz will be briefly described.

  FIG. 3 schematically shows the head of the driver. As shown in FIG. 3, the face orientation of the driver can be represented by angles about three mutually perpendicular axes X, Y, Z. An axis X in the drawing is an axis extending in the lateral direction of the head of the driver, and is an axis extending in the direction perpendicular to the traveling direction of the vehicle 50 and in the horizontal direction. The angle θx of the face of the driver about the axis X is expressed as a pitch angle. The axis Y in the figure is an axis extending in the vertical direction of the head of the driver, and is an axis extending in the vertical direction. The angle θy of the face of the driver about this axis Y is represented as a yaw angle.

  The axis Z in the figure is an axis extending in the front-rear direction of the driver's head, and is an axis extending in the traveling direction of the vehicle 50. The angle θz of the face of the driver about this axis Z is represented as a roll angle. Therefore, when the driver bites his / her head, the roll angle θz of the driver changes.

  Next, with reference to FIG. 4, a specific method of detecting the face orientation angle of the driver by the face orientation angle detection unit 21 will be described. In particular, since the face direction angle detection unit 21 of the present embodiment detects at least the roll angle of the face of the driver, a method of detecting the roll angle will be mainly described. FIG. 4 is a diagram showing feature points used to detect the face orientation angle of the driver.

  When detecting the face orientation angle of the driver, each part is extracted as a feature point by matching processing of each part of the face (mouth, nose, eyes). In the present embodiment, the face direction angle detection unit 21 extracts the outer end of the eye and the tip of the nose from the face image of the driver captured by the driver monitor camera 10 as a feature point. Specifically, as shown in FIG. 4, the face direction angle detection unit 21 extracts the outer end (left end) L of the left eye, the outer end (right end) R of the right eye, and the nose N Do. Then, as shown in FIG. 4, the intersection point of the perpendicular line drawn from the point N (head of the nose) to the line segment RL (line segment connecting the outer ends of the left and right eyes) and the line segment RL is a feature point C Extract as

  Then, the face direction angle detection unit 21 detects the face direction angle of the driver based on the feature points extracted in this manner. In particular, with regard to the roll angle θz, when the face direction of the driver is inclined, the angle of the line segment RL with respect to the horizontal line of the driver monitor camera 10 (hereinafter, also simply referred to as “angle of line segment RL”) changes. Therefore, in the present embodiment, the face direction angle detection unit 21 calculates the roll angle θz of the driver based on the angle of the line segment RL with respect to the horizontal line of the driver monitor camera 10.

  Further, at this time, the roll angle θz is calculated, for example, based on the direction in which the angle of the line segment RL with respect to the horizontal line is 0 °. That is, the roll angle θz is calculated assuming that the driver's face is not inclined to the left and right and is in the upright state as 0 °. Therefore, when the driver tilts the face to the right, the angle of the line segment RL becomes larger than 0 ° and the roll angle θz becomes a positive value, and the absolute value of the roll angle θz becomes larger as the angle of tilting the face becomes larger. Become. Conversely, when the driver tilts the face to the left, the angle of the line segment RL becomes smaller than 0 ° and the roll angle θz becomes a negative value, and the absolute value of the roll angle θz becomes larger as the angle of tilting the face becomes larger. growing.

  In the example shown in FIG. 4, the face direction angle detection unit 21 extracts three feature points based on the face image captured by the driver monitor camera 10, and the face direction angle of the driver based on these feature points. Is detected. However, the face orientation angle detection unit 21 may be configured to detect the face orientation angle of the driver by another method. Therefore, the face orientation angle detection unit 21 may be configured to detect the face orientation angle of the driver based on, for example, more feature points, or the face image may be fitted (matched) to the 3D model. The 3D model may be used to detect the face direction angle.

  The abnormality diagnosis unit 22 diagnoses an abnormality of the driver based on the roll angle of the face direction detected by the face direction angle detection unit 21. In the present embodiment, the abnormality diagnosis unit 22 determines that an abnormality occurs in the driver when the absolute value of the face-oriented roll angle is maintained at the predetermined reference angle or more for the predetermined reference time or more. .

  Here, when an abnormality occurs in the driver due to sudden illness or the like, the driver can not hold the face upright. As a result, the driver's face is greatly inclined to the left and right, and the absolute value of the face-oriented roll angle is thus increased. Therefore, when an abnormality occurs in the driver due to a sudden illness or the like, the absolute value of the roll angle in the face direction detected by the face direction angle detection unit 21 is maintained at a large value. Since the abnormality diagnosis unit 22 of the present embodiment determines that the driver has an abnormality when the absolute value of the face-oriented roll angle is maintained at the reference angle or more over the reference time, the abnormality of the driver is abnormal. Can be properly diagnosed.

  The reference time is set to such a time that a driver having no abnormality does not keep tilting his face, and is set to, for example, about ten seconds or several tens of seconds. In addition, the reference angle is set to an angle that does not cause the driver without abnormality to continue tilting the face for a long time, for example, 20 ° or more, preferably 30 ° or more, more preferably 40 °. It is set above.

  The change degree detection unit 23 detects the change degree of the face of the driver based on the face image captured by the driver monitor camera. A change in the driver's face means that the driver moves at least a part of the face to change the driver's face state. Therefore, changes in the driver's face include, for example, the driver blinking (eye movement), the driver speaking (mouth movement), and the driver's expression changing (movement of the entire face surface). included.

  Specifically, in the change degree detection unit 23, first, as described above, each part is identified by the matching process of each part of the face (mouth, nose, eyes). Then, when detecting the blink of the driver, the maximum distance between the upper and lower eyelids, that is, the opening amount of the eyelids, is calculated based on the boundary of the parts of the eyes. It is determined that the eyelid is currently closed when the eyelid opening amount calculated in this manner is less than a predetermined threshold value, and the eyelid is currently open when the eyelid opening amount is greater than the threshold value. It is judged. Then, when the amount of eyelid opening calculated in this way instantaneously falls below the threshold and returns to a value larger than the threshold, it is detected that the driver blinks. In addition, as a method of processing the image input from the driver monitor camera 10 and detecting whether the driver has blinked, it is not necessary to necessarily use the above-described method, and any appropriate method can be used. .

  Similarly, it is detected that the driver is speaking and that the expression of the driver has changed. That is, the opening amount of the mouth is calculated based on the boundary of the parts of the mouth, and it is detected that the driver is speaking when the opening amount of the mouth is continuously changing. Further, the expression of the driver is calculated based on the relative positional relationship or the like of the plurality of parts. Therefore, it is detected that the expression of the driver is changing when the relative positional relationship or the like of the plurality of parts is changing. As a method of processing an image input from the driver monitor camera 10 to detect whether the driver is speaking and whether the driver's expression has changed, it is not necessary to use the above-described method, and any appropriate method may be used. Methods can be used.

  Further, the degree of change of the driver's face means the degree of change of the driver's face. The degree of change in the driver's face includes, for example, the number of blinks of the driver, the amount of change in the driver's mouth opening, and the amount of change in the relative positional relationship of face parts (change in facial expression). It can be said that the degree of change in the driver's face is larger as the number of blinks of the driver per unit time, the amount of change in the driver's mouth opening and the amount of change in the relative positional relationship of face parts increase. The degree of change in the driver's face may be calculated by another method.

  The HMI 30 is an interface for inputting and outputting information between a driver or a vehicle occupant and the driver monitoring system 1. The HMI 30 includes an information providing device for providing various information to the driver, specifically, a display for displaying character information and image information, and a speaker for audio output. In addition, the HMI 30 is provided with a microphone for recognizing the voice of the driver, and an operation button or a touch panel for the driver to perform an input operation.

  The HMI 30 is connected to the ECU 20 by wire or wirelessly. Therefore, the information input by the driver or the like is transmitted from the HMI 30 to the ECU 20, and the information providing device of the HMI 30 transmits the information to be provided to the driver from the ECU 20 to the HMI 30. For example, when it is determined by the abnormality diagnosis unit 22 of the ECU 30 that an abnormality has occurred in the driver, an abnormality alarm command is transmitted from the abnormality diagnosis unit 22 to the HMI 30. As described above, when the abnormal alarm command is transmitted, the HMI 30 performs, for example, guidance by voice from the speaker and generates an alarm sound, and causes the display to display an alarm indicating that the posture should be corrected. Thus, the HMI 30 functions as an alarm unit that warns the driver of the abnormality when the driver has an abnormality.

  When the vehicle 50 is an autonomous driving vehicle capable of autonomously driving, the vehicle 50 performs automatic emergency evacuation traveling when the abnormality determination by the abnormality diagnosis unit 22 continues even after the above-described warning. In the automatic emergency evacuation traveling, the vehicle 50 is controlled so as to bring the vehicle 50 close to the road shoulder and stop the vehicle without driving operation by the driver.

<Problems in Diagnosis of Abnormalities>
As described above, the abnormality diagnosis unit 22 determines that the driver has an abnormality when the absolute value of the face-oriented roll angle is maintained at the reference angle or more for the reference time or more. However, the driver may drive or monitor the vehicle, for example, with a cane attached. In this case, the absolute value of the face-oriented roll angle is maintained at a relatively large angle for a long time. For this reason, the abnormality diagnosis unit 22 determines that the driver has an abnormality when the driver merely attaches a cheek stick and no abnormality occurs in the driver. As a result, although there is no abnormality in the driver, generation of an alarm sound by the HMI 30 and automatic emergency evacuation traveling by the vehicle 50 will be performed.

<Control in abnormality diagnosis unit>
By the way, when the driver has an abnormality due to sudden illness or the like, the driver's face does not change. That is, in such a case, the driver does not blink, speak no more, and his expression does not change. On the other hand, if the driver simply wears a cane and there is no abnormality in the driver, the driver's face usually changes due to blinking or the like.

  Therefore, in the present embodiment, the abnormality diagnosis unit 22 changes the degree of change even when the absolute value of the roll angle in the face direction is maintained by the change degree detection unit 23 over the reference time or more. If the degree of change in the driver's face detected by the detection unit 23 is equal to or higher than a predetermined reference, it is determined that no abnormality has occurred in the driver.

  Therefore, for example, when determining the degree of change in the driver's face based on the number of blinks of the driver, if the number of blinks of the driver per unit time is equal to or greater than a reference number (for example, one), the driver It is determined that no abnormality has occurred. When determining the degree of change in the driver's face based on the amount of change in the driver's opening, if the amount of change in the driver's opening per unit time is greater than or equal to the reference change (an amount close to zero) It is determined that there is no abnormality in the driver. In addition, when the degree of change in the face of the driver is determined based on the amount of change in the relative positional relationship of the parts of the face, the amount of change in the relative positional relationship of the parts of the face per unit time is the reference change (close to zero) If it is above, it is determined that no abnormality has occurred in the driver.

  FIG. 5 is a flowchart showing a control routine of abnormality diagnosis control performed by the abnormality diagnosis unit 22. The illustrated control routine is performed at fixed time intervals.

  First, in step S11, the current roll angle θz of the face of the driver detected by the face direction angle detection unit 21 is acquired by the abnormality diagnosis unit 22. Next, in step S12, it is determined whether the absolute value of the roll angle θz acquired in step S11 is equal to or greater than the reference angle θzref. When the inclination of the face of the driver is small and it is determined that the acquired absolute value of the roll angle θz is less than the reference angle θzref, the process proceeds to step S20 without determining that the driver has an abnormality. And proceed. On the other hand, when the face of the driver is largely inclined and it is determined that the acquired absolute value of the roll angle θz is equal to or larger than the reference angle θzref, the process proceeds to step S13.

  In step S13, the value obtained by adding 1 to the value of the counter CT so far is taken as the value of the new counter CT. The counter CT represents an elapsed time after the driver's face starts to be greatly inclined.

  Next, in step S14, the change degree detection unit 23 determines whether a blink of the driver has been detected. Therefore, in step S14, the degree of change in the driver's face is determined by the number of blinks. Then, if it is determined in step S14 that a blink of the driver is detected, that is, if the number of blinks is one or more within the reference time, it is determined that an abnormality occurs in the driver. Instead, the process proceeds to step S20. On the other hand, when it is determined that the blink of the driver is not detected, the process proceeds to step S15.

  In the present embodiment, when the number of blinks of the driver is one or more within the reference time, it is determined that the degree of change of the driver's face is the reference or more. However, when the number of blinks is two or more, or more than two, it may be determined that the degree of change in the driver's face is equal to or higher than the reference.

  In step S15, whether or not the movement of the mouth is detected by the change degree detection unit 23, or more specifically, whether the change amount of the driver's mouth opening amount detected by the change degree detection unit 23 is equal to or more than the reference change amount It is determined whether or not. When the driver's mouth opening is largely changed due to the driver's speaking, etc., it is determined in step S15 that the driver opening amount change amount is equal to or greater than the reference change amount, and the driver has an abnormality. The process proceeds to step S20 without being determined. On the other hand, when it is determined in step S15 that the change amount of the driver's mouth opening amount is less than the reference change amount, the process proceeds to step S16.

  In step S16, whether or not the change degree detection unit 23 detects a change in the facial expression of the driver, more specifically, the change amount of the relative positional relationship of the face parts detected by the change degree detection unit 23 is a reference change amount It is determined whether it is above or not. If the facial expression of the driver has changed significantly, it is determined in step S16 that the amount of change in the relative positional relationship of the driver's face parts is the reference amount of change, and the driver is determined not to be abnormal. , And proceeds to step S20. On the other hand, when it is determined in step S16 that the amount of change in the relative positional relationship of the face parts of the driver is less than the reference amount of change, the process proceeds to step S17.

  In step S17, it is determined whether or not the value of the counter CT is equal to or greater than a predetermined reference value CTref, that is, whether the elapsed time since the driver's face has started to be largely inclined is equal to or longer than the reference time. If the value of the counter CT is less than the reference value CTref, the control routine is ended. On the other hand, when it is determined in step S17 that the value of the counter CT is equal to or greater than the reference value CTref, the process proceeds to step S18.

  In step S18, it is determined that an abnormality has occurred in the driver. As a result, the abnormality diagnosis unit 22 transmits an abnormality alarm command to the HMI 30. Next, at step S19, the counter CT is reset to 0, and the control routine is ended.

  On the other hand, also in step S20, since no abnormality occurs in the driver, the counter CT is reset to 0, and the control routine is ended.

<Operation and effect>
In the present embodiment, it is determined that no abnormality occurs in the driver when the degree of change in the driver's face is large, even when the face-oriented roll angle is maintained large. Here, when the degree of change in the driver's face is large, it is basically considered that no abnormality such as sudden illness has occurred in the driver. Therefore, according to the present embodiment, when the driver wears a cane or the like, the driver's abnormality determination is performed when the face-oriented roll angle is increased despite the occurrence of an abnormality in the driver. It is controlled to

<Modification>
In the above embodiment, as the change degree detection unit 23, the change degree detection unit 23 detects the change degree of the face by all the number of blinks, the change amount of the driver's mouth opening amount, and the change amount of the relative positional relationship of the driver's face parts. Is detected. However, the change degree detection unit 23 does not necessarily have to detect all these parameters, and may detect at least one of these parameters. Therefore, the abnormality diagnosis unit 22 may perform abnormality diagnosis of the driver based on at least one of these parameters.

  Further, in the above embodiment, when the degree of change of the face is large, the abnormality determination of the driver is canceled. However, for example, even when the driver operates switches, it is considered that no abnormality such as sudden illness has occurred in the driver. Therefore, the abnormality determination of the driver may be canceled even when switches around the driver are detected.

1 driver monitor system 10 driver monitor camera 20 ECU
21 face direction angle detection unit 22 abnormality diagnosis unit 23 change degree detection unit 30 HMI
50 vehicles

Claims (1)

A driver monitor system comprising: an imaging device for capturing a face image of a driver; and a control device to which the face image captured by the imaging device is transmitted.
The controller is
A face direction angle detection unit that detects a roll angle of a face direction of a driver based on a face image captured by the imaging device;
An abnormality diagnosis unit that determines that the driver has an abnormality when the absolute value of the face-direction roll angle detected by the face-direction angle detection unit is maintained at a predetermined angle or more for a predetermined time or more;
And a change degree detection unit configured to detect a change degree of a face of a driver based on a face image captured by the imaging device.
The abnormality diagnosis unit detects the face of the driver detected by the change degree detection unit even when the absolute value of the roll angle in the face direction is maintained at the predetermined angle or more for the predetermined time or more. The driver monitor system, which determines that the driver is not abnormal if the degree of change of is greater than or equal to a predetermined reference.

Images