JP7158561B2 - DRIVER'S CONDITION MONITORING DEVICE AND DRIVER'S CONDITION MONITORING METHOD - Google Patents

Description

The present invention relates to a driver condition monitoring device and a driver condition monitoring method for monitoring the condition of a driver based on an image captured by an imaging device.

Conventionally, near-infrared light is emitted from a floodlight to the driver's face, and the near-infrared light reflected from the driver's face is captured by a near-infrared camera. A driver monitoring device has been proposed that clearly captures a driver's face (Patent Document 1 below).

JP 2010-097379 A

As in Patent Document 1, in order to clearly image the driver's face without being greatly affected by the outside light, it is necessary to project near-infrared rays stronger than the outside light from the projector onto the driver's face. be. However, in such a case, external light information, such as that the driver's face is exposed to external light, is lost. If the external light information is lost, for example, when the driver is squinting his eyes, it becomes impossible to distinguish whether the driver is squinting because the external light is bright or sleepy.
On the other hand, when the intensity of the near-infrared light projected from the light projector is made weaker than the intensity of the outside light in order to acquire the outside light information, it becomes difficult to take a clear image of the driver's face.
Further, by providing an illuminance meter in addition to the near-infrared camera, it is possible to obtain information on external light, but the cost increases by the amount of the addition of the illuminance meter.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and provides a driver condition monitoring device that can obtain information on external light while clearly capturing a driver's face with a near-infrared camera. intended to

A driver condition monitoring device according to the present invention controls a near-infrared light emitting element that emits near- infrared light toward the face of a driver of a vehicle and a near-infrared camera that captures an image of the driver's face. and an imaging control unit that synchronizes the light emission period and extinction period of the near-infrared light emitting element with the imaging timing of the near-infrared camera, and the light emission image captured by the near-infrared camera during the light emission period and the near-infrared camera during the extinction period. an image acquisition unit that acquires the extinction image captured by the light emission image; a driver state determination unit that detects the driver's face information from the light emission image and determines the driver's state based on the face information; an external light state determination unit that detects brightness around the eyes and determines that the driver's eyes are illuminated with external light when the brightness is equal to or greater than the brightness threshold.

The driver condition monitoring device of the present invention can obtain information on outside light while clearly capturing a driver's face with a near-infrared camera.

1 is a block diagram showing the configuration of a driver's condition monitoring device 2 according to Embodiment 1. FIG. 4 is a timing chart for explaining imaging timing of the near-infrared light emitting element 11 and light emission timing of the near-infrared light emitting element 12 according to Embodiment 1. FIG. 4 is a flow chart showing the operation of the driving state monitoring device 2 according to Embodiment 1 for detecting luminance around the eyes. FIG. 4 is a diagram for explaining a method of determining the direction of external light from the shadow pattern formed on the driver's face according to the first embodiment; 5A and 5B are diagrams showing a hardware configuration example of the driver condition monitoring device 2 according to the first embodiment. 4 is a flowchart showing the operation of the driver's condition monitoring device 2 according to Embodiment 1; FIG. 10 is a block diagram showing the configuration of a driver's condition monitoring device 2B according to Embodiment 2; 7 is a flow chart showing the operation of a driver's condition determination unit 23B in a driver's condition monitoring device 2B according to Embodiment 2;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. The same or corresponding parts in each figure are denoted by the same reference numerals, and redundant description thereof will be appropriately simplified or omitted.

Embodiment 1
FIG. 1 is a block diagram showing the configuration of a driver's condition monitoring device 2 according to the first embodiment.
The vehicle is equipped with an imaging device 1, a driver condition monitoring device 2, a control device 3, an alarm device 4, a light blocking control device 5, an automatic braking device 6, a head-up display device 7, and a car navigation device 8. The person condition monitoring device 2 is electrically connected to the imaging device 1 and the control device 3 . Also, the control device 3 is electrically connected to an alarm device 4 , a light blocking device 5 , an automatic brake device 6 , a head-up display device 7 and a car navigation device 8 .
In Embodiment 1, the driver's condition monitoring device 2 and the control device 3 are separate devices, but the driver's condition monitoring device 2 may include the control device 3 .

The imaging device 1 includes a near-infrared camera 11 that captures near-infrared light and a near-infrared light emitting element 12 that emits near-infrared light. It is installed inside the vehicle at a position and angle such that the face of the vehicle driver is present in the vehicle. In addition, the near-infrared camera 11 and the near-infrared light emitting element 12 may be one module, or may be separated.
As described above, the image capturing apparatus 1 captures an image of the subject illuminated by the near-infrared light emitting element 12, thereby obtaining an image of the subject with clear contrast without being affected by external light such as sunlight. .

The driver's condition monitoring device 2 includes an imaging control unit 21 , an image acquisition unit 22 , a driver's condition determination unit 23 and an outside light condition determination unit 24 .
The imaging control unit 21 sets the pulse emission timing of the near-infrared light emitting element 12 and causes the near-infrared light emitting element 12 to emit pulse light. The light emission timing may be a fixed value, or may be arbitrarily changeable.
Furthermore, the imaging control unit 21 synchronizes the imaging timing of the near-infrared camera 11 with the light emission timing, and controls the near-infrared element 12 to capture images during the light emission period and the extinction period. Specifically, it will be described with reference to FIG.

FIG. 2 is a timing chart for explaining the imaging timing of the near-infrared camera 11 and the light emission timing of the near-infrared light emitting element 12 in the first embodiment.
In FIG. 2, T1 is a period (imaging period) during which the near-infrared camera 11 captures the optical signal, and T2 is a period (blanking period) during which the near-infrared camera 11 does not capture the optical signal.
As shown in FIG. 2, the near-infrared light emitting element 12 repeats light emission and extinction. Light emission of the near-infrared light emitting element 12 rises during the blanking period, and light emission of the near-infrared light emitting element 12 falls during the blanking period. By controlling the near-infrared camera 11 and the near-infrared light emitting element 12 in this way, the imaging control unit 21 can produce a light emission image when the near-infrared light emitting element 12 is emitting light and a light emission image when the near-infrared light emitting element 12 is extinguished. , and the extinction image can be alternately captured.

When the near-infrared light emitting element 12 captures an image during the light emission period, the driver's face illuminated by the near-infrared light emitting element 12 is captured, so that the driver's face is clearly captured without being affected by outside light. can be imaged. On the other hand, since the extinguished image captures the driver's face illuminated by outside light, the driver's face is less clear than the luminescent image, but information on the outside light can be obtained from the luminance of the pixels. That is, as described above, external light is detected by alternately capturing an emission image when the near-infrared light emitting element 12 emits light and an extinguished image when the near-infrared light emitting element 12 is extinguished. A single near-infrared camera can capture both a clear image of the driver's face and an image containing information about external light without adding an illuminance meter or the like for the purpose of capturing the image.

In addition, the imaging control unit 21 sequentially transmits the imaging timing of the near-infrared camera 11 and the light emission timing of the near-infrared light emitting element 12 to the image acquisition unit 22 .
In the above description, the imaging timing of the near-infrared camera 11 is synchronized with the emission timing of the near-infrared light emitting element 12. You may make it synchronize.

The image acquisition unit 22 acquires an image captured by the near-infrared camera 11 and determines whether the acquired image is a luminescence image or an extinction image based on the imaging timing and the luminescence timing received from the imaging control unit 21 . Then, when the acquired image is a light emission image, the image is transmitted to the driver state determination unit 23, and when the acquired image is the extinction image, the image is transmitted to the external light state determination unit 24.

When acquiring the light emission image from the image acquisition unit 22, the driver state determination unit 23 detects the driver's face from the light emission image, and detects face information such as the degree of eye opening and the face direction of the driver. Then, based on the detected face information, it is determined whether or not the driver is in a state unsuitable for driving the vehicle. Here, the state unsuitable for driving includes, for example, a state of dozing off, a state of looking aside, a state of being unable to drive (a dead man state), and the like.
When determining that the driver is unsuitable for driving, the driver state determination unit 23 notifies the control device 3 that the driver is unsuitable for driving.

When acquiring the extinction image from the image acquisition unit 22, the external light condition determination unit 24 detects the brightness around the eyes of the driver from the extinction image. For example, the external light condition determination unit 24 detects the average luminance of pixels included in a predetermined range based on the eye position. When the driver's eyes can be detected from the extinction image, the luminance around the eyes is detected based on the position of the eyes detected from the extinction image. The light state determination unit 24 acquires the eye positions detected from the light emission image by the driver state determination unit 23, and detects the luminance around the eyes based on the eye positions detected from the light emission image. Specifically, it will be described with reference to FIG.

FIG. 3 is a flow chart showing the operation of detecting the luminance around the eyes of the driving state monitoring device 2 according to the first embodiment.
After acquiring the extinction image, the external light information acquisition unit 24 determines whether the driver's eyes can be detected from the extinction image (step S001), and if the driver's eyes can be detected from the extinction image (step S001: YES). , the luminance around the eyes of the driver is detected based on the position of the driver's eyes detected from the extinction image (step S002), and the process is terminated. On the other hand, when the eyes of the driver cannot be detected from the extinguished image (step S001: NO), the outside light condition determination unit 24 determines the position of the eyes detected by the driver condition determination unit 23 from the light emission image. Acquired from the unit 23 (S003), and based on the position of the eye detected from the acquired luminescence image, the brightness around the eye is detected (S004). Then, the process ends.

In this way, when the eyes of the driver cannot be detected from the extinguished image, the external light condition determination unit 24 detects the luminance around the eyes based on the position of the eyes detected from the luminescence image. Based on the position of the driver's eyes detected from the illuminated image, which is not affected by external light, even in situations where the driver's face in the extinguished image is overexposed due to the light hitting the driver's face. can detect the luminance around the driver's eyes.
In the above description, when the eyes of the driver cannot be detected from the extinguished image, the luminance around the eyes is detected based on the position of the eyes detected from the emitted light image. The driver's condition determination unit 23 may detect the brightness around the eyes of the driver based on the positions of the eyes detected from the light emission image.

Then, it is determined whether or not the brightness around the driver's eyes is equal to or higher than the brightness threshold, and if the brightness around the driver's eyes is equal to or higher than the brightness threshold, it is determined that the driver's eyes are being irradiated with external light. judge.
When the external light state determination unit 24 determines that the driver's eyes are being irradiated with external light, the external light state determination unit 24 notifies the control device 3 that the driver's eyes are being irradiated with external light. notification.
Further, when it is determined that the driver's eyes are being irradiated with the external light, the external light condition determination unit 24 detects from which direction the external light is being irradiated to the driver's face. may

A method of determining the irradiation direction of external light from the pattern of shadows formed on the driver's face will be described with reference to FIG.
FIG. 4 is a diagram for explaining a method of determining the direction of external light from the shadow pattern formed on the driver's face according to the first embodiment.
FIG. 4A shows the pattern formed on the driver's face when the sun is present in front left of the vehicle, while FIG. 4B shows the pattern of shadows formed on the driver's face when the sun is present in front right of the vehicle. is. As can be seen from FIGS. 4A and 4B, a protruding portion of the driver's face, particularly near the nose, is shadowed in a direction opposite to the irradiation direction of external light. Therefore, by storing in advance the pattern of the shadow formed on the driver's face with respect to the irradiation direction of the external light, the irradiation direction of the external light can be determined from the pattern of the shadow formed on the driver's face obtained from the extinction image. becomes possible.

It should be noted that, when the outside light state acquisition unit 24 determines the irradiation direction of the outside light, it outputs the irradiation direction of the outside light to the control device 3 .

The control device 3 includes an alarm control section 31 , a light blocking control section 32 , a vehicle control section 33 and a display control section 34 .
The alarm control unit 31 controls the alarm device 4 to sound an alarm to the driver when notified by the driver state determination unit 23 that the driver is unsuitable for driving.
When notified by the external light state determination unit 24 that the driver's eyes are being irradiated with external light, the light shielding control unit 32 controls the light shielding device 5 provided in the vehicle so that the driver's eyes are illuminated. To shield the external light irradiating the eyes. The light blocking device 5 is, for example, a sun visor or a windshield with variable transmittance.
Further, when the light shielding control unit 32 acquires the irradiation direction of the external light from the external light state acquisition unit, the light blocking device 5 is controlled based on the irradiation direction of the external light, thereby more efficiently blocking the external light. It is possible to block light.

When the vehicle control unit 33 is notified by the driver state determination unit 23 that the driver is unsuitable to be a driver, or when the external light state determination unit 24 indicates a state in which the driver's eyes are exposed to external light. is notified, the automatic brake device 6 is controlled so that the automatic brake operates earlier than usual.

When the external light condition determination unit 24 notifies that the driver's eyes are illuminated with external light, the display control unit 34 brightens the display brightness of the head-up display device 7 and the car navigation device 8. In addition, the head-up display device 7 and the car navigation device 8 can be easily recognized visually even in a state where the driver feels dazzling. In addition to increasing the brightness, the display color may be changed to a color that is easy to see.

Next, a hardware configuration example of the driver's condition monitoring device 2 will be described.
5A and 5B are diagrams showing a hardware configuration example of the driver condition monitoring device 2. FIG.
Each function of the imaging control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 in the driver condition monitoring device 2 is realized by a processing circuit. That is, the driver's condition monitoring device 2 has a processing circuit for realizing each function described above. The processing circuit may be a processing circuit 100a that is dedicated hardware as shown in FIG. 5A, or a processor 100b that executes a program stored in a memory 200c as shown in FIG. 5B. good.

As shown in FIG. 5A, when the imaging control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 are dedicated hardware, the processing circuit 100a is, for example, a single circuit, Composite circuits, programmed processors, parallel programmed processors, ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), or combinations thereof. The functions of each unit of the imaging control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 may be realized by processing circuits, respectively, or the functions of each unit may be integrated into one processing circuit. may be realized.

As shown in FIG. 5B, when the imaging control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 are the processor 100b, the function of each unit is software, firmware, or software and firmware. It is realized by a combination of Software or firmware is written as a program and stored in memory 100c. The processor 100b implements the functions of the imaging control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 by reading and executing programs stored in the memory 100c. That is, when the imaging control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24 are executed by the processor 100b, each step shown in FIG. 3 and FIGS. will be executed as a result. It can also be said that these programs cause a computer to execute the procedures or methods of the imaging control unit 21, the image acquisition unit 22, the driver state determination unit 23, and the external light state determination unit 24.

Here, the processor 100b is, for example, a CPU (Central Processing Unit), a processing device, an arithmetic device, a processor, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor). The memory 100c may be, for example, non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM). However, it may be a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a mini disk, a CD (Compact Disc) or a DVD (Digital Versatile Disc).

Next, operation of the driver's condition monitoring device 2 will be described with reference to FIG.
FIG. 6 is a flow chart showing the operation of the driver's condition monitoring device 2 according to the first embodiment.
When the driver's condition monitoring device 2 is powered on, the imaging control unit 21 determines the pulse emission timing of the near-infrared light emitting element 12 and causes the near-infrared light emitting element 12 to emit pulse light (step S101). Then, the imaging control unit 21 synchronizes the imaging timing of the near-infrared camera 11 with the emission timing of the near-infrared light emitting element 12 so that the near-infrared camera 11 takes an image during the light emission period and the extinction period of the near-infrared light emitting element 12. (step S102), and outputs the imaging timing and the light emission timing to the image acquisition unit 22 (step S103). The image acquisition unit 22 acquires an image captured by the near-infrared camera (step S104), and determines whether the acquired image is a luminescence image or an extinction image based on the imaging timing and the luminescence timing received from the imaging control unit 21 (step S105). If the image acquired by the image acquisition unit 22 is a luminescence image (step S105: YES), the luminescence image is transmitted to the driver state determination unit 23, and the driver state determination unit 23 determines based on the transmitted luminescence image. It is determined whether or not the driver is unsuitable for driving (step S106). Then, the determination result in step S106 is output to the control device 3 (S107), and the process returns to step S104. On the other hand, if the image acquired by the image acquisition unit 22 is an extinction image (step S105: NO), the extinction image is transmitted to the external light condition determination unit 24, and the external light condition determination unit 24 converts the transmitted extinction image into Based on this, it is determined whether or not the driver's eyes are being irradiated with outside light (step S108). Then, the determination result in step S106 is output to the control device 3 (S107), and the process returns to step S104.

As described above, according to Embodiment 1, the near- infrared light emitting element that emits near-infrared light toward the face of the driver of the vehicle and the near-infrared camera that captures the face of the driver are controlled, An imaging control unit that causes the light emitting element to emit pulsed light and synchronizes the light emission period and the extinction period of the near infrared light emitting element with the imaging timing of the near infrared camera, and the light emission image captured by the near infrared camera during the light emission period and the extinction period. An image acquisition unit that acquires an extinction image captured by an infrared camera, a driver state determination unit that detects facial information of the driver from the light emission image and determines the state of the driver based on the face information, and a driving system based on the extinction image. and an external light state determination unit that detects the brightness around the eyes of the driver and determines that the driver's eyes are illuminated with external light when the brightness is equal to or higher than the brightness threshold. The information of outside light can be acquired while the camera captures a clear image of the driver's face.

Embodiment 2
FIG. 7 is a block diagram showing the configuration of a driver's condition monitoring device 2B according to the second embodiment.
The driver condition monitoring device 2B according to the second embodiment is designed to detect whether the driver is squinting because he is sleepy or because the outside light is dazzling when the driver is squinting his eyes more than usual. I tried to determine if there is.

FIG. 8 is a flowchart showing the operation of the driver's condition determining section 23B in the driver's condition monitoring device 2B according to the second embodiment.
When acquiring the luminescence image from the image acquiring unit 22, the driver state determination unit 23B sequentially detects the degree of eye opening of the driver from the acquired luminescence image (step S201). Then, it is determined whether or not the state in which the degree of eye openness is equal to or less than the threshold has continued for a predetermined period of time (step S202). Step S202: YES), the external light condition determination unit 24B detects the brightness around the eyes of the driver from the extinction image (step S203). On the other hand, if the degree of eye openness has not continued for a predetermined period of time below the eye openness threshold value (step 202 : NO), the process returns to step 201 . The external light condition determination unit 24B determines whether or not the brightness around the eyes of the driver detected in step S203 is equal to or less than the brightness threshold (step S204), and if the brightness around the eyes of the driver is equal to or less than the brightness threshold (step S204: YES), it is determined that the driver is asleep (step S205). On the other hand, if the brightness around the eyes of the driver is higher than the brightness threshold (step S204: NO), it is determined that the driver feels that the outside light is dazzling (step S206). Then, the process ends.

As described above, the driver's state determination unit 23B of the second embodiment detects the driver's eye periphery detected by the external light state determination unit 24B when the state in which the degree of eye opening is equal to or less than the threshold continues for a predetermined period of time. If the brightness around the driver's eyes is below the brightness threshold, it is determined that the driver is dozing off, and if the brightness around the driver's eyes is higher than the brightness threshold It is determined that the driver feels that the external light is dazzling. As a result, when the driver's eyes are squinting more than usual, it is determined whether the driver is squinting because he is sleepy or because the outside light is dazzling.

1 imaging device 2, 2A driver condition monitoring device 3 control device 4 alarm device 5 light blocking device 6 automatic braking device 7 head-up display device 8 car navigation device 11 near-infrared camera 12 near-infrared emission element, 21 imaging control unit, 22 image acquisition unit, 23, 23B driver state determination unit, 24, 24B external light state determination unit, 31 alarm control unit, 32 light shielding control unit, 33 vehicle control unit, 34 display control unit, 100a processing circuit, 100b processor, 100c memory

Claims (7)

A near- infrared light emitting element that irradiates a near-infrared light toward the face of a driver of a vehicle and a near-infrared camera that captures the face of the driver are controlled to cause the near-infrared light emitting element to emit pulses, and the near-infrared light an imaging control unit for synchronizing the light emitting period and the extinction period of the light emitting element with the imaging timing of the near-infrared camera;
an image acquisition unit that acquires a luminescence image captured by the near-infrared camera during the luminescence period and an extinction image captured by the near-infrared camera during the extinction period;
a driver state determination unit that detects facial information of the driver from the luminescent image and determines the state of the driver based on the facial information;
an external light state determination unit that detects brightness around the driver's eye from the extinguished image and determines that the driver's eye is illuminated with external light when the brightness is equal to or greater than a brightness threshold;
A driver condition monitoring device comprising: The driver state determination unit detects, when the driver's eye openness detected from the light emission image continues for a predetermined time or longer, the external light state determination unit detects 2. The driver condition monitoring apparatus according to claim 1, wherein the driver is determined to be dozing if the luminance around the eyes of the driver is equal to or less than a luminance threshold. The driver state determination unit detects the eye position of the driver from the luminescence image,
2. The driver's condition monitoring apparatus according to claim 1, wherein said external light condition determination unit detects the luminance around the driver's eyes based on the positions of said driver's eyes detected by said driver condition determination unit. When the position of the driver's eyes can be detected from the extinction image, the external light condition determination unit detects the luminance around the eyes of the driver based on the eye position detected from the extinction image, 4. The brightness around the eyes of the driver is detected based on the position of the eyes of the driver detected by the driver condition determining unit when the position of the eyes of the driver cannot be detected from the extinguished image. driver condition monitoring device. 2. The driver's condition monitoring according to claim 1, wherein the external light condition determination unit detects the brightness of the driver's face from the extinction image, and determines the irradiation direction of the external light based on the brightness of the driver's face. Device. A light shielding control unit that controls a light shielding device provided in the vehicle so as to shield external light when it is determined that the driver's eyes are irradiated with external light, wherein the light shielding control unit includes: 6. The driver's condition monitoring apparatus according to claim 5, wherein the light shielding device is controlled according to the irradiation direction of the external light determined by the external light condition determination unit. The imaging control unit controls a near- infrared light emitting element that emits near-infrared light toward the face of the driver of the vehicle and a near-infrared camera that captures the face of the driver, and causes the near-infrared light emitting element to emit pulse light. and causing the near-infrared camera to image in synchronization with the light emission period and the extinction period of the near-infrared light emitting element;
an image acquisition unit acquiring a luminescence image captured by the near-infrared camera during the luminescence period and an extinction image captured by the near-infrared camera during the extinction period;
a driver state determination unit detecting facial information of the driver from the luminescent image and determining the state of the driver based on the facial information;
A step in which an external light state determination unit detects brightness around the eyes of the driver from the extinguished image, and when the brightness is equal to or greater than a brightness threshold value, determines that the driver's eyes are illuminated with external light. When,
A driver condition monitoring method comprising:

Images