JP2008065705A - Dozing detector and dozing detecting method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a dozing detector and a dozing detecting method, capable of detecting dozing even when a face orientation is downward. <P>SOLUTION: The dozing detector 10 for detecting dozing based on a facial image of a driver comprises a photographing means 2 taking the facial image of the driver, a nostril detecting means detecting a nostril from the facial image taken by the photographing means 2, and a driver state determination means 4c determining that the driver is dozing when no nostril is detected from the facial image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a driver's snoozing detection device and a snoozing detection method, and more particularly to a snoozing detection device and a snoozing detection method for detecting a driver's snoozing by analyzing a driver's face image.

  There has been proposed a method of detecting dozing based on eye-closing time and eye opening in order to detect drowsiness of an operator who drives a vehicle and perform control such as alerting and deceleration (see, for example, Patent Document 1). .) In Patent Document 1, a driver's face image is taken, the eye position is detected from the grayscale image, and the eye opening change is detected based on the increase / decrease of the black eye region. Or, determine that there is a tendency.

Also, in order to detect the face orientation and call attention, a face orientation detection method using the symmetry of the face image has been proposed (see, for example, Patent Document 2). In Patent Document 2, a driver's face image is taken, four feature portions of both eyes and nostril are detected from the driver's face image, and the change in the length of a straight line connecting these four points and the size of the plane. The driver's face direction is detected. If the driver's face direction is detected as being downward for a predetermined time or longer, it can be determined that the driver is falling asleep or has a tendency to doze.
JP 2000-301962 A JP 2005-266868 A

  However, in the dozing detection method that detects a change in the opening degree of the eye as in Patent Document 1, it is necessary to track the position of the eye, and therefore the dozing determination is prohibited when it becomes difficult to track the position of the eye. Therefore, when the face is turned down due to the falling asleep and it becomes difficult to detect the position of the eyes, the dozing determination is not performed, and there is a possibility that the falling asleep will not be detected.

  In addition, when detecting the face orientation by paying attention only to the symmetry of the face, the face orientation cannot be accurately determined due to the light environment at the time of photographing the face image or due to individual differences such as the presence of hairstyle or glasses. There is.

  In view of the above problems, an object of the present invention is to provide a dozing detection device and a dozing detection method that can detect a dozing even when the face is turned downward.

  In order to solve the above-described problems, the present invention is a dozing detection device that detects dozing based on a driver's face image, and includes a photographing unit that captures a driver's face image, and a face image captured by the photographing unit. It has a nostril detection means for detecting a nostril and a driver state determination means for determining that the driver is asleep when no nostril is detected from the face image.

  According to the present invention, since dozing is detected based on the presence or absence of a nostril in the face image, it is not necessary to track the eyes, and even if the driver is in a posture in which the eyes are not detected because the face is turned down, the dozing is detected. be able to.

  In one embodiment of the present invention, when the nostril detection means detects only one nostril, the driver state detection means determines that the driver is facing sideways. According to the present invention, an aside look operation can be detected.

  Also, in one aspect of the present invention, there is a nostril detection level detection means for detecting difficulty in detecting the nostril, and the nostril detection means sets the control value so that the nostril detection becomes easier as the nostril detection becomes difficult. It is characterized by setting.

  According to the present invention, by detecting the level of easy detection of the nostril and making it easy to detect when the nostril is difficult, it is possible to reduce misjudgment and make it difficult for the driver to feel annoyance. .

  It is possible to provide a dozing detection device and a dozing detection method that can detect a dozing even when the face is turned downward.

  DESCRIPTION OF THE PREFERRED EMBODIMENTS The best mode for carrying out the present invention will be described below with reference to the drawings with examples.

  FIG. 1 is a schematic configuration diagram of the dozing detection device 10. The dozing detection device 10 of the present embodiment detects the face direction based on the symmetry of the face image taken by the camera 2 as the first step, and the final driver's face direction based on the nostril detection state as the second step. Or detect dozing. In addition, you may detect dozing based on the detection state of a nostril, without detecting a driver | operator's face direction.

  The camera 2 is disposed at a position where the face of the driver 1 faces from the front or slightly below, for example, in the meter panel 3 or in the steering column. The camera 2 has, for example, a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) photoelectric conversion element, photoelectrically converts incident light according to its intensity, and reads the accumulated charge as a voltage. After further amplification and A / D conversion, a digital image (hereinafter referred to as a face image) having a predetermined luminance gradation (for example, 256 gradations) is output.

  The camera 2 is connected to an image processing ECU (Electronic Control Unit) 4, and the image processing ECU 4 analyzes the driver's face image and detects a doze. When the collision determination ECU 5 receives the dozing detection signal from the image processing ECU 4, the collision determination ECU 5 alerts the driver to the meter ECU 6, the brake ECU 7, and the seat belt ECU 8 connected via an in-vehicle LAN such as a CAN (Controller Area Network). Therefore, a control signal for executing the control is sent out.

  For example, the collision determination ECU 5 communicates with the seat belt ECU 8 to transmit a seat belt winding request signal, and alerts the driver by lighting a warning lamp or sounding an alarm sound to the meter ECU 6. Further, a brake request signal may be sent to the brake ECU 7 to avoid a collision with a preceding vehicle or the like or to reduce the impact.

  The collision determination ECU 5 is connected to a radar device and a lane detection camera. When collision with an obstacle is unavoidable, pre-crash control such as seat belt hoisting and brake control is executed. When there is a possibility of deviating from the traveling lane, control is performed to maintain the lane by steering the electric steering.

  The image processing ECU 4 is a CPU that executes a program, a RAM that is a work area for program execution or temporarily stores data, an EEPROM (Electronically Erasable and Programmable Read Only Memory) that retains data even when the ignition is turned off, An input / output interface that serves as an interface with other ECUs, a communication controller that communicates with other ECUs, a ROM that stores programs, and the like are configured by a microcomputer connected by a bus. When the CPU executes the program, the face orientation detection unit 4a, the nostril detection unit 4b, and the driver state determination unit 4c are realized. Other ECUs such as the collision determination ECU 5 are also configured by a microcomputer in which a CPU, a RAM, an EEPROM, an input / output interface, a communication controller, a ROM, and the like are connected by a bus.

  FIG. 2A shows an example of a face image photographed by the camera 2. The face orientation detection means 4a performs image processing on the face image as shown in FIG. 2A to detect the face orientation, and the nostril detection means 4b detects the nostril. Since there are two nostrils, both are detected if all are detectable, and a single nostril is detected if only one is detectable.

  Detection of face orientation and nostril detection will be described. First, the face orientation detection means 4a detects the face orientation by utilizing the fact that parts such as the eyes and nose of a human face are arranged symmetrically. As shown in FIG. 2B, the face orientation detection unit 4a obtains edge information by differentiating the luminance value of the face image in the vertical direction and the horizontal direction.

  FIG. 2C shows an example of edge information obtained from the face image. Since parts such as eyebrows, eyes, and nostrils are arranged almost evenly on the left and right sides of the human face, it is considered that the edge information obtained from the parts is also distributed almost evenly on the left and right. Therefore, the face direction detection means 4a sets the center line of the face so that the number of left and right edge information is substantially equal, and thereafter, the face direction is detected by tracking the center line.

  By the way, since the background and clothes are reflected in the face image, the edge information may be approximated to a human face, and the face orientation may be detected erroneously. Therefore, in detecting the face orientation and the nostrils, it is preferable to detect the face outline from the face image. The face contour is detected by, for example, a method of calculating a difference between sequentially captured face images, pattern matching using a standard face image (template), or detecting a skin color region using a color face image. The

  When using the differential calculation of sequentially inputted facial images, the contour of the face can be raised from the facial image due to the slight vibration of the car or the occupant. That is, although the background is stationary, the image inside the face outline can move, so that the face outline is detected when the brightness of the outline portion slightly fluctuates. The face orientation detection unit 4a detects a region having a large variation in luminance in a predetermined number of face images as a face contour.

  The nostril detection means 4b detects the nostril from the face orientation and the face outline thus obtained. Since the nostril exists slightly below the center of the face, the detection accuracy can be improved by setting a predetermined area slightly below the center line as the detection target area and detecting the nostril from the detection target area.

  Further, since the nostril is photographed dark in the face image, the nostril is extracted by binarizing the pixel value (luminance) of the face image with a predetermined threshold value for extracting only the dark part. FIG. 3A shows an example of a face image obtained by binarizing the face image shown in FIG. Since binarization only extracts pupils, eyebrows, and the like, detection is made from a detection symmetric area where the number of pixels in the area where the black area is connected is included in a predetermined range about the size of the nostril as a nostril To do. Further, since the nostrils have a substantially elliptical shape, the nostrils may be narrowed down by the aspect ratio of the circumscribed rectangle.

  FIG. 3B is a diagram illustrating an example of a nostril detected in the contour of a face that is raised by image processing. As described above, the nostril detection means 4b detects the nostril from the face image. Then, the driver state determination unit 4c determines that the driver is asleep when the nostril is not detected.

  From the shooting range of the camera 2 and the position of the driver's face, two nostrils can be detected when the driver is facing the front, and either the left or right when the driver is facing the left or right A nostril can be detected. Therefore, when neither of the nostrils is detected, it can be estimated that the driver is depressed, and the driver state determination means 4c is sleeping when the time during which no two nostrils are detected continues for a predetermined period or longer. Is determined.

  By the way, the detection result of the nostril can be fed back to the detection result of the face orientation. When the edge information is detected, the background or the like may be erroneously detected as the edge information provided by the face part as described above, and thus the face contour may be detected at a position that is greatly different in the vertical direction.

  Therefore, when the nostril is detected, the detection accuracy of the face direction can be improved by feeding back the detection position of the nostril to the detection position of the face direction. When two nostrils are detected, the nostril detection means 4b confirms that there is a center line between the two nostrils. When one nostril is detected, the nostril detection means 4b Make sure that there is a center line between the contour line. In addition, when detecting dozing based on the opening of the eye, the eye and the nostril have a relative positional relationship, so the nostril position is effective information for monitoring the opening of the eye.

  Thus, the detection accuracy of the face orientation can be further improved by feeding back the detection result of the nostril to the detection result of the face orientation.

  FIG. 4 is a flowchart showing a procedure in which the dozing detection device 10 detects a dozing. The process of the flowchart in FIG. 4 starts when the ignition is turned on and the shift position is shifted to a forward traveling position such as a drive.

  First, the face direction detection unit 4a detects the face direction from the face image taken by the camera 2, and the nostril detection unit 4b detects the nostril using the face direction detection result (S1). The detection of the nostril may be performed on all face images, or may be performed every predetermined cycle time.

  The driver state determination means 4c determines the state of the driver according to the nostril detection result (S2). If both nostrils are detected, the driver is considered to be facing the front, so the driver state determination means 4c repeats nostril detection.

  When only one nostril is detected, the driver state determination unit 4c determines that the driver is facing either the left or right side (S3). During driving, the driver may look in the horizontal direction for safety confirmation, etc., but if the time in which the driver is facing in the horizontal direction exceeds a predetermined time, a message is displayed or an alarm is sounded.

  In addition, whether or not an alarm is sounded is considered in consideration of the steering direction of the steering, and when the steering direction is turned, prohibiting the alarm sounding can prevent the driver from feeling troublesome. In addition, when the radar or the like detects an obstacle while facing in the horizontal direction, the avoidance operation can be performed with sufficient margin by sounding an alarm earlier than usual.

  And when neither nostril is detected, since it is thought that the driver is facing down, the driver state determination means 4c determines that the driver is asleep (S4). As described above, the image processing ECU 4 sends a doze detection signal to the collision determination ECU 5 (S5). By the above processing, the collision determination ECU 5 can execute control for alerting the driver.

  As a case where no nostrils are detected, there may be a case where the driver is greatly turned sideways. For example, when looking at the navigation screen, it is about 20 degrees with respect to the front, but when looking at the left door mirror, it is 50-60 degrees, the driver's face is horizontal, and there are two nostrils. Neither is detected. In such a case, if it is determined that the face is directed laterally from the face immediately before the nostril disappears, the annoyance can be prevented by prohibiting the alerting control. Also, if the driver is looking backwards because he is traveling backward, an alarm should not be sounded. Therefore, if the shift position “R (rear)” is detected even if the driver is facing sideways, a warning control is performed. Is prohibited.

  According to the present embodiment, since it is not necessary to detect the position and opening degree of the eyes, it is possible to detect dozing even when the eyes are not detected as in the case where the face is lying down. In addition, when only one is detected, a side look can be detected.

  In the present embodiment, a dozing detection device 10 that sets a control value for detecting a nostril according to the photographing state of a face image will be described.

  The detection of the nostril utilizes the fact that the nostril becomes a dark part, but it may be difficult to detect the dark part formed by the nostril due to light adjustment. Therefore, in this embodiment, the control value for detecting the nostril is made variable according to the time and the weather, and the control value is set so that the nostril can be easily detected in the photographing situation where the nostril is difficult to detect.

  This makes it possible to detect a nostril even in a shooting situation in which the nostril is difficult to detect, and can prevent false detection of dozing. The control value for detecting the nostril is, for example, a threshold for binarizing the face image or the number of times the face image for detecting the nostril is captured. In addition, as control values, the reference in the process of detecting nostrils, such as the size of the detection target area provided slightly below the center in the contour, the upper and lower limits for determining the size and shape of the black area of the nostril, etc. Values or combinations thereof may be included.

  There are the following imaging situations that affect the detection of the nostril. Level n (n: 1, 2, 3,...) Represents that the larger the number, the easier the nostril is detected.

Level 1: It is difficult to detect a dark part when it is sunny and daylight, for example, when the incident direction of light and the position of the face are in a backlight relationship. Conversely, when light enters the driver's face from the front, the light enters the nostril, making it difficult to detect the nostril.

Level 2: When the wiper is turned on, the wiper is turned on in rainy weather or the like, but when it is raining, light is directly incident on the inside of the vehicle and the face image is less affected.

Level 3: When the taillight is turned on The taillight is turned on from evening to night, but at night, etc., light is less directly incident than when it is raining, so the detection accuracy of the nostrils is high. Become.

  The dozing detection device 10 detects a photographing situation that affects the detection of the nostril and makes the control value variable. FIG. 5A shows a functional block diagram of the image processing apparatus of the present embodiment. In FIG. 5 (a), the same components as those in FIG.

  5A differs from FIG. 1 in that the image processing ECU 4 has nostril detection level detection means 4d and a control value 4e. The nostril detection level detection means 4d obtains time information, on / off signal, and solar radiation amount information from the “clock, wiper SW, tail lamp SW, solar radiation sensor (conlight sensor)”, and determines the level n indicating the difficulty in detecting the nostrils. To do.

  And the nostril detection means 4b extracts the control value 4e according to this level n, and detects a nostril. FIG. 5B is a diagram showing an example of the relationship between the level n indicating the difficulty of detecting the nostril and the control value 4e. The control value is set such that the smaller the value, the easier the detection of the nostril.

  When the control value is a threshold value for binarization, when the level n is small (detection of the nostril is difficult), the dark portion can be easily detected by reducing the threshold value. Further, in the case where the control value is the number of times the face image is detected to detect the nostril, the nostril can be easily detected by increasing the number of times of shooting when the level n is small.

  Accordingly, the nostril can be detected even in a shooting situation where it is difficult to detect the nostril, and erroneous determination of dozing can be prevented. The relationship between the level n and the control value 4e shown in FIG. 5B is stored in advance in the ROM of the image processing ECU 4.

  FIG. 6 is a flowchart illustrating a procedure in which the dozing detection device 10 detects a dozing according to the shooting state. In FIG. 6, the same steps as those in FIG.

  The flowchart in FIG. 6 is different in that the photographing situation is detected before the nostril is detected (S10). The nostril detection level detection means 4d determines whether or not the taillight is on. If the taillight is on, it is level 3. If the taillight is off and the wiper is on, the level 2 is detected. If it is off, it is determined as level 1. In level 1, it is also possible to detect solar radiation information such as the direction of solar radiation and the amount of solar radiation with a solar radiation sensor, and set the control value accordingly.

  The nostril detection means 4b refers to the control value 4e according to the level n determined by the nostril detection level detection means 4d, and sets the control value according to the level.

  The subsequent steps are the same as the processing procedure of FIG. That is, the nostril detection means 4b detects a nostril (S1), and when both nostrils are not detected, it determines that the driver is asleep (S4).

  According to the present embodiment, by making the threshold value for detecting the nostril variable according to the photographing situation, it is possible to reduce erroneous determination of dozing detection even in the photographing situation in which the nostril is difficult to detect.

  As described above, the dozing detection device 10 according to the present embodiment does not need to detect the position of the eyes, and thus can detect dozing even when the face is turned downward.

It is a schematic block diagram of a dozing detection apparatus. It is an example of the face image image | photographed with the camera. An example of a binarized face image is shown. It is a flowchart figure which shows the procedure in which image processing ECU detects dozing. It is a figure which shows an example of the relationship between the level which shows the difficulty of detection of a nostril, and a control value. It is a flowchart figure which shows the procedure in which image processing ECU detects a doze according to an imaging state.

Explanation of symbols

1 Driver 2 Camera 3 Meter panel 4 Image processing ECU
4a Face direction detection means 4b Nostril detection means 4c Driver state determination means 4d Nostril detection level detection means 4e Control value 5 Collision determination ECU
6 Meter ECU
7 Brake ECU
8 Seat belt ECU

Claims (4)

A dozing detection device that detects dozing based on a driver's face image,
Photographing means for photographing the face image of the driver;
Nostril detection means for detecting a nostril from the face image photographed by the photographing means;
If no nostrils are detected from the face image, driver state determination means for determining that the driver is dozing,
A dozing detection device characterized by comprising: When the nostril detection means detects only one nostril, the driver state detection means determines that the driver is facing sideways,
The dozing detection device according to claim 1. Having nostril detection level detection means for detecting difficulty in detecting nostrils,
The nostril detection means sets the control value so that the nostril detection becomes easier as the detection of the nostril becomes difficult.
The dozing detection device according to claim 1. A dozing detection method for detecting dozing based on a driver's face image,
A shooting step for shooting the face image of the driver;
A snoozing detection device detects a nostril from the face image photographed by the photographing step, and a nostril detection step;
If no nostrils are detected from the face image, a driver state determination step for determining that the driver is dozing,
A dozing detection method characterized by comprising:

Images