JP2007034436A - Arousal estimation device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an arousal estimation device and method, in which estimation accuracy of arousal can be improved even in a case of erroneous detection of eyes. <P>SOLUTION: In the device 30, an open and close eye determination part 33 determines, for the eyes detected by an eye detection part 32, whether the eyes are open or close, and an arousal estimation part 34 estimates the arousal of a subject based on the open/close eye determination results. Further, the eye detection part 32 decides if the detected eyes of the subject results from erroneous detection, and the estimation part 34 decides the arousal of the subject based on the determination results for erroneous detection. Therefore, even if a frame of glasses or the eyebrows are erroneously detected to be the eyes, the arousal is estimated, considering information for the erroneous detection. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an arousal level estimation apparatus and method.

2. Description of the Related Art Conventionally, many wakefulness estimation apparatuses that estimate the wakefulness level of a detected person based on the closed eye duration of the eyes of the detected person are known. As one of them, there is known a wakefulness estimation device that corrects an estimated wakefulness to the wakefulness side when the position of the detected eye varies, that is, when the eye is moving (see Patent Document 1).
Japanese Unexamined Patent Publication No. 2004-341554

  However, in the apparatus described in Patent Document 1, there is a possibility that a frame of eyeglasses or an eyebrow may be erroneously detected as an eye, and in the case of erroneous detection, estimation of the arousal level of the detected person is performed. Accuracy will be reduced.

  The present invention has been made to solve such a conventional problem, and an object thereof is to be awakening capable of improving the estimation accuracy of the awakening degree when an eye is erroneously detected. An object is to provide a degree estimation apparatus and method.

  The arousal level estimation apparatus according to the present invention includes eye detection means, open / closed eye determination means, and arousal level estimation means. The eye detection means inputs image data including the face of the person to be detected, and detects the eye of the person to be detected from this image. The open / close eye determination means determines the open / close eye for the eye detected by the eye detection means. The arousal level estimation means estimates the awakening level of the detected person based on the open / closed eye determination result determined by the open / close eye determination means. Further, the eye detection means determines whether the detected eye of the detected person is due to erroneous detection, and the arousal level estimation means uses the eye detection means in addition to the opening / closing eye determination result determined by the opening / closing eye determination means. Based on the erroneous detection determination result, the awakening level of the detected person is estimated.

  According to the present invention, the open / closed eye is determined for the detected eye, and the arousal level of the detected person is estimated based on the determined open / closed eye determination result. The decrease can be estimated. Furthermore, it is determined whether the detected eye of the detected person is due to erroneous detection, and the degree of arousal of the detected person is estimated based on the determination result of whether the eye is due to erroneous detection. For this reason, for example, when a frame of eyeglasses or an eyebrow is erroneously detected as an eye, the arousal level can be estimated in consideration of the erroneous detection information. Therefore, when the eye is erroneously detected, the estimation accuracy of the arousal level can be improved.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described with reference to the drawings. In the following embodiments, a case where the awakening level estimation device is mounted on a vehicle will be described as an example.

  FIG. 1 is a hardware configuration diagram of an arousal level estimation system 1 according to an embodiment of the present invention. As shown in the figure, the arousal level estimation system 1 of the present embodiment estimates the arousal level of the detected person from the eye-closing frequency of the detected person's eyes, and preferably the arousal level of the vehicle driver. Is used as an estimate. Hereinafter, a wakefulness level estimation system 1 that estimates the wakefulness level of a vehicle driver will be described as an example.

  The awakening level estimation system 1 includes a camera 10, a lighting device 20, and an awakening level estimation device 30. The camera 10 shoots the face of a vehicle driver who is a person to be detected, and is installed near the instrument in front of the driver, that is, in front of the driver's seat so that almost the entire face of the driver can be captured. ing. The camera 10 has a configuration using an image sensor such as a CCD or C-MOS. The camera 10 is configured to output a video signal when the driver's face is photographed.

  The illumination device 20 is provided in the vicinity of the camera 10 and can emit light so as to illuminate the driver's face. The lighting device 20 is configured to emit, for example, infrared light instead of white light so that the driver does not dazzle during driving.

  When the video signal from the camera 10 is input, the arousal level estimation device 30 analyzes the input video signal, determines the eye closure frequency of the detected person's eyes, and estimates the arousal level. Further, the awakening level estimation device 30 is configured to output the information as a waking level signal when the awakening level is estimated. This arousal level signal is connected to, for example, a notification device. The notification device is configured to notify a decrease in arousal level or the like by outputting an alarm sound, music, voice, or the like by controlling an audio device or the like. In addition, the notification device may be configured to notify the driver by display such as an LED or a liquid crystal monitor, or may be configured to notify the driver by controlling a seat belt or an air conditioner. Furthermore, the notification device may be configured to notify the driver by applying vibration to the seat or the handle, or configured to notify the driver by power window opening / closing control, accelerator or brake control, or the like. May be.

  FIG. 2 is a detailed configuration diagram of the arousal level estimation apparatus 30 shown in FIG. As shown in the figure, the arousal level estimation device 30 includes a face image acquisition unit 31, an eye detection unit (eye detection unit) 32, an open / closed eye determination unit (open / closed eye determination unit) 33, and an arousal level determination unit (wake level determination). Means) 34, an open / close determination result processing unit (delay output unit) 35, a processing state output unit 36, and a wakefulness signal output unit 37.

  The face image acquisition unit 31 receives a video signal from the camera 10 and acquires data of an image including the driver's face (hereinafter referred to as a face image) from the video signal.

  The eye detection unit 32 receives the face image data acquired by the face image acquisition unit 31 and detects the driver's eyes from the face image. Here, the eye detection unit 32 includes an eye position detection unit (first detection unit) 32a and an eye position tracking unit (second detection unit) 32b. The eye position detection unit 32a detects the driver's eyes from the entire face image. The eye position tracking unit 32b detects the driver's eyes from a partial region of the face image.

  More specifically, the eye position detection unit 32a detects the eye position from the entire face image when the driver's eye position cannot be detected, and the eye position tracking unit 32b When an eye is detected, a partial area is set including the position of the eye, and the position of the eye is detected from within the area. For this reason, the apparatus 1 performs processing on the entire image at least once, but once the eye is detected, the processing is performed on a part of the image, and the processing is always performed on the entire image. Compared with the case where it carries out, a quick process can be performed now.

  The open / close eye determination unit 33 determines the open / close eye for the eye detected by the eye detection unit 32. The open / close eye determination unit 33 determines the open / closed eye of the eye based on the vertical width of the eye detected by the eye detection unit 32. Here, the open / close eye determination unit 33 includes an open / close eye learning unit (threshold setting unit) 33a and an open / close eye determination unit (open / close eye determination unit) 33b.

The open / close eye learning unit 33a measures the vertical width of the eye detected by the eye detection unit 32 and sets a threshold value for determining the open / closed eye. The open / close eye determination unit 33b determines that the eye is open when the vertical width of the eye detected by the eye detection unit 32 is equal to or greater than the threshold set by the open / close eye learning unit 33a.
When it is less than the threshold value, it is determined that the eye is closed.

  Thus, the open / close eye determination unit 33 measures the vertical width of the eye and sets the threshold for determining the open / closed eye based on the measured vertical width of the eye. An appropriate threshold value can be set after learning.

  The arousal level estimation unit 34 estimates the driver's arousal level based on the open / closed eye determination result determined by the open / closed eye determination unit 33. For example, the arousal level estimation unit 34 estimates a decrease in the arousal level when the open / closed eye determination unit 33 determines that the driver's closed eye ratio is high.

  Further, the eye detection unit 32 described above has a function of determining whether the detected eye is due to erroneous detection. For example, when the distance between the detected left and right eyes is too short, or when the eyebrows are not present above the detected eye image, the eye detection unit 32 determines that the detected eye is due to erroneous detection. Further, when it is determined that the detected eye is due to erroneous detection, the eye detection unit 32 outputs information to that effect to the awakening level estimation unit 34. Thereby, the arousal level estimation unit 34 of the present embodiment determines the driver's arousal level based on the erroneous detection determination result by the eye detection unit 32 in addition to the open / closed eye determination result determined by the open / closed eye determination unit 33. Will be estimated. That is, the arousal level estimation device 30 of the present embodiment estimates the arousal level in consideration of erroneous detection information, and can improve the estimation accuracy.

  More specifically, the eye detection unit 32 stores a condition for determining that the detected eye is due to erroneous detection, and the number of images to be determined as erroneous detection when the condition is satisfied. . Here, when it is determined that an eye is erroneously detected from an image, there is a high possibility that the eye has been erroneously detected before that image. For this reason, the eye detection unit 32 determines that the eye open / closed eye determination result based on the number of images that should be determined as erroneous detection (images corresponding to the number of images that should be determined as erroneous detection) is incorrect when the erroneous detection condition is satisfied. Is output to the arousal level estimation unit 34. Thereby, the arousal level estimation unit 34 can estimate the awakening level more appropriately.

  Here, an example of erroneous detection will be shown. FIG. 3 is an explanatory diagram illustrating an example of erroneous detection, where (a) illustrates a first example and (b) illustrates a second example. As shown in FIG. 3A, for example, when the vehicle driver is wearing glasses, the device 30 may erroneously detect that the glasses frame is an eye. Further, as shown in FIG. 3B, the device 30 may erroneously detect that the driver's eyebrows are eyes. In order to determine erroneous detection in such a case, the eye detection unit 32 stores a condition (see FIG. 18) for determining that the detected eye is due to erroneous detection.

  FIG. 4 is shown as a reference. FIG. 4 is an explanatory diagram illustrating an example of erroneous determination of the open / closed eyes, where (a) illustrates a first example and (b) illustrates a second example. 4 (a) and 4 (b), when the driver's face is tilted (the driver turns his / her face to the left and right), the possibility of erroneously determining that the eyes are open is increased. I'll add it.

  Reference is again made to FIG. The open / closed determination result processing unit 35 inputs the open / closed eye determination result determined by the open / closed eye determination unit 33, delays it for a predetermined time, and outputs it to the arousal level estimation unit 34. That is, the opening / closing eye determination result determined by the opening / closing eye determination unit 33 is not immediately input to the awakening level estimation unit 34 by the opening / closing determination result processing unit 35, and is not used for the awakening level estimation unless a predetermined time has elapsed. It becomes. The eye detection unit 32 is configured to determine whether the detected driver's eyes are due to erroneous detection during the predetermined time delay. That is, the arousal level estimation apparatus 30 according to the present embodiment prevents the open / closed eye determination result from being immediately used for the awakening level estimation due to the delay by the open / close determination result processing unit 35, and considers whether or not it is erroneously detected. The arousal level is estimated.

  The processing state output unit 36 determines the state of the arousal level estimation device 30 according to the operating state of each unit 31 to 35 and outputs information on the determined state. Specifically, the processing state output unit 36 determines and outputs at least three states of “being detected”, “learning”, and “being estimated”. Here, “being detected” means a state where eyes are detected from the entire face image. “Learning” means a state in which a threshold value for determining an open / closed eye is learned. “During estimation” means a state in which the driver's arousal level is estimated.

  The arousal level signal output unit 37 generates and outputs an arousal level signal from the information on the arousal level estimated by the arousal level estimation unit 34. As described above, the awakening level signal output unit 37 outputs the awakening level signal to a notification device or the like.

  Next, an outline of the operation of the arousal level estimation apparatus 30 according to the present embodiment will be described. FIG. 5 is an explanatory diagram showing the operation of the arousal level estimation apparatus 30 according to the present embodiment, where (a) is a state transition diagram, and (b) is each part when the apparatus 30 is “detecting”. The mode of operation | movement of 31-37 is shown. (C) shows the behavior of the units 31 to 37 when the device 30 is “learning”, and (d) shows the operations of the units 31 to 37 when the device 30 is “estimating”. The state of operation is shown.

  First, when the ignition switch of the vehicle is turned on, the arousal level estimation device 30 is activated, and the state A is “detecting” shown in FIG. In this state A, the arousal level estimation device 30 detects the eyes of the vehicle driver from the entire face image. And the awakening degree estimation apparatus 30 repeats this state until detection is completed. At this time, as shown in FIG. 5B, the eye detection is performed by operating the face image acquisition unit 31, the eye position detection unit 32 a, and the processing state output unit 36 of the arousal level estimation device 30. It becomes. And if an eye is detected, it will transfer from the state A to the other states B and C.

  If the eye is detected in the “detecting” state A and the eye learning is not completed, the wakefulness estimation device 30 enters the “learning” state B, and the eye learning (that is, threshold setting) is performed. Do. And the arousal level estimation apparatus 30 repeats this state until learning is completed. At this time, as shown in FIG. 5C, the face image acquisition unit 31, the eye position tracking unit 32b, the open / closed eye learning unit 33a, and the processing state output unit 36 of the arousal level estimation device 30 are activated, Will be learned. When the learning is completed, the state B is shifted to the “estimating” state C. On the other hand, if the eye is completely lost during eye learning, the arousal level estimation device 30 transitions to the “detecting” state A again.

  When the eye is detected in the “detecting” state A and the eye learning is completed, or when the eye learning is completed in the “learning” state B, the wakefulness estimation device 30 is configured as shown in FIG. The state C is in the “estimating” state shown in FIG. 5 and the driver's arousal level is estimated. Thereafter, the awakening level estimation device 30 repeatedly estimates the awakening level. Then, the arousal level signal output unit 37 outputs the arousal level signal to a notification device or the like. In the state C, as shown in FIG. 5D, the face image acquisition unit 31, the eye position tracking unit 32b, the open / closed eye learning unit 33a, the open / closed eye determination unit 33b, and the arousal level in the awakening level estimation device 30. The estimation unit 34, the open / close determination result processing unit 35, the processing state output unit 36, and the arousal level signal output unit 37 are activated. Also, the awakening level estimation device 30 transitions to the “detecting” state A again when the eye is completely lost during the estimation of the awakening level.

  Next, the arousal level estimation method according to the embodiment of the present invention will be described. FIG. 6 is a flowchart showing details of the arousal level estimation method according to the present embodiment. First, the arousal level estimation device 30 receives a video signal from the camera 10. And as shown in FIG. 6, the eye detection part 32 judges whether the eye was detected in the last process (ST1). Here, when the eye has not been detected in the previous process (ST1: NO), the eye position detection unit 32a of the eye detection unit 32 executes the eye detection process (ST2).

  7 to 9 are explanatory views showing an outline of the eye detection process (ST2) shown in FIG. 6, and FIG. 7 is an explanatory view showing an example of image data input by the eye detecting unit 32. FIG. FIG. 9 is a graph showing the density value of the pixel column Yb in the vertical direction of the image shown in FIG. 7, and FIG. 9 is an explanatory diagram showing how the eye is detected.

  When the image as shown in FIG. 7 is input, the eye position detection unit 32a detects the pixel density along the vertical pixel column (vertical line) of the image, and each local increase in density in the pixel column. Each element is defined as an extraction point. Then, the eye position detection unit 32a determines extraction points for all lines in the pixel column in the vertical direction of the image.

  Here, as shown in FIG. 8, the density value of the pixel column Yb in the vertical direction of the image is locally reduced at the eye, the spectacle frame, and the mouth. For this reason, the eye position detection unit 32a determines the pixels of the eye, the spectacle frame, and the mouth portion as extraction points for the Yb line.

  Thereafter, the eye position detection unit 32a obtains a pixel group in which the extraction points are assembled. Thereby, a pixel group is obtained near the eye, the spectacle frame, and the mouth. Here, the eye has a shape extending in the lateral direction. For this reason, the eye pixel group shown in FIG. 9 is continuous in the horizontal direction of the image in shape. Therefore, the eye position detection unit 32a can extract the eye pixel group by selecting the pixel group extending in the horizontal direction. Note that the eyeglass frame and the like also extend in the horizontal direction, so the eye position detection unit 32a more accurately specifies the eye by using the preset eye length and the left-right symmetry. May be. Further, the eye position detection unit 32a may use a template matching process together.

  In this way, the eye position detection unit 32a detects the eyes from the entire face image. FIG. 10 is a flowchart showing details of the eye detection process (ST2) shown in FIG. In the present embodiment, the eye position detection unit 32a first detects the position of one eye, and then detects both eyes based on the detected one eye. Accordingly, the position detection of one eye is initially executed.

  As shown in FIG. 10, first, the eye position detection unit 32a determines whether to perform one-eye candidate detection or to perform binocular determination processing (ST21). If it is determined that single eye candidate detection is to be performed (ST21: single eye candidate detection), the eye position detection unit 32a provides a single eye candidate detection processing area on the face image (ST22).

  FIG. 11 is an explanatory diagram showing details of the eye detection process (ST2) shown in FIG. 6, where (a) shows a processing area for detecting one eye candidate, and (b) shows detected continuous data. (C) shows a processing area for binocular determination. As shown in FIG. 11A, the eye position detection unit 32a sets a processing area for detecting one eye candidate on the face image. This processing area is set, for example, in the right half or the left half of the image.

  Refer to FIG. 10 again. After setting the processing area, the eye position detection unit 32a obtains continuous data from within the processing area (ST23). At this time, the eye position detection unit 32a obtains continuous data as described with reference to FIGS. In the example illustrated in FIG. 11, the eye position detection unit 32a determines extraction points for each pixel column in the vertical direction at intervals of 10 pixels.

  Thereafter, the eye position detection unit 32a narrows down one eye candidates (ST24). That is, as shown in FIG. 11B, the eye position detection unit 32a narrows down the continuous data for the eyes from a plurality of continuous data. At this time, the eye position detection unit 32a narrows down the continuous data of the eyes based on the upwardly convex shape for the upper eyelid. Further, the eye position detection unit 32a may narrow down the continuous data based on a predetermined eye length or left-right symmetry as described above. Here, in the example shown in FIG. 11B, the continuous data G1 that is convex upward among the continuous data G1 and G2 is narrowed down as the continuous data of the eyes.

  Refer to FIG. 10 again. After narrowing down one eye candidate, the eye position detection unit 32a determines whether to perform one eye candidate detection or binocular confirmation processing in the next frame (ST25). Here, in the processing of steps ST23 and ST24, if the detection of one eye candidate has failed, the eye position detection unit 32a determines that one eye candidate detection is performed in the next frame. On the other hand, when the one-eye candidate has been successfully detected, the eye position detection unit 32a determines that the binocular determination process is performed in the next frame. And a process transfers to step ST5 of FIG. Note that the eye position detection unit 32a may perform one-eye candidate detection continuously until the one-eye candidate is successfully detected several times even if the one-eye candidate is successfully detected.

  When it is determined that binocular confirmation processing is to be performed (ST21: binocular confirmation processing), the eye position detection unit 32a provides a binocular confirmation processing area on the face image (ST22). At this time, as shown in FIG. 11C, the eye position detection unit 32a sets a binocular determination processing area on the face image based on the previously detected position of one eye.

  Next, the eye position detection unit 32a obtains continuous data from the processing area (ST27), and detects both eye candidates (ST28). At this time, the eye position detection unit 32a obtains continuous data and detects binocular candidates as in steps ST23 and ST24. In step ST28, the eye position detection unit 32a performs pattern recognition (template matching) and determines whether or not the binocular candidate is the driver's eyes. Here, a plurality of templates used for pattern recognition are registered in advance, and a cross-correlation method is employed for determining whether the eye is binocular. The eye position detection unit 32a may include a neural network, and may determine whether the eye is a binocular through learning of the neural network.

  Thereafter, the eye position detection unit 32a determines whether to perform one-eye candidate detection in the next frame, to perform binocular confirmation processing, or to perform eye tracking (that is, eye detection from a partial area). (ST29). Here, in the processes of steps ST27 and ST28, when the binocular confirmation has failed and the one-eye candidate is lost or the detected one-eye candidate is not an eye in the first place, the eye position detection unit 32a performs the next step. It is determined that one-eye candidate detection is performed in the frame. If the binocular confirmation fails but the position of the one-eye candidate is not lost, the eye position detection unit 32a determines that the binocular confirmation process is performed in the next frame. When the binocular confirmation is successful, the eye position detection unit 32a determines that the eye is to be tracked in the next frame. And a process transfers to step ST5 of FIG.

  FIG. 6 will be referred to again. In step ST1 of FIG. 6, when an eye has not been detected in the previous process (ST1: NO), the eye position tracking unit 32b of the eye detection unit 32 sets a face part search region (partial region) (ST3). ). At this time, the eye position tracking unit 32b sets a face part search region around the eye position, including the previously detected eye position. 12A and 12B are explanatory diagrams showing an outline of the operation of the eye position tracking unit 32b. FIG. 12A shows a face part search area, and FIG. 12B shows how the opposite eye is detected. As shown in FIG. 12A, the eye position tracking unit 32b sets a rectangular face part search region on the face image with the position of one eye (reference eye) detected last time as the center. The face part search area is, for example, a size obtained by adding a margin (for example, 0.5 times the size of the eye) to the size of the entire eye.

  Please refer to FIG. After setting the face part exploration region, the eye position tracking unit 32b executes eye tracking processing (ST4). In the eye tracking process, the reference eye is detected from the face part search area shown in FIG. Then, based on the detected position of the reference eye and the interocular distance calculated up to the previous processing, the face part search region of the other eye (opposite eye) as shown in FIG. 12B is set. Is done. Next, the opposite eye is detected from the face region search region of the opposite eye.

  FIG. 13 is a flowchart showing details of the eye tracking process (ST4) shown in FIG. First, the eye position tracking unit 32b detects the reference eye from the face part search region (ST31). The detection of the reference eye is performed in the same manner as described with reference to FIGS. Thereafter, the eye position tracking unit 32b determines whether or not the reference eye has been detected (ST32).

  If it is determined that the reference eye has been detected (ST32: YES), the process proceeds to step ST37. On the other hand, when it is determined that the reference eye could not be detected (ST32: NO), the eye position tracking unit 32b sets the opposite eye as the reference eye (ST33) and detects the reference eye (ST34). Then, the eye position tracking unit 32b determines whether or not the reference eye has been detected (ST35).

  If it is determined that the reference eye has not been detected (ST35: NO), the eye position tracking unit 32b determines that eye tracking has failed, and the process proceeds to step ST5 in FIG. On the other hand, if it is determined that the reference eye has been detected (ST35: YES), the process proceeds to step ST37.

  In step ST37, the eye position tracking unit 32b detects the opposite eye (ST37). Then, the eye position tracking unit 32b determines whether or not the opposite eye has been detected (ST38). If it is determined that the opposite eye has been detected (ST38: YES), the coordinate position of the detected eye is determined as the reference eye and the opposite eye (ST39). Thereafter, the process proceeds to step ST5 in FIG.

  On the other hand, when it is determined that the opposite eye could not be detected (ST38: NO), the eye position tracking unit 32b fixes the detected coordinates of the reference eye as the position of the reference eye, and changes from the previous reference eye to the current reference eye. It is assumed that the opposite eye has moved by the amount of movement to, and the position obtained based on that assumption is determined as the position of the opposite eye. Thereafter, the process proceeds to step ST5 in FIG.

  In step ST5 of FIG. 6, the eye detection unit 32 determines whether or not the eye has been detected (ST5). When the eye cannot be detected (ST5: NO), the eye detection unit 32 outputs information indicating that the eye has not been detected to the arousal level estimation unit 34 (ST6). Then, the process ends. On the other hand, when the eye can be detected (ST5: YES), the open / close eye determination unit 33 performs open / close eye determination (ST7).

  FIG. 14 is a flowchart showing details of the open / close eye determination (ST7) shown in FIG. As shown in FIG. 14, the open / close eye determination unit 33 first sets an open / close diagnosis parameter (ST41). FIG. 15 is an explanatory diagram of the open / close eye determination (ST7) shown in FIG. 6, and (a) shows details of step ST41 of FIG. As shown in FIG. 15A, the open / close eye determination unit 33 sets the upper, lower, left, and right ends of the eye when the eye is closed and when the eye is opened as the open / close diagnosis parameter. As a result, the open / closed eye determination unit 33 can recognize the upper end, lower end, right end, and left end positions of both eyes when the eyes are open and when the eyes are closed.

  Reference is again made to FIG. After setting the opening / closing diagnosis parameters, the opening / closing eye determination unit 33 obtains the opening degree of the eye detected in step ST2 or step ST4 (ST42). FIG. 15B shows details of step ST42 in FIG. As shown in the figure, the opening degree of the eye means the vertical width of the eye. For this reason, the open / close eye determination unit 33 detects the upper and lower positions of the eye and obtains the vertical width of the eye from the difference between these positions.

  Reference is again made to FIG. After obtaining the opening, the open / close eye determination unit 33 determines whether the eye is convex upward, convex downward, or horizontal (ST43). Next, the open / close eye determination unit 33 determines whether or not the open / close learning is finished (ST44). When it is determined that the opening / closing learning has not ended (ST44: NO), the opening / closing eye learning unit 33a performs learning (ST45). At this time, the open / close eye learning unit 33a learns the opening degree of the eye determined to be convex upward in step ST43, and obtains a threshold value for determining the open / close eye.

  FIG. 16 is an explanatory diagram showing details of step ST45 shown in FIG. As shown in the figure, the open / close eye learning unit 33a includes a counter therein, and records the value of the opening detected in step ST42 in the counter. Then, after recording in the counter, the process proceeds to step ST8 in FIG. When the number of opening values recorded in the counter reaches a certain amount, the open / close eye learning unit 33a sums three consecutive values among the values recorded in the counter, and the total value is a constant value. If it exceeds (for example, “20” in FIG. 16), learning is performed by determining that the opening is detected from an upwardly convex eye. Then, the opening / closing eye learning unit 33a obtains an average value of the learned opening values, and sets a value of about 30% of the average value as a threshold for determining the opening / closing eyes. After obtaining the threshold value, the process proceeds to step ST8 in FIG.

  By the way, when it is determined that the learning of opening / closing has been completed (ST44: YES), the opening / closing eye determination unit 33b compares the opening detected in step ST42 with the threshold, and determines that the opening is open when the opening is equal to or greater than the threshold. When the degree is less than the threshold, it is determined that the eyes are closed. FIG. 17 is a second explanatory diagram showing details of step ST45 shown in FIG. As shown in the figure, the case where the opening degree is lower than the threshold value is determined to be closed. Then, after the opening / closing eye is determined, the process proceeds to step ST8 in FIG.

  The opening / closing eye determination result obtained in step ST7 is not immediately output to the arousal level estimation unit 34 as described above, but is delayed for a predetermined time (as shown in step ST12 described later). It will be output to the arousal level estimation unit 34.

  Please refer to FIG. In step ST8, the open / close eye determination unit 33 determines whether learning has been completed (ST8). If learning has not been completed (ST8: NO), the process ends. On the other hand, if the learning has been completed (ST8: YES), the eye detection unit 32 makes an erroneous detection determination (ST9). At this time, as described above, the eye detection unit 32 determines that the detection is erroneous when the distance between the detected left and right eyes is too short, or when the eyebrows are not present above the detected eye image.

  FIG. 18 is an explanatory diagram showing details of step ST8 shown in FIG. As illustrated in FIG. 18, the eye detection unit 32 determines that a false detection has occurred when any one or more of the conditions 2 to 8 for determining that the detected eye is due to a false detection are satisfied. When the eye detection unit 32 determines erroneous detection, the eye detection unit 32 outputs information to that effect to the awakening level estimation unit 34. Further, the eye detection unit 32 identifies images that are erroneously detected based on the information on the number of images, and outputs information that the open / closed eye determination result obtained from these images is incorrect to the arousal level estimation unit 34. To do.

  Furthermore, the eye detection unit 32 determines whether or not an eye has not been detected in step ST8. Specifically, when the condition 1 and the conditions 9 to 13 in FIG. 18 are satisfied, it is determined that the eye is not detected, that is, the eye cannot be detected. The undetected determination may be executed in step ST8, but may be executed in the above-described step ST2 or step ST4.

  Please refer to FIG. After the erroneous detection determination, the open / close determination result processing unit 35 determines whether or not replacement is required (ST10). Here, the replacement will be described. FIG. 19 is an explanatory diagram showing details of the replacement determination (ST10) shown in FIG. 6, in which (a) shows a case where replacement is necessary, and (b) shows a case where replacement is not necessary.

  Specifically, the opening / closing determination result processing unit 35, when the time-series opening / closing eye determination result input during the delay of a predetermined time corresponds to a pattern stored in advance, which is inappropriate opening / closing of the driver's eyes, An appropriate opening and closing of the driver's eyes will be replaced with a previously stored pattern. For example, as illustrated in FIG. 19A, it is assumed that the time-series open / closed eye determination result continues as “closed eyes”, “open eyes”, and “closed eyes”. Here, in general, it is difficult to think of a movement of a human eye that opens for a moment during closing and returns to closing again. For this reason, the open / close determination result processing unit 35 determines that the “open eye” among the “closed eyes”, “open eyes”, and “closed eyes” is incorrect, and determines that the determination result of “open eyes” should be replaced with “closed eyes”.

  Further, as shown in FIG. 19B, it is assumed that the time-series open / closed eye determination result continues as “open eye”, “closed eye”, and “open eye”. Here, closing the eye for a moment during eye opening means that a normal blinking operation has been performed, and this is a possible human action. For this reason, the open / close determination result processing unit 35 determines that there is no error and determines that replacement is not necessary.

  Please refer to FIG. If it is determined in step ST10 that replacement is not necessary (ST10: NO), the open / close determination result processing unit 35 does not perform replacement, and the process proceeds to step ST12. On the other hand, when it is determined that the replacement is necessary (ST10: YES), the open / close determination result processing unit 35 performs the replacement, and then the process proceeds to step ST12.

  In step ST12, the opening / closing determination result processing unit 35 outputs the opening / closing eye determination result that has passed (delayed) for a predetermined time after acquisition to the awakening degree estimation unit 34 among the acquired opening / closing eye determination results (ST12).

  FIG. 20 is an explanatory diagram showing details of the delay processing (ST12) shown in FIG. As illustrated in FIG. 20, the open / close determination result processing unit 35 includes a plurality of storage areas, and stores the open / close eye determination results from the open / close eye determination unit 33 in the storage area as needed. Then, after the storage, the open / closed eye determination result after a predetermined time is output to the arousal level estimation unit 34.

  Please refer to FIG. After the output, the arousal level estimation unit 34 estimates the driver's arousal level based on the input opening / closing eye determination result (ST13). Thereafter, the process ends. The above process is repeatedly executed until the ignition switch of the vehicle is turned off and the arousal level estimation device 30 is stopped.

  FIG. 21 is an explanatory diagram showing details of the arousal level estimation process (ST13) shown in FIG. The arousal level estimation unit 34 obtains a closed eye time ratio when estimating the arousal level. Specifically, the arousal level estimation unit 34 closes the eye t6 (= t2−) out of the time t2 excluding the time t4 when the eye was not detected and the time t3 when the eye was erroneously detected from the specified time t1. The proportion of the time at t5) is defined as the closed eye time ratio, and the arousal level is estimated from this closed eye time ratio. As described above, the awakening level estimation unit 34 sets the ratio of the time t6 that has been closed to the closed time ratio out of the time t2 excluding undetected and erroneous detection. Inaccuracy is suppressed, and the arousal level can be accurately estimated.

  FIG. 22 is a second explanatory diagram showing details of the arousal level estimation process (ST13) shown in FIG. The awakening level estimation unit 34 determines the ratio of the time t6 (t2′−t6 ′ + t7 + t8) in which the eyes are closed out of the time t2 ′ excluding the time t4 when the eyes cannot be detected from the specified time t1 as the closed eye time ratio. And the degree of arousal may be estimated from the closed eye time ratio. Here, if the erroneous detection time t7 + t8 is added to the eye closing time t6, the eye closing time ratio becomes excessively high. On the other hand, even if the erroneous detection time t7 + t8 is added to the eye opening time t5 and a new eye opening time t5 'is exerted, the effect on the eye closing time ratio is small. For this reason, the arousal level estimation unit 34 determines the ratio of the closed eye time ratio of the closed eye time t6, the open eye time t5, and the false detection time t7 + t8 that excludes the undetected time t4 from the specified time t1 to the closed eye time ratio. By doing so, it is possible to accurately estimate the arousal level.

  Next, the case where the arousal level estimation part 34 is comprised so that a several awakening level estimation method is executable is demonstrated. When the arousal level estimation unit 34 estimates awakening for each arousal level estimation method based on a plurality of arousal level estimation methods, the open / close determination result processing unit 35 varies the predetermined time to be delayed for each awakening level estimation method.

  This will be described more specifically. FIG. 23 is a second explanatory diagram showing details of the delay process (ST12) shown in FIG. 6, and shows a state of delay when executing a plurality of arousal level estimation methods.

For example, the first arousal level estimation method is a method for estimating a decrease in arousal level based on a closed eye time ratio, and the second awakening level estimation method is a method for estimating a decrease in arousal level based on a transition of the closed eye time ratio. Suppose there is. Here, when estimating a decrease in the arousal level based on the transition of the closed eye time ratio as in the second arousal level estimation method, it is desirable to check the transition of the closed eye time ratio in real time as much as possible. The decrease in the arousal level is estimated based on the transition of the eye-closing time ratio in a somewhat shorter time such as 3 seconds. On the other hand, in the first arousal level estimation method that does not require confirmation of the state of change, a decrease in the arousal level is estimated based on a closed eye time ratio in a somewhat longer time such as 30 seconds.

  In this case, in the first arousal level estimation method, the delay time is preferably about 30 seconds corresponding to the above, and in the second arousal level estimation method, the delay time is similarly corresponding to about 3 seconds. It is desirable. Thus, by making the predetermined time of the second arousal level estimation method shorter than the predetermined time of the first arousal level estimation method, the delay for each arousal level estimation method is not too short and not too long. it can.

  FIG. 24 is an explanatory diagram illustrating a calculation result of the eye-closing time ratio by the arousal level estimation apparatus 30 according to the present embodiment. In the example shown in FIG. 24, the eye-closing time ratio is calculated when the driver is driving normally, when the driver's face is shaken, or when the driver's arousal level is low. In the example shown in FIG. 24, when the eye-closing time ratio becomes larger than the threshold value, it is determined that the driver's arousal level is reduced.

  As shown in FIG. 24, in the conventional method, the eye-closing time ratio may exceed the threshold during normal operation due to erroneous detection. However, in the present embodiment, since the closed eye time ratio is calculated in consideration of information at the time of erroneous detection, the arousal level is accurately estimated without the calculated closed eye time ratio exceeding the threshold value. Further, when the driver's face is shaken, the eye-closing time ratio does not exceed the threshold for both the conventional method and the present embodiment. Further, in the present embodiment, when the driver's arousal level is reduced, the closed eye ratio is calculated to be high as in the conventional method without reduction.

  FIG. 25 is an explanatory diagram showing the calculation result of the eye-closing time ratio and the warning output by the arousal level estimation apparatus 30 according to the present embodiment. In the example shown in FIG. 25, the eye-closing time ratio is simply calculated by having the driver drive for a long time. Also, the judgment threshold for alarm output was set to 25% of the closed eye time ratio.

  In the conventional method, the eye-closing time ratio increases due to erroneous detection. For this reason, an unnecessary alarm output is invited at the time of 13 minutes shown in FIG. However, in this embodiment, since an increase in the eye-closing time ratio due to erroneous detection can be suppressed, only a time point of 16 minutes when the driver's arousal level is reduced without an alarm being output at the time point of 13 minutes. An alarm is output at.

  Thus, according to the arousal level estimation apparatus 30 and the method according to the present embodiment, the open / closed eye is determined for the detected eye, and the driver's arousal level is estimated based on the determined open / closed eye determination result. Therefore, it is possible to estimate a decrease in arousal level when closed eyes frequently appear. Furthermore, it is determined whether the detected eye of the detected person is due to erroneous detection, and the degree of arousal of the detected person is determined based on the determination result of whether the eye is due to erroneous detection. For this reason, for example, when a frame of eyeglasses or an eyebrow is erroneously detected as an eye, the arousal level can be estimated in consideration of the erroneous detection information. Therefore, when the eye is erroneously detected, the estimation accuracy of the arousal level can be improved.

  In addition, the eye position tracking unit 32b includes either the eye position detected by the eye position detection unit 32a or the eye position detected by the previous eye position tracking unit 32b. ) Is set. For this reason, once an eye is detected by the eye position detection unit 32a, the eye of the detection subject is detected from a partial area until the eye position tracking unit 32b fails to detect the eye thereafter. As a result, it is not necessary to detect eyes from the entire face image every frame, and the processing can be speeded up.

  Further, a threshold value for determining the open / closed eye is set by measuring the vertical width of the detected eye, and when the detected vertical eye width is equal to or greater than the set threshold value, the eye is determined to be open, and the vertical width is the threshold value. If it is less than that, it is decided to close the eyes. For this reason, it is possible to set an appropriate threshold value after learning the vertical width of the eyes of the individual to be detected, and to accurately determine the open / closed eyes.

  Further, the eye detection unit 32 stores a condition for determining that the detected eye is due to erroneous detection, and the number of images to be determined as erroneous detection when the condition is satisfied. The information that the eye open / closed eye determination result based on the image determined to be incorrect is output to the arousal level estimation unit 34. For this reason, the arousal level estimation unit 34 can take appropriate measures such as not estimating the arousal level from an image that may be erroneously detected. Therefore, it is possible to improve the estimation accuracy of the arousal level.

  In addition, it further includes an opening / closing determination result processing unit 35 that inputs the determined opening / closing eye determination result, delays it for a predetermined time, and outputs it to the arousal level estimation unit 34, and the eye detection unit 32 performs a predetermined operation by the opening / closing determination result processing unit 35. During the time delay, it is determined whether the detected eye of the detected person is due to erroneous detection. For this reason, the situation that the estimation of the arousal level is finished when the eye detection unit 32 determines that the detection is false is prevented without delay. Therefore, it is possible to estimate the arousal level by taking into account whether or not a false detection has occurred.

  In addition, when the time-series opening / closing eye determination result input during the delay of a predetermined time corresponds to a pattern stored in advance that is an inappropriate opening / closing of the eye of the detected person, the appropriate eye of the detected person's eye The pattern is opened and closed and replaced with a previously stored pattern. For this reason, for example, when an eye is opened for only one frame due to spike noise while the eyes are closed, it is possible to replace the eye for one frame with the eye opening, and to suppress a decrease in the estimation accuracy of the arousal level due to the spike noise. it can.

  Further, of the time t2 excluding the time t4 when the eye could not be detected and the time t3 when the eye was erroneously detected from the specified time t1, the ratio occupied by the time t6 that was closed is defined as the closed eye ratio. Estimate arousal level from time ratio. For this reason, of the closed eye time t6 and the open eye time t5 excluding the undetected time t4 and the erroneous detection time t3, the ratio occupied by the closed eye time t6 is set as the closed eye time ratio, so that erroneous detection and undetected are excluded. Therefore, the arousal level can be estimated accurately.

  Alternatively, of the time excluding the time t4 when the eye could not be detected from the specified time t1, the ratio occupied by the time t6 when the eyes were closed is defined as the closed eye time ratio, and the arousal level is estimated from the closed eye time ratio. Here, if the erroneous detection time t4 is added to the eye closing time t6, the eye closing time ratio becomes excessively high. On the other hand, even if the erroneous detection time t4 is added to the eye opening time t5, the influence on the eye closing time ratio is small. For this reason, the awakening degree is accurately estimated by setting the ratio of the closed eye time t5 to the closed eye time t6, the open eye time t5 excluding the undetected time t4, and the erroneous detection time t7 + t8 as the closed eye ratio. can do.

  In addition, since the predetermined delay time is made different according to each of the plurality of arousal level estimation methods, the delay time is not too short and not too long. Can be.

  In particular, the second arousal level estimation method for estimating a decrease in arousal level based on the closed eye time ratio and the second awakening level estimation method for estimating a decrease in arousal level based on the transition of the closed eye ratio The predetermined time of the degree estimation method is shorter than the predetermined time of the first arousal level estimation method. For example, the first arousal level estimation method estimates a decrease in the arousal level based on the closed eye time ratio. That is, it is assumed that the first arousal level estimation method is a method for estimating a decrease in the arousal level based on a closed eye time ratio that occupies a somewhat longer time such as 30 seconds. In addition, the second arousal level estimation method is a method for estimating a decrease in arousal level based on a transition of the closed eye time ratio in a somewhat shorter time such as 3 seconds.

  In this case, in the first arousal level estimation method, the delay time is preferably about 30 seconds, and in the second arousal level estimation method, the delay time is preferably about 3 seconds. Thus, by making the predetermined time of the second arousal level estimation method shorter than the predetermined time of the first arousal level estimation method, the delay for each arousal level estimation method is not too short and not too long. it can.

  As described above, the present invention has been described based on the embodiment, but the present invention is not limited to the above embodiment, and may be modified without departing from the gist of the present invention. For example, in the above-described embodiment, the example in which the vehicle driver is the detected person has been described. However, the present invention is not limited to this, and the driver of another vehicle may be imaged as the detected person. Further, the arousal level estimation device 30 may be a type of test device that tracks the eye of the subject and records the eye movement of the subject.

It is a hardware block diagram of the arousal level estimation system which concerns on embodiment of this invention. It is a detailed block diagram of the arousal level estimation apparatus shown in FIG. It is explanatory drawing which shows the example of a misdetection, (a) shows the 1st example and (b) has shown the 2nd example. It is explanatory drawing which shows the example of the misjudgment of an opening-and-closing eye, (a) shows the 1st example, (b) has shown the 2nd example. It is explanatory drawing which shows operation | movement of the arousal level estimation apparatus which concerns on this embodiment, (a) is a state transition diagram, (b) shows the mode of operation | movement of each part when this apparatus is "under detection". (C) shows the operation of each part when the apparatus is “learning”, and (d) shows the operation of each part when the apparatus is “estimating”. It is a flowchart which shows the detail of the arousal level estimation method which concerns on this embodiment. It is explanatory drawing which shows the outline | summary of the eye detection process (ST2) shown in FIG. 6, and is explanatory drawing which shows an example of the image data which an eye detection part inputs. FIG. 8 is an explanatory diagram showing an outline of the eye detection process (ST2) shown in FIG. 6, and is a graph showing the density value of the pixel column Yb in the image vertical direction shown in FIG. It is explanatory drawing which shows the outline | summary of the eye detection process (ST2) shown in FIG. 6, and is explanatory drawing which shows the mode of an eye detection. It is a flowchart which shows the detail of the eye detection process (ST2) shown in FIG. It is explanatory drawing which shows the detail of the eye detection process (ST2) shown in FIG. 6, (a) has shown the process area for one eye candidate detection, (b) has shown the detected continuous data, (C) shows a processing area for binocular determination. It is explanatory drawing which shows the outline | summary of operation | movement of an eye position tracking part, (a) shows a face part search area | region, (b) has shown the mode of detection of the eye of the other side. It is a flowchart which shows the detail of the eye tracking process (ST4) shown in FIG. It is a flowchart which shows the detail of opening-and-closing eye determination (ST7) shown in FIG. It is explanatory drawing of the opening-and-closing eye determination (ST7) shown in FIG. 6, (a) shows the detail of step ST41 of FIG. 14, (b) has shown the detail of step ST42 of FIG. It is explanatory drawing which shows the detail of step ST45 shown in FIG. FIG. 16 is a second explanatory diagram showing details of step ST45 shown in FIG. 15. It is explanatory drawing which shows the detail of step ST8 shown in FIG. FIG. 7 is an explanatory diagram showing details of the replacement determination (ST10) shown in FIG. 6, in which (a) shows a case where replacement is necessary, and (b) shows a case where replacement is unnecessary. It is explanatory drawing which shows the detail of the delay process (ST12) shown in FIG. It is explanatory drawing which shows the detail of the arousal level estimation process (ST13) shown in FIG. FIG. 7 is a second explanatory diagram showing details of the arousal level estimation process (ST13) shown in FIG. FIG. 7 is a second explanatory diagram showing details of the delay process (ST12) shown in FIG. 6 and shows a state of delay when executing a plurality of arousal level estimation methods. It is explanatory drawing which shows the calculation result of the eye-closing time ratio by the arousal level estimation apparatus which concerns on this embodiment. It is explanatory drawing which shows the calculation result and warning output of the eye-closing time ratio by the arousal level estimation apparatus which concerns on this embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Arousal degree estimation system 10 ... Camera 20 ... Illuminating device 30 ... Arousal degree estimation apparatus 31 ... Face image acquisition part 32 ... Eye detection part (eye detection means)
32a ... Eye position detection unit (first detection means)
32b ... Eye position tracking unit (second detection means)
33 ... Open / close eye determination unit (open / close eye determination means)
33a ... Opening / closing eye learning unit (threshold setting means)
33b ... Opening / closing eye determining unit (opening / closing eye determining means)
34. Arousal level estimation unit (wake level determination means)
35 ... Opening / closing determination result processing section (delay output means)
36 ... Processing state output unit 37 ... Arousal level signal output unit

Claims (10)

Eye detection means for inputting image data including the face of the person to be detected and detecting the eye of the person to be detected from this image;
Open / close eye determination means for determining open / close eyes for the eyes detected by the eye detection means;
Wakefulness estimation means for estimating the wakefulness level of the detected person based on the open / closed eye determination result determined by the open / closed eye determination means,
The eye detection means determines whether the detected eye of the detected person is due to erroneous detection,
The wakefulness estimation means estimates the wakefulness of the person to be detected based on the erroneous detection determination result by the eye detection means in addition to the open / closed eye determination result determined by the open / close eye determination means. Awakening level estimation device. The eye detection means includes
First detection means for detecting the eyes of the detected person from the entire image including the face of the detected person;
Second detection means for detecting the eye of the detected person from a partial region of the image including the face of the detected person,
The second detection means sets a partial region including either the eye position detected by the first detection means or the eye position previously detected by the second detection means. The arousal level estimation apparatus according to claim 1. The opening / closing eye determination means includes
Threshold setting means for setting a threshold for determining the open / closed eye by measuring the vertical width of the eye detected by the eye detecting means;
When the eye width detected by the eye detection means is greater than or equal to the threshold value set by the threshold setting means, it is determined that the eye is open, and when the eye width detected by the eye detection means is less than the threshold value Open / closed eye determining means for determining closed eyes;
The awakening level estimation apparatus according to claim 1, wherein the awakening level estimation apparatus is provided.   The eye detection means stores a condition for determining that the detected eye is due to erroneous detection, and the number of images to be determined as erroneous detection when the condition is satisfied. The information that the eye open / closed eye determination result based on the number of images to be determined is incorrect is output to the arousal level estimation unit. Wakefulness estimation device. A delay output unit that inputs the open / closed eye determination result determined by the open / closed eye determination unit, delays it for a predetermined time, and outputs it to the arousal level estimation unit;
The eye detection unit determines whether the detected eye of the detected person is due to a false detection during a predetermined time delay by the delay output unit. The arousal level estimation apparatus according to item 1.   The delay output means, when a time-series opening / closing eye determination result inputted during a delay of a predetermined time corresponds to a pattern stored in advance that is an inappropriate opening / closing of the eye of the detected person, 6. The arousal level estimation apparatus according to claim 5, wherein the eye opening level is appropriately opened and closed and the pattern is stored in advance. The arousal level estimation means estimates the arousal level for each arousal level estimation method based on a plurality of arousal level estimation methods,
The wakefulness estimation apparatus according to claim 5, wherein the delay output unit changes a predetermined delay time for each wakefulness estimation method. If the eye is not detected, the eye detection means outputs information to that effect to the arousal level estimation means,
The arousal level estimation means sets the ratio of the time when the eyes were closed out of the time excluding the time when the eyes could not be detected and the time when the eyes were erroneously detected from the specified time as the closed eye ratio. Estimate the degree of arousal from the time ratio, or out of the time excluding the time when the eye could not be detected from the specified time, the ratio of the time when the eyes were closed was taken as the closed eye ratio, and the degree of arousal was calculated from this closed eye ratio The wakefulness estimation device according to claim 7, wherein The wakefulness level estimation means includes a first wakefulness level estimation method for estimating a decrease in wakefulness level based on a closed eye time ratio, a second wakefulness level estimation method for estimating a decrease in wakefulness level based on a transition of the closed eye time ratio, and Have
The wakefulness estimation device according to claim 8, wherein the delay output means makes the predetermined time of the second wakefulness estimation method shorter than the predetermined time of the first wakefulness estimation method. An eye detection step of inputting image data including the face of the detected person and detecting the detected person's eyes from this image;
Open / close eye determination step for determining open / close eyes for the eyes detected in the eye detection step;
A false detection determination step of determining whether the eye detected in the eye detection step is due to a false detection;
An arousal level estimation step for estimating the arousal level of the detected person based on the open / closed eye determination result determined in the open / closed eye determination step and the erroneous detection determination result in the eye detection step;
A wakefulness estimation method characterized by comprising: