JP2008246013A - Sleepiness sensor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a sleepiness sensor capable of easily and quickly sensing the sleepiness of a target person to be observed with high robustness, especially useful when applied to a vehicle. <P>SOLUTION: The sleepiness sensor comprises a photographing means 2 for photographing the face of the target person A, a pupil area detecting means 4 for detecting the area of the pupil from the captured face image, and a sleepiness sensing means 5 for sensing the sleepiness of the target person by comparing the detected pupil area with a reference value. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an apparatus that detects an area of a pupil of a subject based on a photographed image and detects sleepiness of the subject based on the detected area.

The driver's drowsy driving in the transportation industry using vehicles, particularly large trucks, is likely to cause many victims, and measures for preventing drowsy driving are an important issue in the near future. Therefore, it has been studied to apply a means for detecting the drowsiness of the subject person as a means for preventing the driver of the vehicle from falling asleep.
For sleepiness detection, it is generally considered to be reliable to measure brain waves, heartbeats, and pulse, and detect changes from them. It is difficult to put these measuring devices on the driver and it is difficult to put it into practical use.
In order to detect drowsiness in a non-contact state with the driver, development focusing on facial expressions and eye states in the driver's face image has also been made.

For example, Patent Document 1 discloses that an apparatus for detecting a driver's consciousness level using an image adjusts infrared light incident on the driver's eyes according to the size of the driver's pupil.
Patent Document 2 discloses that the state of the driver is detected by detecting the opening degree of the eyes.
Patent Document 3 discloses that the vehicle is controlled on the assumption that the operator is in a tension state when the change speed of the pupil diameter is larger than a predetermined value.

  Furthermore, a technique called PERCLOS is also promising to detect a decrease in the driver's consciousness level. Here, PERCLOS (Percent of the time eyelids are closed) is a means of measuring the driver's fatigue, which is approved by the US Road Traffic Safety Agency, and represents the percentage of time the eyes are closed. (D. F. Dinges, M. Mallis, G. Maislin, and J. W. Powell.Evaluationof techniques for ocular measurement as an index of fatigue and the basis foralertness management.Department of Transportation Highway Safety publication808 762, April, 1998.)

There is also a technique that focuses on the opening and closing speed of the eye, ACEM (the average eye closure / opening speed, ie, the amount of timeneeded to fully close the eyes and to fully open the eyes). For example, QiangJi and Xiaojie Yang, Real Time Visual Cues Extraction for Monitoring Driver Vigilance "Lecture Notes in Computer Science: Computer Vision Systems, vol. 2095, (Ed.) B. Schiele, G. Sagerer (Eds.), Springer, pp. 107, June, 2001.
Describes the case of measuring PERCLOS and ACEM.
JP-A-7-32907 JP 2000-142164 A JP 2000-351337 A

As described above, in order to detect the state of the driver, detection of the eyelid state and the pupil diameter is also being studied. By the way, the pupil diameter here is assumed to be the pupil diameter in the lateral direction. This is because, in Patent Document 3, “the maximum value of the number of pixels in the horizontal direction of the image data corresponding to the detected pupil position is detected as the pupil diameter L. Here, attention is paid to the horizontal direction in the vertical direction. This is because the correct pupil diameter cannot be detected due to the influence of “blink” by the driver. This is because the pupil is hidden by the upper and lower eyelids, so that it is difficult to detect the diameter in the vertical direction compared to the case in the horizontal direction.
However, even in the case of the pupil diameter in the horizontal direction, it naturally occurs that it is hidden by the eyelids and cannot be accurately detected even if it is not as long as the vertical direction.

It is known that when the degree of arousal is high, the pupil is large and the size is stable, but when the degree of arousal decreases, the average pupil diameter becomes small and vibrates. This is a phenomenon (hippus) in which the pupil area fluctuates when the degree of relaxation increases or the degree of arousal decreases. Hippus is a phenomenon in which the pupil area vibrates at a frequency of about 0.2 Hz (once every 5 seconds), and has long been considered to be related to sleepiness. (Mathematical procedures in data recording and processing of pupillary fatigue waves, Vision Research, Volume 38, Issue 19, October 1998, Pages 2889-2896, Holger Ludtke, Barbara Wilhelm, Martin Adler, Frank Schaeffel and Helmut Wilhelm).
In this paper, experimentally, both eyes are covered with a cover that blocks the visible light but transmits the near-infrared light, irradiates the near-infrared light from the outside, and is also photographed from the outside by the camera. Was taken. However, it may be possible to detect the lateral length of the pupil in an environment where the eye is projected from the side of the eye, but as shown in this paper, blinking (blinking) In this way, it is difficult and accurate to measure the horizontal diameter of the pupil accurately unless the camera and the light source are placed next to the eyes.
In addition, the horizontal diameter of the pupil may be divided by the eyelashes, and there is a problem that it is difficult to detect as a stable value.

There is also a method of detecting the eyelid by image processing, detecting the opening and closing of the eyes from the movement, and detecting blinks to detect the state of the subject. However, it is difficult to detect eyelids when the ambient light environment changes greatly as in a car.
Furthermore, since the line of sight tends to turn downward when sleepy, a method using line-of-sight measurement has also been proposed. In this case as well, it is common to detect the head of the eye, the corner of the eye, and the like by the template matching method and use a separation degree filter, etc., but both are vulnerable to changes in the surrounding light environment. Recently, there is a method of recording a three-dimensional shape of detected features by template matching using a stereo camera, and trying to detect the eyes and the corners of the eyes based on the recorded three-dimensional shape. However, even this method is vulnerable to changes in the surrounding light environment. Therefore, it is difficult at present to detect changes in facial expressions.

On the other hand, there are many truck accidents from midnight to morning. Therefore, in practical use of the drowsiness detection device, if drowsiness can be detected in a dark environment, it is sufficiently useful even if the daylight is not always detected.
By the way, in the method for detecting drowsiness from the face image, near infrared light must be irradiated to the face if the surroundings are dark. However, due to the light source, reflected light (glasses reflection) generated on the surface of the spectacle lens or the frame is generated, and this has a fatal effect on the method of detecting eyelids, the black eye detection and the template matching method. Also, depending on the shape of the glasses, the eyelids may be easily hidden by the glasses frame, which is not practical.

Under the circumstances as described above, the present inventor can solve the above problem for the following reason by detecting the area of the pupil from the face image and detecting the sleepiness of the subject. Thus, the present invention has been achieved.
That is, the first reason is to detect the pupil from the face image, define the pupil part, and detect the area as a factor of sleepiness determination, factors in other places, especially the pupil used in the conventional method It is easier and more robust than detecting the diameter and eyelid state.
In particular, at night, when the practical use of sleepiness detection is most desired in driving a vehicle, the pupil is wide open when there is no sleepiness, and detection of the pupil portion and its area is easy. Further, it is further easier to use a captured image of a bright pupil by irradiating near infrared rays from the vicinity of the camera or the like. In addition, for detecting the pupil and its area, a method of taking the difference between the bright pupil and the dark pupil can be adopted. In this case, the detection can be made easier and more accurate.

Further, as a next reason, in the conventional technique for detecting the pupil diameter, no countermeasure is taken into account for the fact that the pupil is hidden by the eyelid. However, in the present invention, this problem can also be solved by using the area of the pupil for detecting drowsiness.
In other words, when it becomes sleepy, the eyelid slowly closes, but the pupil is hidden by the eyelid, so that the area reflected in the camera is reduced and the pupil is actually reduced by sleepiness almost simultaneously, We have found that if the purpose is to detect drowsiness, it is not necessary to distinguish them.
Thus, the area of the pupil detected from the front of the face by the camera changes not only due to the actual pupil area change, but also when the upper and lower eyelids cover the pupil. In that case, the shape of the pupil changes, but in the present invention, the sleepiness of the subject can be detected easily, accurately and quickly by simply measuring the pupil area which is a factor regardless of the shape of the pupil reflected in the camera. .

  The present invention has been made as described above, and an object of the present invention is to provide a drowsiness detection device that can detect sleepiness of an observation subject easily, quickly, and with high robustness, and is particularly applicable to a vehicle. To do.

In order to achieve such an object, the drowsiness detection device of the present invention is detected by an imaging unit that captures the face of the subject, a pupil area detection unit that detects an area of the pupil from the captured face image, and It comprises drowsiness detection means for detecting drowsiness of the subject by comparing the pupil area with a reference value.
According to the present invention, the location and area of the pupil can be detected from the face image of the subject imaged by the imaging means. Then, by comparing the pupil area with the reference value, the subject's pupil becomes smaller due to drowsiness and the pupil is blocked by the eyelids. Regardless of the pupil area calculated easily and accurately, the sleepiness of the subject can be detected easily, quickly, and with high robustness.

  Further, in the present invention, when the drowsiness detection unit includes a calculation unit that removes a signal due to blinking and generates a pupil area signal, the drowsiness detection is performed without being affected by a short blink unrelated to sleepiness. can do.

  Further, in the present invention, the drowsiness detection means sets a threshold smaller than the average area of the pupil, and means for determining drowsiness when the ratio of the pupil area below the threshold exceeds a preset ratio within a certain period. When equipped, it is possible to make a stable sleepiness determination without disturbance by setting according to the situation.

Further, in the present invention, the pupil area detecting means is a computing means for converting the number of pixels obtained by binarizing a difference image obtained by image difference between a bright pupil image and a dark pupil image to obtain a pupil area. When equipped, the pupil area can be detected without hindrance even when the surroundings are bright or when there is reflection of eyeglasses or eyelashes near the pupil of the subject. Note that the conversion of the number of pixels into the pupil area also includes setting the number of pixels itself as the value of the pupil area.
Moreover, in this invention, when the drowsiness detection apparatus is for vehicle-mounted use, it can detect the drowsiness of the vehicle driver who is an observation object quickly.

  Further, in the present invention, when a drowsiness of the subject person is detected by the drowsiness detection means, when a means for emitting a warning sound is provided, a warning sound is emitted to the subject person when drowsiness is detected. It is possible to prevent the observation subject from having an accident due to sleepiness.

  Further, in the present invention, when a means for correcting the drowsiness determination is provided according to the intensity of ambient light in photographing the face of the subject, the pupil area of the subject changes due to a change in ambient light of the photographing, thereby It is possible to prevent an error in determining sleepiness.

  INDUSTRIAL APPLICABILITY The present invention can provide a drowsiness detection device that can detect sleepiness of an observation subject easily, quickly, and with high robustness, and is particularly applicable to a vehicle.

In the following, preferred embodiments of the apparatus according to the present invention will be described with reference to the drawings.
FIG. 1 is a conceptual diagram of the drowsiness detection device 1 of the present embodiment. The drowsiness detection device 1 includes an imaging unit 2, a control unit 3, a pupil area detection unit 4, and a drowsiness detection unit 5.
The photographing means 2 includes a camera 21 and light sources 22 and 23 and photographs a face image including the pupil P of the photographer A. Since the camera 21 is installed in the vehicle, it is preferable to shoot from the subject's face about 70 to 80 cm and shoot from the front lower side and obliquely to shoot the pupil.
As the camera 21, for example, an NTSC system camera or the like is employed. Shooting by the camera 21 and lighting of the light source are controlled by a control device 3 such as a personal computer, for example.
The captured face image is analyzed by the pupil area detecting means 4 to calculate the area of the pupil.
The calculated pupil area value in the pupil area detection means 4 is calculated in the drowsiness detection means 5 to determine whether the subject is drowsy or not, and a warning sound is emitted as necessary. Here, the control means 3, the pupil area detection means 4, and the drowsiness detection means 5 may be separate devices or the same personal computer.
Further, if the difference image between the bright pupil image and the dark pupil image is not used for detecting the pupil area, it is not necessary to use two light sources for the photographing means 2 as shown in the figure. Further, even when a difference image is used, it is possible to use one light source.
In addition, a band pass filter that allows only light emitted from the light source to pass is attached to the camera. As a result, extra visible light can be cut to some extent for pupil detection.

  FIG. 2 shows an example of processing steps in the pupil area detecting means 4 when detecting the pupil area using the difference image between the bright pupil image and the dark pupil image. Here, in step 41, a bright pupil image and a dark pupil image are acquired, and the difference between them is calculated to define a difference image of the pupil part (step 42), and the number of pixels of the obtained difference image is counted. To calculate the pupil area (step 43). The pupil area may be an average value when both pupils are detected, or may be the pupil area when only one is detected.

Here, the bright pupil and the dark pupil will be described.
In FIG. 1, when a light source 22 such as a near infrared ray is provided near the opening of the camera 21 and the face of the subject A is irradiated with light along the optical axis of the camera, the light reaches the retina from the pupil P. And returns to the opening of the camera 21 through the crystalline lens and cornea. In this image, the pupil is photographed brightly, and the image is called a bright pupil image. On the other hand, when the subject's face is irradiated with light from a light source 23 such as near infrared rays separated from the opening of the camera 21, the light reflected from the retina hardly enters the opening of the camera 21, so the pupil is dark. The photographed image is called a dark pupil image. The pupil also has eyelashes around it, and some spectacles in the vicinity of the pupil may cause reflection in a spectacle wearer. For this reason, it may be difficult to detect the pupil and determine its area, especially when it is bright during the day and the pupil is small.
However, if the difference calculation is performed by subtracting the image obtained by photographing the dark pupil from the image obtained by photographing the bright pupil, the surrounding portions other than the pupil portion have almost the same brightness in both images. Only the pupil part with a difference in thickness becomes bright and embossed. As a result, the pupil can be easily detected and the accurate area can be counted.

For example, an odd field and an even field that constitute one frame every 1/30 second in NTSC camera shooting are acquired as a bright pupil image and a dark pupil image, respectively, and a difference image between them is calculated, and a P tile method is further obtained. Thus, by determining the threshold value of the luminance and binarizing the pixels, the number of pixels in the portion where the pupil exists can be obtained, and this converted value can be used as the pupil area.
In addition, the position of the pupil changes as the subject's face moves, but a small window including the pupil is set in advance, and the position change of the window is predicted by a prediction model such as a Kalman filter, It is also possible to make the calculation more efficient by providing the next window in the predicted range and making a difference only within the window.

Next, FIG. 3 shows an example of processing steps in the drowsiness detection means 5.
As described above, for example, when the pupil area based on the difference image between the bright pupil image and the dark pupil image is detected every 1/30 seconds, even in a normal state where the subject does not feel sleepy, In addition, the detected pupil area may be zero or a small value close to it due to a sudden movement of the face, or may be a sharply protruding value due to the reaction. If such a pupil area where the signal suddenly fluctuates is used as drowsiness determination data, accurate determination cannot be performed. For this purpose, in step 51, calculation is performed to remove the data signal whose pupil area has fluctuated due to such a short blink.
As a means for that purpose, various methods such as removing a signal that has changed suddenly and rapidly by a logic circuit or a filter, or smoothing the signal can be used.
Then, in the following processing steps, a calculation for sleepiness determination is performed based on the pupil area from which a short-time fluctuation signal based on a short blink or some measurement error is removed.

In step 52, the average area of the pupil is calculated. Since this value also serves as one reference value in subsequent sleepiness determination, it is desirable to obtain the average of pupil detection at the initial stage when the subject does not have sleepiness.
However, on the other hand, for example, if the reference value is determined in a bright garage just before leaving on a business truck at night, the above reference value is calculated to be smaller than the optimum value, resulting in a warning. It becomes difficult to do. Therefore, for example, it is also effective to always use the average value of the previous three minutes for about 10 minutes as the reference value. Conversely, if the reference value is determined in a dark place and goes out to a bright place, the pupil is always smaller. In this case, a warning may continue to be issued. However, in this case, if a warning is issued when the condition for raising a warning is not satisfied once and is satisfied again, there is no problem because only a warning is required once.
Next, a threshold value to be compared with the pupil area is set (step 53). For example, 55% of the average value of the pupil area is set as the threshold value.
Then, the detected continuous pupil area data is compared with the threshold value one after another (step 54), and the detected pupil area is determined for a predetermined fixed period, for example, 5 seconds (150 frames) or 10 seconds (300 frames). When the ratio below the threshold exceeds 50%, for example, it is determined that the subject falls asleep (step 55).
When sleepiness is determined, a warning sound is emitted to the subject to wake up sleepiness. This warning sound may be a sound such as a buzzer or a voice, and may notify not only the target person but also the business office at the same time.
Here, the sleepiness is determined by setting a threshold smaller than the average area of the pupil, and the ratio that the pupil area falls below the threshold within a certain period exceeds the preset ratio. However, it is not limited to such a method, simply set a certain threshold, and when the detected pupil area falls below this, sleepiness is determined, or when the threshold falls below this threshold a predetermined number of times within a certain period It is also possible to determine sleepiness.

Next, a drowsiness determination correcting means based on the intensity of ambient light in photographing will be described.
The pupil area becomes small even if the surrounding brightness goes from a dark place to a bright place. In order to prevent erroneous sleepiness detection due to this, the ambient light intensity hitting the face may be measured. Ambient light such as sunlight (including light of the same wavelength as the light source in many cases) also affects the brightness of the face image in the camera of the apparatus according to the present invention.
If the current amount of the light source is kept constant, the light pupil image or dark pupil image maintains the light and darkness of the image that originates from the light source, while the light and darkness that originates from the ambient light is added. Therefore, the intensity of the ambient light can be determined by recording the brightness of the image in the dark and measuring how bright the image is. When a condition for issuing a warning from a reduction in pupil area is satisfied, it is further determined whether to issue a warning in view of ambient light.
Even when the current of the light source is changed, the relationship between the current value and the brightness of the image is measured in advance and used for determination.
As another method of determining the intensity of ambient light, when the condition for issuing a warning from the reduction in pupil area is satisfied, it is only necessary for one frame, so the light source is turned off to obtain an unilluminated image, The intensity of the ambient light is determined from the brightness of the image, and it is determined whether or not to issue a warning.
Further, as a method for accurately measuring the visible light that affects the pupil area, there is a method in which the visible light intensity corresponding to the face is separately measured by a camera or an optical sensor that captures the face.
According to the method as described above, it is possible to prevent a warning from being issued erroneously, particularly when the surroundings become brighter rapidly and the pupil becomes smaller.

Next, examples actually experimented on the present embodiment will be described with reference to FIGS.
Since it is difficult to actually observe the driver's dozing state during driving, an experiment simulating an in-vehicle experiment was conducted in a dark room as shown in FIG. A CCD camera 2 was placed facing the face of the subject A, and a video of driving on the highway was projected using a projector.
In the experiment, since the night was assumed so that the pupil became large, the illumination was turned off, and the room temperature was set to 25 ° C. or more so that the sleep was easy. The subject was supposed to move the handle as if he was actually driving for 15 minutes.
As described for the pupil area detecting means 4, the bright pupil image and the dark pupil image are photographed to obtain a difference image of the pupil, as shown in the photograph of FIG. The number of pixels of the difference image was defined as the pupil area. The pupil area was defined as an average value of the pupil areas (number of pixels) of both pupils when both pupils were detected, and the pupil area when only one pupil was detected.

"Extraction of pupil area fluctuation waveform"
When the eyes are closed by blinking, the pupil disappears, so that the detected pupil area becomes zero.
An example of changes in pupil area is shown in FIG. Here, the experiment started at 0 seconds and totaled about 15 minutes. At the start of the experiment, the pupil area is large, but it gradually begins to drop around 3 minutes. An alarm sounded at the time indicated by the arrow. Then, the pupil suddenly increased. Here, the warning sounded when it was confirmed that the subject completely closed his eyes. That is, it means that the subject slept immediately before the warning. Note that the warning given here does not necessarily coincide with the generation of the warning sound in the present invention. In the present invention, as will be described later, it is possible to emit a warning sound more quickly by analyzing the pupil area fluctuation waveform, but here, in order to observe various conditions of the subject, Warning at such a stage.

FIG. 7 shows an enlarged view of 0 second from 120 seconds immediately after the start of the experiment in FIG. In this figure, it can be seen that the pupil area gradually changes in the range of 300 pixels to 500 pixels, and that the actual area of the pupil itself sometimes becomes smaller and returns to the original area. Further, in the graph, the portion where the blinking occurs is the portion where the pupil area jumps down and the pupil area is almost zero.
Representative face images between 0 and 120 seconds are shown in FIGS. These are images at about 5 seconds and about 47 seconds, respectively. The pupil is slightly smaller in the latter, but both eyes are wide open and it can be seen that the pupil is large. In either face image, it can be seen that the subject's facial expression is not particularly drowsy.
FIG. 10 shows an enlarged view of the pupil area change in FIG. 6 from 360 seconds to 540 seconds. Here, it can be seen that the pupil area changes greatly. 11 to 13 show face images at about 395 seconds, about 418 seconds, and about 482 seconds, respectively. When FIG. 11 and FIG. 12 are compared, the pupil area also changes, but the degree of opening and closing of the eyelids also changes. In FIG. 13, the pupil is almost invisible, but the result of FIG. 10 shows that the pupil can be detected and the area is not zero. Here, it can be seen from FIG. 11 to FIG. 13 that the subject clearly feels sleepy. For this reason, at least it will be necessary to issue a warning at this point.

FIG. 14 is an enlarged view of the range from 630 seconds to 780 seconds in FIG. Here, a warning sound is emitted around 660 seconds and 750 seconds. 15 to 18 show face images from 748 seconds to 768 seconds. At 748 seconds, the eyes were almost completely closed. In FIG. 14, the pupil area was zero, but immediately after the warning sounded, the eyes opened at 752 seconds (FIG. 16) and 759 seconds (FIG. 17). It can be seen that the pupil is enlarged. This tendency is clearly seen in FIG. Further, at 768 seconds (FIG. 18), it can be read from the facial expression that the consciousness has returned.
As described above, it can be seen from the facial expression that the pupil area corresponds to drowsiness.

"Drowsiness detection"
The pupil area waveform contains blinks and some errors. Further, as can be seen from FIG. 10, even if the degree of arousal is high, there is a change in the pupil area due to light reflection or the like. Therefore, it is necessary to remove the influence and capture that the pupil area has decreased.
Signal processing for that purpose will be described next.
In FIG. 7, the part protruding upward indicated by a large number of arrows is a part in which the pupil area is overestimated. In order to remove this, (1) every five consecutive frames (150 ms), Replace the second value from the highest value to zero. Thereafter, (2) the maximum pupil area is obtained from 30 consecutive frames (1 second), and the pupil area of the last frame is replaced. This process is executed while shifting one frame at a time. By this processing, when zero continues for a short time (for example, 6 frames) like a blink, even if the pupil area becomes zero by the processing of (1), it is corrected to become zero. Here, by selecting 30 frames in (2), when the subject closes their eyes for 30 frames (1 second) or more, the pupil area after processing becomes zero for the first time. In other words, the maximum delay time is 35 frames (1.15 seconds).

  By performing the above processing, for example, the original pupil area waveform in FIG. 19A (same as FIG. 6) is shaped as shown in FIG. FIG. 20 shows the same experimental results for another subject. FIG. 20A shows an original waveform of the pupil area, and FIG. 20B shows a pupil area waveform obtained by performing the above processing. Again, when a warning sound is emitted, the pupil area has almost returned to the initial state.

FIGS. 21A and 21B show the results of sleepiness detection processing using the pupil area waveform after the processing of FIG. 19 (2 to 10 minutes from the start of the experiment).
The drowsiness detection method uses 55% of the average value of the pupil area after the first 10 seconds (300 frames) after the start of the experiment as a threshold. When the ratio below the threshold value exceeds 50% for 5 seconds (150 frames) or 10 seconds (300 frames), the square wave in FIGS. .
At this point, the subject can be determined to be drowsy.

"Comparison with conventional methods"
Here, in order to show the effectiveness of the present invention, a conventional representative method is applied to the pupil detection result in this experiment and compared with the present embodiment of the present invention.
In order to contrast with the above-mentioned method by PERCLOS (Percent of the time eyelids are closed), the ratio of the eyes closed in 300 consecutive frames (10 seconds) was determined for each frame. Here, the number 300 was selected as the number of frames that would make it easy to determine drowsiness.
Further, in order to contrast with the technique based on the blink frequency, the number of transitions from the pupil detection state to the non-detection state in 300 consecutive frames (10 seconds) was counted as the number of blinks.

As can be seen from FIG. 19, in this example of (b), the pupil area gradually decreases until the first warning is issued. In PERCLOS in (c), it can be seen that it has stood up before the first warning. However, although it is difficult to understand from FIG. 19, the portion where PERCLOS is rising shows a peak of about 50%, and the eyes are closed for 5 seconds per 10 seconds. That is, it means that it may be said that it is already sleeping.
Moreover, the peak is not clear in the number of blinks of (d).
In the example of FIG. 20, PERCLOS is about 50% between 12.5 minutes and 13 minutes, and it can be seen that the eyes closed for a long time and fell asleep. From the perspective of predicting dozing, we would like to detect drowsiness by increasing PERCLOS or blinking frequency between 11.5 minutes and 12 minutes, but surrounded by a dashed ellipse in FIG. 20 rather than the value at that time It shows a much higher value. The time zone around the ellipse was just after the start of the experiment, and even if I looked at the facial expression, I was still not feeling sleepy, but the frequency of blinking with PERCLOS was high.
From this, it is presumed that PERCLOS and blink frequency are not very effective for sleepiness detection. On the other hand, it can be seen that the value of the pupil area of the present embodiment (b) gradually decreased from around 9.5 minutes. If this is detected, it will be understood that the pupil area is more effective than PERCLOS and blink frequency for detecting drowsiness.

  In FIG. 21, if a warning sound is generated when shifting from a low level to a high level, the case of FIG. 21A of this embodiment as described above is about 6.6 minutes, and FIG. In the case of (b), it means that a warning is issued around 8 minutes. On the other hand, in PERCLOS, a warning cannot be issued unless the time is 9.4 minutes (it is difficult to adjust the threshold because blinking may occur frequently even if you are not sleepy as shown in FIG. 20). Thus, in the present embodiment, a warning can be issued one minute or more earlier than PERCLOS.

Next, the pupil diameter detection of the conventional method is compared with the embodiment of the present invention.
FIG. 22 shows that the pupil size and the pupil shape change depending on several pupil diameters and eyelid closure. Here, the black portion is the pupil area that is revealed by image analysis, and the width of the rectangle passing through the upper and lower ends and the left and right ends of the detected pupil portion is called the horizontal pupil diameter, and the height is called the vertical pupil diameter. To do. Here, what is conventionally called the pupil diameter in other patents corresponds to the lateral pupil diameter.
FIG. 23 and FIG. 24 show the horizontal pupil diameter, the vertical pupil diameter, and the pupil area from the top up to about 11.5 minutes from the start of the experiment, corresponding to the data shown in FIG. 19 and FIG. 20, respectively. . From both figures, it can be seen that the horizontal pupil diameter and the pupil area change greatly while the horizontal pupil diameter does not change much. Further, it can be seen that the pupil area changes more greatly than the vertical pupil diameter. (Here, the original form is shown as it is.)
Thus, the detection of the change by the pupil area of the present invention can be said to be easy and accurate.

It is a conceptual diagram which shows the drowsiness detection apparatus of embodiment of this invention. It is a block diagram about the pupil area detection means of the embodiment of the present invention. It is a block diagram about the drowsiness detection means of the embodiment of the present invention. It is a photograph which shows the experimental apparatus in the Example of embodiment of this invention. It is a photograph which shows the result in the Example of embodiment of this invention. It is a graph which shows the result in the Example of embodiment of this invention. It is a graph which shows the result in the Example of embodiment of this invention. It is a photograph which shows the result in the Example of embodiment of this invention. It is a photograph which shows the result in the Example of embodiment of this invention. It is a graph which shows the result in the Example of embodiment of this invention. It is a photograph which shows the result in the Example of embodiment of this invention. It is a photograph which shows the result in the Example of embodiment of this invention. It is a photograph which shows the result in the Example of embodiment of this invention. It is a graph which shows the result in the Example of embodiment of this invention. It is a photograph which shows the result in the Example of embodiment of this invention. It is a photograph which shows the result in the Example of embodiment of this invention. It is a photograph which shows the result in the Example of embodiment of this invention. It is a photograph which shows the result in the Example of embodiment of this invention. It is a graph which shows the result in the Example of embodiment of this invention. It is a graph which shows the result in the Example of embodiment of this invention. It is a graph which shows the result in the Example of embodiment of this invention. It is explanatory drawing of the pupil diameter detection for contrast with this invention. It is a graph which contrasts a pupil diameter and pupil area detection. It is a graph which contrasts a pupil diameter and pupil area detection.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Sleepiness detection apparatus, 2 ... Imaging | photography means, 21 ... Camera, 22, 23 ... Light source, 3 ... Control means, 4 ... Pupil area detection means, 5 ... Sleepiness detection means

Claims (7)

Photographing means for photographing the face of the subject;
Pupil area detection means for detecting the area of the pupil from the photographed face image;
A drowsiness detection apparatus comprising: drowsiness detection means for detecting drowsiness of a subject by comparing the detected pupil area with a reference value.   The sleepiness detection apparatus according to claim 1, further comprising a calculation means that removes a signal caused by blinking and generates a pupil area signal.   The drowsiness detection means comprises means for setting a threshold smaller than the average area of the pupil and determining drowsiness when the ratio of the pupil area below the threshold exceeds a preset ratio within a certain period. The drowsiness detection device according to claim 1 or 2.   The pupil area detection means includes a calculation means for converting the number of pixels obtained by binarizing a difference image obtained by image difference between a bright pupil image and a dark pupil image into a pupil area. The drowsiness detection device according to claim 1. The drowsiness detection device according to claim 1, wherein the drowsiness detection device is for vehicle use.   The sleepiness detection apparatus according to any one of claims 1 to 5, further comprising means for generating a warning sound when the sleepiness of the subject is detected by the sleepiness detection means.   The drowsiness detection device according to claim 1, further comprising means for correcting drowsiness determination according to the intensity of ambient light in photographing the face of the subject.