JP2020095499A - Eye opening and closing detection apparatus, and driver monitoring apparatus - Google Patents

Abstract

To prevent an alarm signal from being errorneously output upon narrowed eye resulting from direct light.SOLUTION: A driver monitor 100 (a driver monitoring apparatus) according to the present invention has an imaging unit 11 for imaging a face of a driver in a vehicle, an image processing unit 2 for subjecting a predetermined processing to a face image captured by the imaging unit 11 to extract eyes of the driver from the face image, an opened and closed degree calculating unit 31 for calculating an opened and closed degree indicating opened and closed states of eyes based on the image of eyes extracted by the image processing unit 2, a weight calculating unit 33 for calculating weight responding to luminance information at a predetermined region of the face image, an opened and closed state determining unit 34 for subjecting the weighting to the opened and closed states calculated by the opened and closed degree calculating unit 31 by using the weight and for determining whether the eyes are opened or closed, based on the opened and closed degree added with the weight, and an alarm output unit 35 for outputting an alarm signal based on the discrimination result of the opened and closed state determining unit 34.SELECTED DRAWING: Figure 1

Description

The present invention relates to a device that detects an open/closed state of a driver's eyes in order to prevent, for example, drowsy driving.

A driver monitor mounted on a vehicle analyzes an image of a driver's face captured by a camera, and monitors whether or not a driver is dozing, based on the eye open/closed state (eye open state or eye closed state). Specifically, the degree of opening/closing, which is a parameter indicating the open/closed state of the eye, is calculated from the image analysis result (for example, Patent Documents 1 to 3). Then, the open/closed degree is compared with a preset threshold value, and the open/closed state of the eye is determined based on the comparison result. If the result of determination is that the eye-closed state continues for a certain period of time or more, it is determined that the driver is dozing and a warning is output by voice or the like.

By the way, the open/closed state of the eye includes not only the state in which the eye is completely open and the state in which the eye is completely closed, but also an intermediate state, that is, a state in which the eye is narrowed (fine eye state). .. In order to prevent drowsiness driving, the driver monitor should output an alarm signal if the driver's eyes are drowsy, that is, if he is in a drowsiness symptom state, if the eyesight state continues for a certain time or longer. Is being done. However, when direct sunlight enters through the windshield of the vehicle, the driver often feels glare and narrows his eyes. Therefore, if the fine-eyed state continues for a certain period of time or longer, the driver monitor erroneously outputs an alarm signal even if the driver is not drowsy.

Patent Document 4 describes that when the fine eye state is based on a smile, it is determined that the eye state is open. Patent Document 5 describes that the amount of light in a region near the eyes is measured, and based on the amount of light and the degree of opening/closing of the eyes, it is determined whether or not the direct light hits the eyes of the driver. .. In Patent Document 6, a visual field image in a predetermined range is extracted from an image in front of the vehicle captured by a front monitoring camera, and a state in which strong light is emitted to the driver's eyes is detected based on the brightness of the visual field image. It is described to do. However, these documents do not disclose any specific means for preventing a false alarm with respect to a fine eye state caused by direct light.

JP, 2010-134490, A Japanese Patent No. 3143819 JP-A-2000-123188 JP, 2016-115120, A JP, 2006-251926, A JP, 2007-276766, A

An object of the present invention is to detect a fine-eyed state in the presence of direct light by distinguishing it from a fine-eyed state due to drowsiness, and for a fine-eyed state caused by direct light, an alarm signal is erroneously output. It is to prevent it.

An eye open/closed detection device according to the present invention includes an image capturing unit that captures a face of a subject, and an image processing unit that performs a predetermined process on a face image captured by the image capturing unit and extracts the eye of the subject from the face image. And, based on the image of the eye extracted by the image processing unit, an opening/closing degree calculation unit that calculates an opening/closing degree that represents the opening/closing state of the eye, and a weight that calculates a weight according to the luminance information of a predetermined area in the face image. A calculator and an open/closed state determination unit that determines the open/closed state of the eye based on the weighted open/closed degree by weighting the open/closed degree calculated by the open/closed degree calculation unit with the weight. There is.

Further, the occupant monitoring device according to the present invention is an image pickup unit for picking up an image of the occupant's face of the vehicle, an image for performing a predetermined process on the face image picked up by the image pickup unit, and extracting the occupant's eyes from the face image. A processing unit, an open/closed degree calculation unit that calculates an open/closed degree indicating the open/closed state of the eye based on the image of the eye extracted by the image processing unit, and a weight according to the brightness information of a predetermined area in the face image A weight calculation unit, and an open/closed state determination unit that determines the open/closed state of the eye based on the weighted open/closed degree, with respect to the open/closed degree calculated by the open/closed degree calculation unit. An alarm output unit that outputs an alarm signal based on the determination result of the open/closed state determination unit is provided.

In the present invention, the eye open/closed degree calculated by the open/closed degree calculation unit is weighted by a weight according to the luminance information of a predetermined area in the face image, and the eye open/closed is performed based on the weighted open/close degree. Determine the state. Therefore, the open/closed state of the eye is detected based on the open/closed degree corrected according to the state of the direct light, so that the fine eye state due to the glare of the direct light is detected separately from the fine eye state due to the drowsiness. It becomes possible. As a result, it is possible to prevent an alarm from being output by misjudging this as a sign of drowsiness when the eye is narrowed under direct light.

The eye open/closed detection device according to the present invention further includes a histogram calculation unit that calculates a histogram showing the distribution of the luminance of a predetermined area in the face image, and a weight table in which weights are set corresponding to the luminance classes. Good. Further, the weight calculation unit may refer to the weight table and select the weight corresponding to the class having the highest frequency in the histogram calculated by the histogram calculation unit.

In this case, the weight calculation unit calculates the slope of the frequency between the class with the highest frequency in the histogram and the class adjacent to and above the class, and based on this slope, A weighting coefficient that increases as the absolute value decreases may be further calculated. Then, the open/close state determination unit may weight the open/close degree by multiplying the weight by a weight coefficient.

The eye open/closed detection device according to the present invention may further include a light emitting unit that irradiates the face of the subject with light. Then, the imaging unit generates a first image captured while the light emitting unit is not emitting light and a second image captured while the light emitting unit is emitting light, and the image processing unit is configured to generate the second image and the first image. The predetermined area in the face image may be extracted based on the difference image that is the difference from the image, and the histogram calculation unit may calculate the histogram of the predetermined area in the first image.

In the eye open/closed detection device according to the present invention, the predetermined region may be a peripheral region of the subject's eye.

In the occupant monitoring device according to the present invention, the alarm output unit outputs an alarm signal when the open/closed state determination unit determines that the eye open/closed state is the eye closed state, and the eye closed state continues for a predetermined time or more. May be.

According to the present invention, it is possible to detect a fine-eyed state in the presence of direct light by distinguishing it from a fine-eyed state due to drowsiness, so that an alarm signal is erroneously output for the fine-eye state caused by direct light. Can be prevented.

It is a block diagram of a driver monitor to which the present invention is applied. It is a figure which shows a mode that a camera images a face. It is a figure which shows the face area|region in a captured image. It is a figure which shows typically an off image, an on image, and a difference image when there is no direct light. It is a figure which shows typically an off image, an on image, and a difference image when there is a direct light. It is a figure which shows the pixel in a face area. It is a figure which shows a weight table. It is a graph of a weighting factor. It is a figure explaining the inclination used for calculation of a weighting factor. It is an example of a histogram of luminance when direct light is not applied. It is an example of a histogram of luminance when direct light is applied. It is a figure which shows an eye open state, a thin eye state, and an eye closed state typically. It is a flow chart which shows operation of a driver monitor. It is a figure which shows the peripheral region of the eye in other embodiment. It is a figure which shows the weight table in other embodiment.

Embodiments of the present invention will be described with reference to the drawings. In each figure, the same or corresponding parts are designated by the same reference numerals. An example in which the present invention is applied to a driver monitor mounted on a vehicle will be described below.

First, the configuration of the driver monitor will be described with reference to FIG. In FIG. 1, the driver monitor 100 is mounted on the vehicle 50 in FIG. 2 and includes a camera 1, an image processing unit 2, a control unit 3, an LED drive circuit 4, and a communication unit 5. The driver monitor 100 is an example of the “eye opening/closing detection device” and the “occupant monitoring device” of the present invention.

The camera 1 includes an imaging unit 11 and a light emitting unit 12. The image pickup unit 11 includes an image pickup element (not shown) such as a CMOS image sensor, and an optical component (not shown) such as a lens. The light emitting unit 12 has a light emitting element (not shown) that irradiates the subject of the camera 1 with light. In this example, an LED that emits near infrared light is used as the light emitting element.

As shown in FIG. 2, the camera 1 is installed at a position facing the face F of the driver 53 sitting on the seat 52 in the vehicle interior. The dotted line represents the imaging range of the camera 1. In this example, the camera 1 is installed on the instrument panel 51 in the driver's seat, but the installation location of the camera 1 is not limited to this. For example, the camera 1 may be provided above the windshield 54. The imaging unit 11 (FIG. 1) of the camera 1 images the face F of the driver 53, and the light emitting unit 12 (FIG. 1) irradiates the face of the driver 53 with near infrared light. The driver 53 is an example of the “subject” and the “occupant” in the present invention.

The image capturing section 11 captures an image captured in a state where the light emitting section 12 does not emit light (hereinafter referred to as “off image”) and an image captured in a state in which the light emitting section 12 emits light (hereinafter referred to as “on image”). ) And are generated and the respective image data are output to the image processing unit 2. The off image corresponds to the "first image" in the present invention, and the on image corresponds to the "second image" in the present invention.

FIG. 4A and FIG. 5A schematically show examples of off-images. FIG. 4A is an off-image taken when the face F is not exposed to direct light such as sunlight, and FIG. 5A is an image captured when the face F is exposed to direct light. It is an off image. The off-image of FIG. 4A when there is no direct light becomes a darker image than the off-image of FIG. 5A when there is direct light.

FIG. 4B and FIG. 5B schematically show examples of on-images. FIG. 4B is an on-image captured when the face F is not exposed to direct light, and FIG. 5B is an on-image captured when the face F is exposed to direct light. is there. The on-image in FIG. 4B is brighter than the off-image in FIG. 4A because light is emitted from the light emitting unit 12. On the other hand, the on-image of FIG. 5B when there is direct light is brighter than the off-image of FIG. 5A because the light irradiation from the light emitting unit 12 is added to the direct light. Since the influence is great, the difference in lightness is not so remarkable as in the off image and the on image in FIG.

The image processing unit 2 generates an image (hereinafter referred to as a “difference image”) obtained by calculating the difference between the on-image and the off-image acquired from the imaging unit 11. 4C and 5C schematically show an example of the difference image. FIG. 4C is a difference image when there is no direct light, and FIG. 5C is a difference image when there is direct light. The difference image of FIG. 4C is a bright image because the difference between the ON image and the OFF image is large, whereas the difference image of FIG. 5C has a small difference between the ON image and the OFF image. Therefore, it becomes a dark image.

Furthermore, the image processing unit 2 extracts a region Z of the face F in the captured image G (hereinafter, referred to as “face region Z”), as shown in FIG. 3, based on the difference image. Then, feature points such as eyes, nose, and mouth in the face area Z are extracted, the direction of the face F is detected, and the direction of the line of sight is detected. These processes are performed using a known method, and the details thereof are well known, and therefore the description thereof is omitted here.

Returning to FIG. 1, the control unit 3 includes a CPU and a memory. The control unit 3 includes an open/close degree calculation unit 31, a histogram calculation unit 32, a weight calculation unit 33, an open/closed state determination unit 34, an alarm output unit 35, and a weight table 36. Each function of the opening/closing degree calculation unit 31 to the alarm output unit 35 is actually realized by a software program executed by the CPU, but in FIG. 1, it is shown as a hardware block for convenience. The weight table 36 is stored in a storage element such as a flash memory.

The open/closed degree calculation unit 31 calculates the open/closed degree indicating the open/closed state of the eye based on the image of the eye detected by the image processing unit 2. As the degree of opening and closing of the eye, for example, the ratio between the vertical width and the horizontal width of the eye as in the above-mentioned Patent Document 1, the distance from the boundary between the upper eyelid and the eyeball as in Patent Document 2 to the boundary between the lower eyelid and the eyeball, and Patent The ratio of the area of the upper eyelid to the area of the entire eye as in Reference 3 can be used. Of course, the open/close degree according to definitions other than these may be used.

Here, for simplicity, it is assumed that the degree of opening and closing obtained by image analysis takes a value of 0 to 100, the degree of opening of the eye increases as the degree of opening and closing increases, and the degree of closing of the eye increases as the degree of opening and closing decreases. And The degree of opening/closing 100 represents a completely opened state, and the degree of opening/closing 0 represents a completely closed state. Further, as an example, the opening/closing degree of 50±α represents a fine eye state (α is a predetermined value).

FIG. 12 schematically shows an example of the open/closed state of the eyes. (A) is a case where the degree of opening and closing is 100, and represents a completely opened state as described above. (B) shows the case where the degree of opening and closing is 70 and shows a fine eye state. (C) shows a state in which the degree of opening and closing is 30, and the eye is narrower than that in (b). (D) is a case where the degree of opening and closing is 0, and represents a completely closed eye state as described above.

When the driver becomes drowsy, the eye condition becomes a fine eye condition as shown in (b) and (c) of FIG. 12, but even when the driver's face is exposed to direct light, Due to the glare, such a microscopic state can occur. Therefore, in a system in which a warning is output when a narrow eye condition continues for a certain period of time or more, it is considered as a sign of drowsiness, even if the fine eye condition is caused by the glare of direct light May be output. The present invention solves this problem by weighting the degree of opening and closing according to the brightness of the direct light, as will be described later in detail.

Returning to FIG. 1 again, the histogram calculation unit 32 calculates a luminance distribution, that is, a histogram in the face area Z (FIG. 3) of the off-image. The weight calculator 33 calculates a parameter for weighting the open/close degree calculated by the open/close degree calculator 31 according to the brightness. The open/closed state determination unit 34 determines the open/closed state of the eyes based on the weighted open/closed degree, and determines whether or not the driver is dozing while depending on the result. Details of these processes will be described later.

When the open/close state determination unit 34 determines that the vehicle is dozing, the alarm output unit 35 outputs an alarm signal for notifying that. This alarm signal is transmitted to an ECU (Electronic Control Unit) 200 mounted on the vehicle via the communication unit 5. The weight table 36 stores a weight value corresponding to the brightness (FIG. 7 described later).

The LED drive circuit 4 is a circuit for driving the LEDs of the light emitting unit 12, and supplies a predetermined drive current to the LEDs to cause the LEDs to emit light. The light amount and the light emission time of the LED are controlled by the control unit 3.

The communication unit 5 is composed of an interface or the like for connecting the driver monitor 100 to the ECU 200. The communication unit 5 is connected to the ECU 200 via a CAN (Controller Area Network) and exchanges signals and data with the ECU 200. Although only one ECU 200 is shown in FIG. 1, a plurality of ECUs are actually mounted on the vehicle according to the control target, and the communication unit 5 is connected to a predetermined ECU among them. To be done.

Next, a specific method of detecting the open/closed state of the eyes on the driver monitor 100 will be described.

FIG. 6 schematically shows pixels in the face area Z of FIG. The image here is an off image captured without causing the light emitting unit 12 to emit light. The face area Z on the image is composed of a large number of pixels P corresponding to the arrangement of the image pickup elements of the image pickup section 11. The histogram calculation unit 32 acquires the luminance information on a pixel-by-pixel basis. The brightness information is a brightness value corresponding to the value of the current photoelectrically converted by each image sensor corresponding to each pixel P.

The histogram calculation unit 32 calculates a histogram showing the distribution of brightness in the face area Z based on the brightness value acquired in pixel units. FIG. 10 is an example of the histogram when the direct light is not applied. 11A and 11B are examples of the histogram when the direct light is applied, where FIG. 11A shows a state where the direct light is weak, and FIG. 11B shows a state where the direct light is strong.

In this example, the brightness value is divided into 256 gradations of 0 to 255, and these are further divided into 16 gradations. Hereinafter, this division is referred to as "class". The brightness of each pixel P belongs to one of the classes on the horizontal axis, and the number (total) of the pixels P belonging to the class represents the frequency on the vertical axis.

As shown in FIG. 7, weights Wm are set in the weight table 36 in correspondence with the luminance classes. Each weight Wm has a value corresponding to the mode value of each class. The ranks 0 to 31 are normal states in which no direct light is applied, and in this case there is no need for weighting, so the weight is Wm=1.0. When the class is 32 or more, it is in a state where it is exposed to the direct light, and in this case, since the weighting is performed according to the brightness, the value of the weight Wm increases according to the class. Here, the maximum value of the weight Wm is 4.0, but this is only an example. The weight Wm is not set for the classes 192-255. This is because the direct light is too strong in the range and there is no difference in brightness between the on-image and the off-image, and when the difference between them is taken, the face image becomes dark and the eyes cannot be detected. This is because the eye open/closed state cannot be determined in the first place.

The weight calculator 33 further calculates the weight coefficient Wg shown in FIG. 8 in addition to the weight Wm obtained from the weight table 36 as a parameter for performing weighting. As shown in FIG. 8, the weighting factor Wg is represented by Wg=2 ^a +1. Here, a is the slope shown in FIG. 9, and this slope a is the class with the highest frequency (16 to 31 in FIG. 9) and the classes adjacent to and above the class (FIG. 9). 9 is the gradient of the frequency between 32 and 47) (the gradient of the straight line connecting the two numerical values). When the frequencies of the adjacent classes are the same, the slope a is calculated between the larger (upper side) class and the next class adjacent thereto. Therefore, the slope a never becomes a positive value or 0, and always becomes a negative value (a<0).

The open/closed state determination unit 34 weights the eye open/closed degree X calculated by the open/closed degree calculation unit 31 by using the weight Wm and the weight coefficient Wg calculated by the weight calculating unit 33 to determine the open/closed degree X. to correct. Specifically, the correction value Y of the opening/closing degree X is calculated by the following equation.
Y=X·Wm·Wg (1)
The corrected opening/closing degree Y is the determination opening/closing degree used for the determination of the eye open/closed state.

The degree of opening and closing of the eye is corrected in this manner in order to detect the fine eye state caused by direct light separately from the fine eye state caused by drowsiness. As described above, when direct light such as sunlight is incident from the windshield 54 of the vehicle 50 and hits the face F of the driver 53, when the driver 53 becomes in a fine-eyed state due to glare (for example, in FIG. 12(c)), the alarm may be erroneously output because it is considered as a sign of dozing. Therefore, in the present invention, the eye open/closed degree X calculated by the open/closed degree calculating unit 31 is first weighted by the weight Wm (FIG. 7) according to the brightness of the direct light. Further, the value of the opening/closing degree Y is increased by weighting the direct light having a large luminance with the large weight Wm in the equation (1). As a result, even if the driver 53 narrows his eyes with direct light, the degree of opening/closing Y has a large value, so that it is possible to prevent the thin eye state due to glare from being erroneously determined as a sign of dozing.

Further, in the present invention, double weighting is performed by using the weighting coefficient Wg in addition to the weight Wm. This is for the following reasons. When the absolute value of the slope a between the maximum frequency class and the next class is large (when a is far from 0), only the brightness of the maximum frequency stands out, whereas when the absolute value of the slope a is small (a is (When it is close to 0), the luminance of the maximum frequency and the luminance of the next frequency are both conspicuous. Therefore, in the latter case, the direct light is stronger than in the former case. Therefore, as shown in FIG. 8, by multiplying the weight Wm by a weighting coefficient Wg that has a larger value as the inclination a is closer to 0 (that is, as direct light is stronger), the degree of eye opening/closing is corrected with higher accuracy. be able to.

The open/closed state determination unit 34 determines the open/closed state of the eyes of the driver 53 by comparing the open/closed degree Y (correction value) for determination calculated by the equation (1) with a predetermined threshold value. As an example, when the threshold value is 50 and the opening/closing degree Y is in the range of 50≦Y≦100 (equal to or more than the threshold value), the open/closed state determination unit 34 determines that the driver 53 is in the eye open state. When the opening/closing degree Y is in the range of 0≦Y<50 (less than the threshold value), the opening/closing state determination unit 34 determines that the driver 53 is in the eye-closed state. The fine eye state is classified into either the closed eye state or the open eye state depending on the value of the opening/closing degree Y.

When the open/closed state determination unit 34 further determines that the eye open/closed state of the driver 53 is the closed eye state, the open/closed state determination unit 34 measures the time period during which the closed eye state continues, and when the closed eye state continues for a predetermined time or more, the driver 53 falls asleep. It is judged that the patient is in a state or a dozing sign. When the open/close state determination unit 34 makes this determination, the alarm output unit 35 outputs an alarm signal for awakening the driver 53. As described above, this alarm signal is transmitted to ECU 200 via communication unit 5. Upon receiving the warning signal, ECU 200 performs a process for giving a drowsiness warning to driver 53 (for example, voice output or buzzer sounding).

FIG. 13 is a flowchart showing the operation of the driver monitor 100 described above.

13, in step S1, the camera 1 captures the face F of the driver 53 without causing the LEDs of the light emitting unit 12 to emit light, and generates an off-image. In step S2, the camera 1 causes the LED of the light emitting unit 12 to emit light, images the face F of the driver 53, and generates an on-image. The order of step S1 and step S2 may be interchanged.

In step S3, the image processing unit 2 calculates the difference between the on-image and the off-image acquired from the camera 1 to generate a difference image. Then, in step S4, the image processing unit 2 extracts the face area Z (FIG. 3) based on the difference image, and also extracts feature points such as eyes, nose, and mouth.

In step S5, the opening/closing degree calculation unit 31 calculates the eye opening/closing degree X based on the image of the eye detected in step S4. In step S6, the histogram calculation unit 32 calculates a histogram (luminance distribution) in the face area Z of the off-image. In step S7, the weight calculation unit 33 refers to the weight table 36 (FIG. 7) and acquires the weight Wm corresponding to the maximum frequency class in the histogram. In addition, the weight calculation unit 33 calculates the slope a (FIG. 9) between the maximum frequency class and the next class in step S8, and also calculates the weight coefficient Wg=2 ^a +1 (FIG. 8) in step S9. Calculate

Next, in step S10, the open/close state determination unit 34 corrects the open/close degree X by the above equation (1) using the open/close degree X, the weight Wm, and the weight coefficient Wg obtained in steps S5, S7, and S9. The value Y is calculated (Y=X·Wm·Wg). As described above, the correction value Y is the opening/closing degree for determination. In the following step S11, the open/closed state determination unit 34 determines the open/closed state of the eye (open or closed) based on the comparison result between the correction value Y and the threshold value. Then, in step S12, the open/closed state determination unit 34 determines whether or not the eye-closed state continues for a predetermined time or more.

As a result of the determination, when the eye-closed state continues for the predetermined time or more (step S12: YES), the process proceeds to step S13, and the alarm output unit 35 outputs an alarm signal. When the eye-closed state has not continued for the predetermined time or more (step S12: NO), step S13 is omitted and the process returns to the beginning. Even if it is determined in step S11 that the eye is open, the determination in step S12 is NO, and step S13 is omitted and the process returns to the beginning.

According to the above-described embodiment, the eye open/closed degree X calculated by the open/closed degree calculating unit 31 is weighted by the weight Wm and the weight coefficient Wg according to the luminance distribution of the face area Z, and the weighted open/close degree is set. The open/closed state of the eyes of the driver 53 is determined based on the degree Y. Therefore, the open/closed state of the eye is detected based on the opening/closing degree Y corrected according to the state of the direct light, so that the fine eye state due to the glare of the direct light is detected separately from the fine eye state due to the drowsiness. It becomes possible to do. As a result, when the driver 53 squints his eyes under direct light, the open/close state determination unit 34 erroneously determines that it is a sign of drowsiness, and the alarm output unit 35 outputs an alarm signal. Can be prevented.

In the present invention, in addition to the embodiments described above, the following various embodiments can be adopted.

In FIG. 6, the example in which the brightness information is acquired for the pixels P included in the entire area of the face area Z is described. However, even if the brightness information is acquired for the pixels P included in a partial area of the face area Z, Good. For example, as shown in FIG. 14, the luminance information may be acquired for the pixels included in the peripheral area Q of the eye in the face area Z. This is for the following reasons.

When the sun visor (not shown) provided on the upper portion of the windshield 54 (FIG. 2) is pulled down to shade the sun, the brightness of the peripheral region of the eyes of the driver 53 decreases. Therefore, if the histogram is calculated based on the brightness of all the pixels included in the face area Z, the value of the opening/closing degree Y may be small, and the open/closed state of the eye may be erroneously determined. Therefore, by calculating the histogram based on the brightness of the pixels in the peripheral region Q of the eye, such a problem can be solved. A sensor for detecting that the sun visor has been lowered is provided, and when the detection signal is output from the sensor, the area for acquiring the brightness information is automatically switched from the face area Z to the peripheral area Q of the eye. You may do it.

In the weight table 36 of FIG. 7, an example in which the histogram class is divided into 16 gradations is given, but the present invention is not limited to this. For example, as shown in FIG. 15, the histogram class is divided into 48 gradations, and the brightness class is classified into four ranks of “low”, “medium”, “high”, and “calculation impossible”, and “low”. The weight Wm may be set for the "medium" and "high" ranks.

6 and 14, the example in which the luminance information (luminance value) is acquired in the unit of each pixel is described, but the present invention is not limited to this. For example, a plurality of pixels may be divided into blocks and the brightness information may be acquired in block units. In this case, the maximum brightness value of each pixel included in one block may be used as the brightness information of the block, or the average brightness value of each pixel may be used as the brightness information of the block.

In FIG. 3, the face area Z in the captured image G has the same shape as the driver's face F, but the face area Z is a region including the face F such as a quadrangle, a rhombus, an ellipse, and a circle. May be

In FIG. 8, the example in which the weighting factor Wg is calculated as Wg=2 ^a +1 has been described, but this is an example, and the weighting factor Wg may be calculated by another arithmetic expression.

In the above embodiment, the driver 53 is given as an example of the occupant of the vehicle, but the occupant who is the subject of the camera 1 may be a person other than the driver.

In the above-described embodiment, the example in which the present invention is applied to the driver monitor 100 mounted on the vehicle has been described, but the present invention can be applied to applications other than the vehicle.

DESCRIPTION OF SYMBOLS 1 camera 11 imaging part 12 light emitting part 2 image processing part 3 control part 4 LED drive circuit 5 communication part 31 switching degree calculation part 32 histogram calculation part 33 weight calculation part 34 open/closed state determination part 35 alarm output part 36 weight table 53 driver 100 Driver monitor (eye opening/closing detection device, occupant monitoring device)
F face G captured image P pixel
Q Eye peripheral area Z Face area

Claims (7)

An imaging unit that images the face of the subject,
An image processing unit that performs a predetermined process on the face image captured by the image capturing unit and extracts the eye of the subject from the face image,
Based on the image of the eye extracted by the image processing unit, an open/closed degree calculation unit that calculates an open/closed degree indicating the open/closed state of the eye,
A weight calculation unit that calculates a weight according to luminance information of a predetermined area in the face image,
An open/closed state determination unit that weights the open/closed degree calculated by the open/closed degree calculation unit with the weight, and determines the open/closed state of the eye based on the weighted open/closed degree, is provided. A device for detecting eye opening/closing. The eye opening/closing detection device according to claim 1,
A histogram calculation unit that calculates a histogram showing the distribution of the brightness of a predetermined area in the face image,
A weight table in which the weights are set corresponding to the luminance classes,
The eye opening/closing detection apparatus, wherein the weight calculation unit refers to the weight table and selects a weight corresponding to a class having a maximum frequency in the histogram calculated by the histogram calculation unit. The eye opening/closing detection device according to claim 2,
The weight calculator is
Between the class with the highest frequency in the histogram and a class adjacent to and above the class, calculate the slope of the frequency,
On the basis of the slope, a weighting coefficient that increases as the absolute value of the slope becomes smaller is further calculated,
The eye open/closed detection device, wherein the open/closed state determination unit weights the open/closed degree by multiplying the weight by the weighting coefficient. The eye open/closed detection device according to claim 2 or 3,
Further comprising a light emitting unit for irradiating the face of the subject with light,
The image capturing unit generates a first image captured while the light emitting unit is not emitting light and a second image captured while the light emitting unit is emitting light,
The image processing unit extracts the predetermined region in the face image based on a difference image that is a difference between the second image and the first image,
The eye opening/closing detection device, wherein the histogram calculation unit calculates a histogram of the predetermined region in the first image. The eye opening/closing detection device according to any one of claims 1 to 4,
The eye opening/closing detection device, wherein the predetermined area is an area around the eye of the subject. An image pickup unit that picks up an image of the face of a vehicle occupant;
An image processing unit that performs a predetermined process on the face image captured by the image capturing unit and extracts the eyes of the occupant from the face image,
Based on the image of the eye extracted by the image processing unit, an open/closed degree calculation unit that calculates an open/closed degree indicating the open/closed state of the eye,
A weight calculation unit that calculates a weight according to luminance information of a predetermined area in the face image,
An open/closed state determination unit that determines the open/closed state of the eye based on the weighted open/closed degree, with respect to the open/closed degree calculated by the open/closed degree calculation unit.
An occupant monitoring device, comprising: an alarm output unit that outputs an alarm signal based on the determination result of the open/closed state determination unit. The occupant monitoring system according to claim 6,
The alarm output unit, the open/closed state determination unit determines that the eye open/closed state is a closed eye state, and when the closed eye state continues for a predetermined time or more, outputs the alarm signal, Occupant monitoring device.