JPH1166304A - Face image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a face image processor which stably and correctly decides the opening and closing of eyes for a long time even when the distance between a photographed person and an image pickup camera changes and the direction of his face is changed. SOLUTION: An eye image is extracted from a face image which is picked up by a camera 1, an image inputting means 2 and an eye image means 3, and an opening/closing evaluating means 4 evaluates the degree of opening/ closing eyes. An opening/closing threshold that is set by a filtering means 5 and an opening/closing threshold setting means 6 and an opening/closing threshold controlling threshold that is set by an opening/closing threshold controlling threshold setting means 9 are inputted to an opening/closing threshold resetting means 10, and an opening/closing threshold that evaluates eyes from opening to closing is set. A looking down eye threshold, etc., from a looking down eye threshold setting means 11 are inputted to an opening/closing deciding means 7, and it decides either a blink or looking down eyes and decides whether eyes are open or closed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a face image processing apparatus for detecting a state of reduced arousal level, a dozing state, and the like from the open / closed state of eyes.

[0002]

2. Description of the Related Art Conventionally, in order to detect a driving state of a vehicle driver, such as looking aside or falling asleep, a driver's face is photographed by a camera provided in a vehicle cabin, and the obtained face image is processed. 2. Description of the Related Art A face image processing apparatus has been developed which extracts eyes, which are feature points of a face, and detects whether the eyes are open or closed. As such a conventional example, after converting a density gradation image into a binary image, eyes are extracted from a black region inside the extracted face outline, and the maximum number of continuous black pixels in the detected eye region is determined. An invention that determines whether the eyes are opened or closed using numbers is disclosed in Japanese Patent Application Laid-Open No. 6-32154.

Conventional example 1. FIG. 12 shows Japanese Patent Application Laid-Open No. 6-32154.
FIG. 13 is a flowchart of a state detection of a driver in Japanese Patent Application Laid-Open Publication (Conventional Example 1), and FIG. 13 is a diagram for determining whether to open or close eyes from a binary image of an eye region extracted from an image in Conventional Example 1. FIG.

In FIG. 1, first, a density gradation image from a camera photographing a driver's face is input to an image input means 2 as a video signal, and is subjected to analog / digital conversion (A / D conversion). The digital gradation image obtained is input to the frame memory in step S90.

Next, the image data stored in the frame memory is read out, and is converted into a binary image by an appropriate binarization threshold by the binarization means in step 91, and the horizontal direction of the face determined in step S92. White pixels are searched in the horizontal direction of the face from the search start line, the number of continuous white pixels is counted in step S93, and the face is counted from the edge of the white pixel area when the maximum of the continuous white pixels is reached in step S94. The width outline is specified.

Further, the horizontal coordinates of the face in the eyeball existing area are set in step S95 from the horizontal outline of the face specified in step S94, and the search start line in the vertical direction of the face is set in step S96. In step S97, a black pixel region where black pixels are continuous from the search start line is searched in the eyeball existing region, and step S98 is performed.
In, the eyeball region is specified based on the positional relationship of the searched black pixel region and the number of black pixels in the vertical direction.

Next, the blink detecting means 7 searches for the maximum value of the number of black pixels in the vertical direction in the specified eyeball area as shown in FIG. ) Is selected, and the number of continuous black colors on this line is detected as the degree of eye opening / closing. That is, when the eye is open, the maximum value of the number of black pixels is large, and when the eye is closed, the maximum value is smaller than that when the eye is open. MAX of the maximum number of black pixels during a minute
Value and MIN value.

Finally, the dozing determination means in step 99 determines dozing based on the open / closed state of the eyes detected by the blink detecting means 7.

Conventional example 2. In addition, ourselves, Japanese Patent Application Hei 7-
In 309260 (Japanese Unexamined Patent Application Publication No. 9-147120; Conventional Example 2), a face image processing apparatus for accurately detecting the opening and closing of eyes even in response to a change in the driver's posture has already been filed. Hereinafter, the invention of the face image processing apparatus will be described.

FIG. 14 shows an embodiment in Japanese Patent Application No. Hei 7-309260. This face image processing apparatus includes a camera 1 for taking a face image, an image input means 2 for inputting a face image, and specifying an eye area. Eye opening / closing evaluation means 4 for outputting an evaluation value according to the eye opening / closing state, time measuring means 12, threshold evaluation means 13, threshold setting means 6, and eye opening / closing determination means 7. .

Next, the operation of the face image processing apparatus will be described. The face image captured by the camera 1 is input to the image input means 2 and A / D-converted, and further converted to a digital gradation image, and converted into a digital gradation image by the eye area determination means 3 in the same manner as in the conventional example 1 described above. Is specified. Next, the eye image in the specified eye area is processed by the eye open / close evaluation means 4, and a value corresponding to the eye open / closed state is output. Here, the eye opening / closing evaluation means 4 is, for example, the maximum value of the number of continuous black in the vertical direction in the above-mentioned eye area,
Any evaluation method may be used as long as the value changes monotonically depending on the eye open / closed state.

Next, the output of the eye open / close evaluation means 4 is input to the time measurement means 12 and the open / close determination means 7. Further, two thresholds are also input to the time measuring unit 12 from the threshold setting unit 6. Next, the operation of the time measuring means 12 will be described with reference to the time chart shown in FIG. FIG.
At 5, the output of the eye opening / closing evaluation means 4 is compared with the two threshold values, the open side threshold value and the close side threshold value, which are input from the threshold value setting means 6, and for example, as shown in FIG. And the group B on the open side with respect to the closed threshold value are created. The time measuring means 12 integrates and measures the time of the hatched portion of the group A and the group B shown in the figure every predetermined time. FIG. 15 shows a state in which the two threshold values set by the threshold value setting means 6 are appropriate values, the open side threshold value is just at the lower end of the fluctuation of the output of the opening / closing evaluation means 4, and the close side threshold value is open. Since the setting is made with an appropriate offset from the side threshold value, there is a sufficient margin for the fluctuation of the output of the eye opening / closing evaluation means 4 in the eye open state, and the eye closing can be detected reliably.

Next, the operation of the threshold evaluation means 13 will be described with reference to FIGS.
This will be described with reference to the time chart shown in FIG. FIG.
Indicates the time when the two thresholds are appropriate values, the open side threshold is just applied to the lower end of the fluctuation in the open state of the output of the opening / closing evaluation means 4, and the close side threshold is an appropriate offset from the open side threshold. , Which indicates that the fluctuation in the open state does not affect the threshold value. Therefore, as can be seen from the figure, when comparing the accumulated time of the group A and the accumulated time of the group B, the accumulated time of the group A is always shorter than the accumulated time of the group B. ).

FIGS. 16 and 17 show the case where the threshold value is set too much to the open side and the case where the threshold value is set too close to the closed side, respectively, as in FIG. Is almost zero, and it can be seen that the threshold value on the closing side is affected by the fluctuation of the open state of the output of the eye open / close evaluation means 4. Group A in such a state
Comparing the cumulative time of the group B with the cumulative time of the group B, the cumulative time of the group A is << the cumulative time of the group B.

On the other hand, in FIG. 17, since the open side threshold value is closer to the closed side than the lower end of the fluctuation of the open state of the output of the eye opening / closing evaluation means 3, the integration time of the group A and the integration time of the group B are compared. If you do, (accumulated time of group A nearly
equal (integration time of group B). Therefore, by comparing and evaluating the accumulated time of the group A and the accumulated time of the group B, it is possible to evaluate whether the two thresholds set by the threshold setting means 6 are appropriate or to which side they are excessive. Can be.

Finally, the opening / closing determination means 7 performs the opening / closing determination using, for example, the closing threshold set by the threshold setting means 6, and specifically, determines the output of the eye opening / closing evaluation means 3 and the closing threshold. The comparison is performed, and it is determined that the eye is open when the opening is higher than the threshold value on the closing side, and that the eye is closed when the opening is closer to the closing threshold.

[0017]

However, in such a conventional apparatus, the threshold value is controlled while evaluating the validity of the opening / closing determination threshold value based on the fluctuation of the output of the eye opening / closing evaluation means within a predetermined time. In addition, since the open / close determination is performed, the responsiveness of the threshold control is not good. for that reason,
If the driver's posture changes during a predetermined time or the state of the external light that hits the driver changes, the eye is mistakenly detected as closed even though the eye is not closed, or the closed eye is detected even if the eye is closed There was a problem that it could not be done.

Further, when the driver covers his eyes to look at the speedometer or the like, the output of the opening / closing evaluation means is accordingly closed, so that the driver is not closed even if the eyes are not actually closed. There is a problem that erroneous detection that the eye is closed for a long time is likely to occur.

In addition, there is a problem that, when the eyes are closed after the eyes are closed once, the eyes are erroneously detected as the long eyes closed despite the short eyes closing time.

In addition, depending on the degree of contact with external light, for example, if the eye image is not sufficiently extracted due to shadowing, or if the eye image is overexposed due to too strong light, the eye may be erroneously closed. There was a problem that it was determined.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and suppresses erroneous detection even when the driver's posture changes or the state of external light changes, and stably opens and closes the eyes. It is an object of the present invention to obtain a face image processing apparatus capable of determining the face image.

[0022]

According to the present invention, there is provided a face image processing apparatus comprising: a camera for picking up a face image; an eye image extracting means for extracting an eye image based on a video signal from the camera;
Opening / closing evaluation means for inputting the eye image and outputting a continuous opening / closing evaluation value according to eye opening / closing, filter means for removing a high-frequency component of the output of the opening / closing evaluation means, Opening / closing threshold setting means for setting an opening / closing threshold at a predetermined distance in the closing direction, and eye opening / closing determination for comparing the output of the opening / closing evaluation means with the output of the opening / closing determination threshold setting means to determine whether the eye is open or closed. Means.

The threshold value for opening / closing is set as a threshold value for determining whether the eye is open or closed, and the threshold value is determined by a predetermined calculation based on the output of the eye opening / closing evaluation means during the closing determination and the threshold value for closing. An opening determination threshold value setting unit that sets a value as an opening determination threshold value from when the eye is closed to when the eye is opened is provided.

Further, at least one opening / closing judgment threshold control threshold is provided at a predetermined distance in the opening direction from the opening / closing judgment threshold, and the output of the opening / closing evaluation means is closer to the closing side than the opening / closing judgment threshold control threshold and the opening / closing judgment is performed. An opening / closing determination threshold value control means for prohibiting the update of the opening / closing determination threshold value for a predetermined time from when the threshold value is opened to the open side is provided. Further, when the output of the opening / closing evaluation means is closer to the closing side than the opening / closing determination threshold during the predetermined period and the closed eye is not detected, the update prohibition of the opening / closing determination threshold is released after a predetermined time has passed, and the opening / closing is newly performed. An opening / closing determination threshold resetting means for setting a determination threshold is provided.

[0025] Further, a closing stick determination threshold is set at a predetermined value in the closing direction from the opening / closing determination threshold, and the output of the opening / closing evaluation means is in a state closer to the opening side than the opening / closing determination threshold and closer to the opening side than the above closing opening determination threshold. Is provided for a predetermined period of time, comprising a closed-eye shape determining means for evaluating the time shape of the output of the eye opening / closing evaluation means during the time, and a close determination is made when the closed-eye shape determining means determines that the eye is closed. Eye closing means which does not perform the operation.

Further, there is provided a means for setting the above-mentioned blindfold judgment threshold value using a predetermined calculation from the output of the eye opening / closing evaluation means when the eye closing judgment time is within a predetermined range and the opening / closing judgment threshold value when the eye closing judgment is made. Was.

Further, after the closed eye determination means determines that the eye is closed, the state of the output of the eye open / closed eye evaluation means is closer to the open side than the open determination threshold and is longer than the close-up eye determination threshold for a predetermined time. In the case where the output of the eye opening / closing evaluation means is on the opening side of the close-up eye determination threshold when the eye opening / closing evaluation means continues, the eye closing detection means is provided.

[0028]

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiment 1 FIG. According to the first embodiment of the present invention, a face of a driver or the like is photographed by a camera, a face image is input to an image input unit, and an eye image is extracted based on the face image by an eye image extracting unit. The eye open / close evaluation unit obtains an evaluation value corresponding to the opening / closing of the eye from the image, obtains an average open / close evaluation value at the time of eye opening by the filter unit, sets an appropriate open / close determination threshold by the open / close threshold setting unit, The opening / closing of the eyes is detected based on the evaluation value and the opening / closing determination threshold.

FIG. 1 is a block diagram showing a configuration of a face image processing apparatus according to Embodiment 1 of the present invention. This face image processing apparatus receives a camera 1 provided in a vehicle interior for capturing a face of a vehicle driver or the like, and receives a video signal as image capturing information of the camera 1 to perform A / D conversion and output a digital gradation image. Image input means 2, an eye image extracting means 3 which receives the digital gradation image of the image input means 2 and specifies an eye area in which the eyes of the vehicle driver are considered to be present, and an opening of the eyes of the vehicle driver Opening / closing evaluation means 4 as opening degree detection means for outputting a detection result corresponding to the degree, filter means 5 for removing a high-frequency component of the output waveform of opening / closing evaluation means 4, Open / close determination threshold value setting means 6 for outputting a waveform separated by a predetermined value from the other, and eye open / close determination means 7 for comparing the output of the open / close evaluation means 4 with the output of the open / close determination threshold value setting means 6 to determine whether the eye is open or closed. It is composed of

Next, the operation of the face image processing apparatus according to the first embodiment will be described. The face image of the vehicle driver or the like taken by the camera 1 is input to the image input means 2 and A / D converted to a digital gradation image in the same manner as described in the conventional example. The eye image extracting means 3 specifies an eye image in the image. Next, the specified eye image is processed by the opening / closing evaluation means 4, and a value corresponding to the eye opening / closing state is output. Here, the opening / closing evaluation means 4
Is, for example, the maximum value of the number of continuous black in the vertical direction in the above-mentioned eye region, and if the value changes monotonically depending on the open / closed state of the eyes, any evaluation method is used. Is also good.

As shown in FIG. 2, the output of the opening / closing evaluation means 4 is obtained by superimposing a change in the value due to the state of the eye image or the posture of the driver and a change in the value due to blinking, as shown in a dotted line 4a. Becomes As shown by the dotted line 4a, in the case of blinking, the movement of the opening / closing evaluation value becomes steeper than that of a change in posture or the like.

Next, when the waveform of the dotted line 4a is inputted to the filter means 5 and the high-frequency component of the waveform is removed, as shown by the solid line 5a, the output change of the open / close evaluation means 4 other than the blink does not change much. A waveform in which a sudden change such as the open / close evaluation value 4b due to blinking is attenuated is obtained. The output 5a of the filter 5 is input to the opening / closing threshold setting means 6, and a waveform 6a (dashed line) separated from the output 5a by a predetermined value on the closing side is output as an opening / closing determination threshold.

Finally, the opening / closing judgment means 7 compares the output 4a of the opening / closing evaluation means with the output 6a of the opening / closing judgment threshold setting means 6, and the output 4a of the opening / closing evaluation means 4 is compared with the output 4a of the opening / closing judgment threshold setting means 6. If the output 6a is on the open side, the open determination is made, and the output 4a of the open / close evaluation means 4 is set to the open / close determination threshold setting means 6
If the output 6a is on the close side, a close determination is made.

As described above, if the open / close determination is made using the procedure shown in the first embodiment, only those having a sudden change in the open / close evaluation means such as the blink attenuated by the filter means 5 are detected as blinks. it can.

In addition, it is possible to set a threshold value so that the eyes can always be correctly determined whether the eyes are opened or closed even if the photographing conditions change due to individual differences between photographed persons. Eye opening and closing can be detected.

Embodiment 2 In the second embodiment, after the open / close determination unit of the first embodiment determines that the eyes are closed, the open / close determination threshold setting unit determines whether the eyes are closed and the open / close evaluation value during the closed eye determination. An open determination threshold value for determination is set, and an open to closed state is detected based on the open / close evaluation value and the open determination threshold value.

FIG. 3 is a block diagram showing the configuration of the face image processing apparatus according to the second embodiment. In addition to the configuration of the first embodiment, an opening determination threshold value setting means 8 is newly added.
The operation of the open / close determining means 7 is slightly different. Hereinafter, this embodiment will be described.

The face image picked up by the camera 1 is input to the image input means 2, converted into a digital gradation image by A / D conversion, and specified by the eye image extracting means 3 to identify an eye image in the image. Then, as in the first embodiment, the specified eye image is processed by the open / close evaluation means 4 and a value corresponding to the open / closed state of the eye is output. Further, the output of the opening / closing evaluation means 4 is input to the filtering means 5, and the output of the filtering means 5 is supplied to the opening / closing threshold setting means 6.
In the same manner as in the first embodiment, the opening / closing threshold is set by inputting the data to the opening / closing threshold value, and the opening / closing determination means 7 determines whether the opening / closing threshold value is set by the output of the opening / closing evaluation means 4 and the output of the opening / closing threshold setting means 6. However, the second embodiment differs from the first embodiment in the method of once determining whether the opening / closing determination unit 7 has closed and then determining the opening. Next, this difference will be described.

FIG. 4 is a time chart describing the operation of the second embodiment. In the figure, the solid line 4a indicates the opening / closing evaluation value output by the opening / closing evaluation means 4, and the solid line 6a
Represents an output waveform set by the opening / closing threshold value setting means 6 at a predetermined distance from the output of the opening / closing evaluation means 4 after the high-frequency component has been removed by the filter means 5, as described in the first embodiment.

In FIG. 4, reference numeral 6b denotes a predetermined value width between the output of the open / close threshold value setting means 6 and the output of the filter means 5,
a is a value obtained by adding a predetermined value to the opening / closing determination threshold value at the time of the first closing determination, 8b is the minimum closing value of the opening / closing evaluation value during the closing determination, and 8c is the amount of change from open to closed between the values 8a and 8b. , 8d are open determination threshold values calculated from the above change amount, solid line 7a is the output of the open / close determination means 7, time 7b is the time when the first close determination is made by the open / close determination threshold 6a, and time 7c is the open / close determination threshold 6a. At the time of making a determination (in the case of Embodiment 1), time 7d indicates the time of making an open determination based on the open determination threshold value 8c, the details of which will be described below.

In FIG. 4, at time 7b when the opening / closing evaluation value 4a is closer to the closing side than the opening / closing judgment threshold 6a, the opening / closing judgment means 7 judges whether the driver's eyes are closed, as in the first embodiment. At this time, the open threshold value setting means 8 records the value 8a obtained by adding the predetermined value 6b to the open side to the value of the open / close determination threshold value at the time of the close determination, and sets the most closed value 8b during the close determination. Record while updating. Here, assuming that the predetermined value 6b is a predetermined value used to set the opening / closing determination threshold from the output of the filter means 5 performed by the opening / closing determination threshold setting means 6, the value 8a is the opening / closing evaluation just when the closing is started. Can be considered a value. Now, since the value 8b is recorded while updating the most closed value during the closing judgment, the value 8b when the eyes are closed is obtained, and the distance 8c between the values 8a and 8b is determined from the opening of the eyes. This corresponds to the amount of change to close. Next, this change amount 8
In the distribution of c, for example, the open determination threshold value 8d
Is set as the output of the open determination threshold value setting means 8. Finally, when the open / close evaluation value 6a is closer to the open side than the open determination threshold 8d at the time 7d, the open / close determination unit 7 determines that the open / closed state is open. As a result, the opening determination time point 7c in the case of the first embodiment
A more accurate eye-closing duration can be detected, and the same eye-closing duration can be expected regardless of the magnitude of the change 8c in the opening / closing evaluation value 4a when the eyes are closed.

Embodiment 3 FIG. 5 is a block diagram showing the configuration of the face image processing apparatus according to the third embodiment. In addition to the configuration of the first embodiment, an opening / closing threshold control threshold setting unit 9 and an opening / closing threshold resetting unit 10 are newly added. Things. The opening / closing threshold control threshold setting unit 9 sets the opening / closing determination threshold control threshold 9a in the opening direction of the eyes based on the opening / closing determination threshold 6a set by the opening / closing threshold setting unit 6, and sets the opening / closing evaluation value 4a to the opening / closing determination threshold. When the control threshold value 9a is on the closed side and the open / close determination threshold value 6a is on the open side, the update of the open / close determination threshold value 6a by the open / close threshold value setting means 6 for a predetermined time is prohibited. In a case where the opening / closing evaluation value 4a is on the closing side of the opening / closing determination threshold for a predetermined period from the start of the update prohibition of the opening / closing determination threshold 6a by 6 and the closed eye determination is not performed, the update prohibition of the opening / closing determination threshold is released, and the current A new opening / closing determination threshold is set based on the opening / closing evaluation value or the output of the filter means.

As shown in FIG. 5, the third embodiment is based on the first embodiment, but there is no problem if it is applied to the second embodiment. Hereinafter, the third embodiment will be described with reference to FIG.

In the same manner as in the first embodiment, the face image picked up by the camera 1 is extracted from the face image by the image input means 2, the eye image extracting means 3, and the opening / closing evaluating means 4, and the face image is extracted. An opening / closing evaluation value corresponding to the opening / closing state is output. Then, the output of the opening / closing evaluation means 4 is input to the filtering means 5 to remove high-frequency components, and the output of the filtering means 5 is input to the opening / closing threshold value setting means 6, as in the first embodiment. An opening / closing threshold value that is separated from the output by a predetermined value in the eye closing direction is set.

The output of the opening / closing threshold setting means 6 is inputted to the opening / closing threshold control threshold setting means 9, and the opening / closing threshold control threshold outputted from the means 9 is inputted to the opening / closing threshold resetting means 10. The output from the opening / closing threshold value setting means 6, the output from the opening / closing evaluation means 4, and the opening / closing signal from the opening / closing determination means 7 are further input to the opening / closing threshold value resetting means 10. The threshold value is output to the open / close determination means 7. The open / close determining means 7 determines open / close based on the output of the open / close evaluation means 4 and the output of the open / close threshold resetting means 10.

FIG. 6 is a time chart describing the operation of the third embodiment. The solid line 4a indicates the output of the open / close evaluation means 4, and the dotted line 6a indicates the output of the open / close threshold setting means 6. This is the same as that described in 2. The solid line 7a indicates the output of the opening / closing determination means 7, and the thick solid line 9a indicates the output of the opening / closing threshold control threshold setting means 9 of the present embodiment.
The thick dotted line 10a indicates the output of the opening / closing threshold value resetting means 10.
In this figure, the output 4a of the opening / closing evaluation means 4 is never closer to the closing side than the opening / closing threshold 6a which is the output of the opening / closing threshold setting means 6, that is, a state in which no blink is detected. .

The opening / closing threshold control threshold setting means 9 outputs a value obtained by adding a predetermined value 9b to the output 6a of the opening / closing threshold setting means 6, and in FIG. Threshold). Here, the predetermined value 9b is
The threshold value is set to be smaller than a predetermined value used when generating the opening / closing threshold value 6a from the output of the filter means 5. Next, at time 9
At the time point c, the opening / closing evaluation value 4a is changed to the opening / closing threshold control threshold 9a.
When the opening / closing threshold value is more closed, the opening / closing threshold value resetting means 10 stops outputting the output 6a of the opening / closing threshold value setting means 6 as it is, and while the opening / closing evaluation value 4a is closer to the closing side than the opening / closing threshold control threshold value 9a,
At least during the time 9e, the value of the open / close determination threshold 6a at the time 9c is held, and the open / close determination threshold as indicated by the thick dotted line 10a is output to the open / close determination means 7. Here, the opening / closing determination means 7 makes a closing determination at time 7e when the opening / closing evaluation value 4a is closer to the output 10a of the opening / closing threshold resetting means 10 within the time 9e. In this way, when the opening / closing determination means 7 performs the closing determination, the opening / closing threshold resetting means 10 retains the opening / closing determination threshold 10a at that time during the closing determination, and the opening / closing evaluation value 4a again determines the opening / closing determination. The opening / closing determination threshold value 10a is held until time 7f when the threshold value is on the opening side from the threshold value 10a. When the open / close determination is made by the open / close determination means 7, the open / close threshold value resetting means 10
Outputs the output 6a of the opening / closing threshold setting means 6 as it is.

On the other hand, at time 9f in the figure, it is assumed that the value at the time of opening the eyes is shifted to the closing side in a step-like manner, for example, when the posture of the driver suddenly changes. Also at this time, the opening / closing evaluation value 4a is changed to the opening / closing threshold control threshold 9 at time 9f.
a, the output 10 of the open / close threshold resetting means 10
Although a is also in the holding state, the opening / closing evaluation value 4a does not become the closing side from the held opening / closing threshold value 10a within a predetermined time, so that the opening / closing determination means 7 does not perform the closing determination. further,
When the predetermined time elapses and the time reaches 9 g, the holding is released,
The output of the opening / closing threshold setting means 10 is the same as the output 6a of the opening / closing threshold setting means 6.

In the above description, a method similar to that of the first embodiment has been presented for the operation of the open / close determination means 7 for determining whether the eye is closed or open. However, the purpose of this embodiment is to provide a method for determining whether the eye is closed or open. However, the method for determining whether the eye is open from the closed state is not specified, and the method for determining whether the closed state is changed to the open state may be the method described in the second embodiment. Also, in the fourth embodiment described below, a method for determining whether the eye is closed or open is described. However, there is no problem if the method is combined with the method.

Embodiment 4 In this embodiment, as shown in FIG. 7, a closing determination method according to the third embodiment and a cover threshold value setting means 11 are added to the second embodiment.
Hereinafter, this embodiment will be described with reference to FIG.

Covert threshold setting means 1 according to the fourth embodiment
Reference numeral 1 includes a gross eye determining threshold value setting means and a gross eye shape determining means. The blindfold determination threshold setting means sets a blindfold determination threshold in the eye closing direction based on the open / close determination threshold set by the open / close determination threshold setting means. When the state on the closed side from the open / close determination threshold and the side on the open side from the cover-off determination threshold have continued for a predetermined time, if the open-close evaluation value during the period has a characteristic such as a small change in the value, it is determined that the cover is open. If the eyes are not closed and the eyes are not closed eyes, the eyes are closed.

Further, the closed eye determination threshold value setting means calculates the closed-close value of the open / close evaluation means when the time during the closed eye determination is within a predetermined time range and the open / close determination threshold value when the closed eye is determined. Set an appropriate cover eye determination threshold value for each.

Further, after the closing judgment is made based on the opening / closing evaluation value and the opening / closing judgment threshold value, the opening / closing evaluation value is determined to be in a state where the opening / closing evaluation value is on the open side and the opening side with respect to the opening judgment threshold value. When the time has been maintained, it is determined that the eyes are closed and then the eyes are closed, and when the opening / closing evaluation value is closer to the open side than the closed eyes determination threshold value, the eyes closed determination is ended and the eyes are determined to be open.

Next, the operation of the fourth embodiment will be described. The face opening / closing evaluation value of the face image captured by the camera 1 is output by the image input unit 2, the eye image extracting unit 3, and the opening / closing evaluating unit 4 in the same manner as in the first embodiment. is there. Further, the output of the opening / closing evaluation means 4 is input to the filtering means 5, and the output of the filtering means 5 is input to the opening / closing threshold value setting means 6, and the opening / closing threshold value is set in the same manner as in the first embodiment. The output of the opening / closing threshold setting unit 6 is input to the opening / closing threshold control threshold setting unit 9, and the opening / closing threshold control threshold is input to the opening / closing threshold resetting unit 10. The output from the opening / closing threshold value setting means 6, the output from the opening / closing evaluation means 4, and the opening / closing signal from the opening / closing determination means 7 are further input to the opening / closing threshold value resetting means 10,
From these inputs, an open / close determination threshold is output to the open / close determination means 7. The output of the opening / closing threshold value resetting means 10 and the output of the opening / closing determination means 7 are input to the covert threshold value setting means 11.
Is input to The opening / closing determination means 7 receives the output of the opening / closing evaluation means 4, the output of the opening / closing threshold value resetting means 10, and the output of the blindfold threshold value setting means 11, and determines whether the eye is open or closed according to a predetermined procedure.

FIG. 8A shows the relationship between the degree of opening of the eyes and the evaluation value of opening and closing, and shows an example in which the larger the eyes are, the larger the value is. Next, FIG. 8B schematically shows a state of the eye image when the driver blinks quickly and a state of a change in the output of the opening / closing evaluation means 4 which changes with the eye image. Similarly, FIG. 8 (c) shows a situation where the driver performs a cover-up for a long time by looking at a speedometer, and FIG. 8 (d) shows a case where the driver closes his eyes for a long time, for example, when he becomes sleepy. It shows the situation at the time.

As can be seen from FIGS. 8 (b) to 8 (d), the respective features are as follows. In the case of FIG. By using a filter for the purpose of noise reduction of
The amount of change between the value at the time of opening the eye and the value of the reclosing value is small. Next, in the case of FIG. 8 (c), the upper eyelid falls because the user is looking at the meter and the like, and the eye is kept in the half-open state for a long time. The change of the closed value is small. On the other hand, in the case of FIG. 8D in which the eyes are closed for a long time due to sleepiness, etc., the eyes are completely closed, so that the closing value is larger than that of FIGS. 8B and 8C. Becomes smaller. Here, the output of the opening / closing threshold value resetting means 10 is shown in FIG.
If the setting is made such that a quick blink as shown in (b) can also be detected, the long-lasting closed eyes shown in FIG. 8 (c) are erroneously detected as long-term closed eyes. Conversely, if the setting is made so as not to detect the cover eyes shown in FIG. 8C, the blinking shown in FIG.

Next, the operation of this embodiment will be described with reference to a time chart shown in FIG. FIG. 9 is similar to FIG.
FIG. 7A shows the operation in the case of blinking fast, FIG. 7B shows the operation in the case of blind eyes, and FIG.

In FIG. 9A, the solid line 4a is the output of the opening / closing evaluation means 4, the dotted line 10a is the output of the opening / closing threshold value resetting means 10, and the dotted line 11a is the predetermined value in the closing direction from the opening / closing threshold value 4a. This is a threshold value set at a distance of 11b.

Although the opening / closing evaluation value 4a is closer to the opening / closing threshold value 10a at time 7e, in the present embodiment, the closing judgment is not performed yet. Next, looking at the most closed value, it is shown that the state returns to the open state without falling below the cover threshold 11a. When such a waveform is input to the open / close determination means 7, for example, in the case of a quick blink as shown by a solid line 4a in FIG. 9A, the open / close determination threshold 10a is opened during the predetermined time 11b. This is determined to be a blink, and a close determination is made as far back as time 7e to detect a blink detection as indicated by a solid line 7a.
On the other hand, the opening / closing evaluation value is a predetermined time 1
If the opening / closing operation does not go to the opening side from the opening / closing threshold value 10a during 1b, the eye opening / closing operation does not stop during a predetermined time, so that the amount of change in the movement is measured by, for example, integrating the differential value of the broken line 4a *. Then, when the integrated amount exceeds a predetermined value, the closing determination is started as far back as time 7e as shown in FIG. Further, when the predetermined time 11b ends and the opening / closing evaluation value 4a * is closer to the opening side than the opening / closing threshold 10a, it is determined to be open.

Next, the operation at the time of cover-up shown in FIG. 9B will be described. Also in the case of a blind eye, the opening / closing evaluation value 4a does not become more open than the opening / closing threshold value 10a during the predetermined time 11b, similarly to the earlier blinking described above. However, in the case of the closed eyes, since the half-open state of the eyes continues, the rate of change of the open / close evaluation value 4a is small. Therefore, as described in the case of early blinking, when the amount of change in movement is measured and integrated, the integrated value is not very large, and the integrated value does not exceed a predetermined value. Therefore, it is possible to distinguish between the blinking and the early blinking, and when the blinking is determined, by setting the opening / closing threshold 10a to the closing side from the opening / closing evaluation value 4a at the time when the predetermined time 11b ends, Until the blink occurs, the open / close evaluation value 4a does not become closer to the open side than the open / close threshold 10a, and the risk of erroneously detecting the closed eye is reduced.

Next, the operation at the time of long blinking shown in FIG. 9C will be described. In the case of a long blink, the opening / closing evaluation value 4a is closer to the closing side than the opening / closing threshold value 10a, and is closer to the closing side than the cover threshold value 11a during the predetermined time 11b. That is, since it is possible to detect that the eyes have been completely closed, the closing determination is performed as far back as time 7e, and the closing determination is continued until the opening / closing evaluation value 4a is closer to the open side than the opening / closing threshold 10a.

FIG. 10 is a time chart showing a case in which the player blinks right after blinking.
As shown in the figure, the opening / closing evaluation value 4a is closer to the opening / closing threshold value 10a at time 7e, and is closer to the closing side than the cover threshold value 11a when fully closed. Is always determined to be closed. Next, unlike the above example, when the eyes are opened, the eyes are in a closed state without completely opening the eyes. In the case of the cover, as described above, the open / close evaluation value 4a is closer to the open / close threshold 10a, and thus the close determination continues during the cover in the above-described method,
False detection will result. However, in the present embodiment, for a predetermined time 11c from the time when the cover-opening threshold value 11a is on the open side,
If the state continues on the open side from the cover threshold 11a and on the close side from the open / close threshold 10a, it is determined in the same manner as the above-described cover off determination whether or not the cover is closed, and it is determined that the cover is open. In this case, as shown in FIG.
It is determined to be open at the time of.

In the above description of the embodiment, the judgment output 7a of the opening / closing judgment means 7 is used for both judgment from opening to closing and from closing to opening based on the opening / closing threshold value 10a. As described in the embodiment, the determination may be made using another threshold.

It can be easily guessed that there is an individual difference in the difference between the evaluation value of opening / closing when the eyes are opened and the evaluation value of opening / closing when the eyes are fully closed. Next, a method of absorbing this individual difference will be described. FIG. 11 describes the blinking length and the change in the opening / closing evaluation value. When the blinking length is 7 hours, the difference between the opening and closing evaluation value of the opening / closing evaluation value and the closing value during the eye-closing determination is 4 h. When the time until eye closing becomes longer as 7i, 7j, 7k, 4j, 4
As k increases, equilibrium is reached at a maximum value of 4k. Therefore, in this figure, the amount of change 4k at the time of blinking for time 7j or more corresponds to the amount of change of the output of the open / close evaluation means 4 when the individual is open and when the individual is fully closed. Therefore, this value 4k
If the threshold value is set so as not to be lower than the threshold value, it is possible to prevent the threshold value described above from being closer to the output than the output of the open / close evaluation means 4 when fully closed. Can be set.

[0065]

According to the first aspect of the present invention, an eye image is extracted from the input face image, the eye opening / closing is evaluated from the eye image, and the opening / closing evaluation result is input to the filtering means. Since the threshold value for judging the opening / closing of the subject is set based on the output and the opening / closing of the eyes of the subject is detected, the opening / closing of the eyes can always be correctly determined even if the subject differs between individuals or the imaging conditions change. As described above, it is possible to set the threshold value, and it is possible to stably detect the opening and closing of the eyes for a long time regardless of individual differences.

According to the second aspect of the present invention, based on the output of the open / close evaluation means during the close determination and the open / close threshold at the time of the close determination, the open threshold at the next open determination is determined. Since it is configured to set, even if the change amount of the opening / closing evaluation value at the time of opening / closing is different, if the same shape is used, it can be detected as a blink having the same length.

According to the third aspect of the present invention, the threshold value for opening / closing threshold control is set at a predetermined distance from the opening / closing threshold value to the open side, and the opening / closing evaluation value is closer to the opening side than the threshold value for controlling the opening / closing threshold value and is higher than the opening / closing threshold value. On the side, since the updating of the opening / closing threshold value is prohibited for a predetermined period of time, it is possible to detect a quick blink in which the amount of change in the opening / closing evaluation value is small. Further, if the threshold value does not become closer to the closing side than the opening / closing threshold value within a predetermined time, the update prohibition of the opening / closing threshold value is canceled and returned to the normal state, so that the eye-closed state is not erroneously detected.

According to the fourth aspect of the present invention, the cover threshold is set at a predetermined value close to the open / close threshold, and the open / close evaluation value is on the close side from the previous open / close threshold and on the open side from the previous cover open threshold. Continues for a predetermined time, if the change amount of the opening / closing evaluation value during the predetermined time is small, it is determined that the eyelid is closed and the eye-close determination is not performed, and if the amount of change is large, it is determined that the eye is blinking. The configuration is such that the closing is determined from the time when it is closer to the closing side than the opening / closing threshold, so that the blink can be detected accurately by distinguishing the blink from the blinking eye.

According to the fifth aspect of the present invention, when the blinking of the individual is blinked, the most-closed value of the blinking evaluation value having a predetermined length of time and the blinking threshold value when the blinking eye closed is determined. Since the amount of change in the evaluation value is detected and the threshold value can be set based on the amount of change so that the threshold value is not on the closed side of the individual's most closed value, the optimal threshold value can be set for each individual. Can be.

According to the sixth aspect of the present invention, after it is determined that the eye is closed, the open / close evaluation value continues to be on the open side from the cover eye threshold value and on the close side from the open / close threshold value for a predetermined time, and the open / close evaluation value during the predetermined time period If the change in the value is small, it is determined that the blinking has been performed after the blink, and the opening / closing evaluation value is configured to perform the open determination when the opening / closing evaluation value is closer to the open side than the blindfold threshold. Even in this case, it is possible to detect an accurate blinking length without detecting the blinking as a long blink.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a face image processing device according to a first embodiment of the present invention;

FIG. 2 is a time chart showing the operation of the first embodiment.

FIG. 3 is a block diagram illustrating a configuration of a face image processing device according to a second embodiment;

FIG. 4 is a time chart showing the operation of the second embodiment.

FIG. 5 is a block diagram illustrating a configuration of a face image processing apparatus according to a third embodiment;

FIG. 6 is a time chart showing the operation of the third embodiment.

FIG. 7 is a block diagram illustrating a configuration of a face image processing apparatus according to a fourth embodiment.

FIG. 8 is a diagram showing a relationship between an eye opening / closing state and an eye opening / closing evaluation value.

FIG. 9 is a time chart showing the operation of the fourth embodiment.

FIG. 10 is another time chart showing the operation of the fourth embodiment.

FIG. 11 is a diagram illustrating a relationship between a blinking length and a difference between an open / closed evaluation value and an eye open / closed evaluation value when the eye is closed;

FIG. 12 is an operation flowchart of the face image processing apparatus of the first conventional example.

FIG. 13 is an explanatory diagram of eye open / close determination of the face image processing apparatus of Conventional Example 1.

FIG. 14 is a schematic configuration diagram of a face image processing apparatus of Conventional Example 2.

FIG. 15 is a time chart showing an operation in a second conventional example when an eye opening / closing threshold is set correctly.

FIG. 16 is a time chart showing an operation when the eye open / close threshold is set to be shifted to the open side in Conventional Example 2.

FIG. 17 is a time chart illustrating an operation when the eye opening / closing threshold is set to be shifted to the closing side in Conventional Example 2.

[Explanation of symbols]

1 camera, 2 image input means, 3 eye image extraction means,
4 open / close evaluation means, 5 filter means, 6 open / close threshold setting means, 7 open / close determination means, 8 open determination threshold setting means, 9
Opening / closing threshold control threshold setting means, 10 opening / closing threshold resetting means, 11 closing threshold setting means.

Claims (6)

[Claims] 1. A camera for capturing a face image, an eye image extracting means for extracting an eye image based on a video signal from the camera, and a continuous open / close evaluation value based on the eye image based on the eye image Opening / closing evaluation means for outputting an opening / closing evaluation means, a filter means for removing a high-frequency component of the output of the opening / closing evaluation means, A face image processing apparatus comprising: an eye open / close determining unit that compares an output of the open / close evaluation unit with an output of the open / close determination threshold setting unit to determine whether the eye is open or closed. 2. The method according to claim 1, wherein the open / close threshold is a close determination threshold for determining from open to closed, and an open determination threshold for determining eyes from closed to open is determined based on the close determination threshold and the open / close evaluation value during the close determination. The face image processing apparatus according to claim 1, wherein the value is obtained by the calculation of (1). 3. An opening / closing threshold control threshold setting means for setting at least one opening / closing threshold control threshold at a predetermined distance in the eye opening direction from an opening / closing threshold set at a predetermined distance from the output of the filter means. When the output of the opening / closing evaluation means is closer to the closing side than the opening / closing threshold control threshold, updating of the opening / closing threshold by the opening / closing threshold setting means is prohibited for a predetermined time, and the output of the opening / closing evaluation means is output during the predetermined time. 3. An open / close threshold resetting means for canceling the update prohibition of the open / close threshold and setting a new open / close threshold when the open / close threshold does not become closer to the closed side. The face image processing apparatus according to the above. 4. A closing threshold value setting means for setting a closing threshold at a predetermined value in the closing direction from the opening / closing threshold, wherein the opening / closing threshold is closer to the opening side than the opening / closing threshold and closer to the opening side than the closing threshold. When the state lasts for a predetermined time, the apparatus further comprises a closed-door determining means for evaluating the shape of the time change of the open / closed evaluation value during the time, and when the closed-door determination is made by the closed-eye determining means, the open / close determination not to perform the close determination. The face image processing apparatus according to any one of claims 1 to 3, further comprising means. 5. A predetermined value for setting the close-up threshold value is set to a predetermined value based on a maximum closing value of the opening / closing evaluation unit when the time during which the eye-close determination is performed is within a predetermined time range and an opening / closing determination threshold value when the eye-close is detected. 5. The face image processing apparatus according to claim 4, further comprising: means for setting using a calculation. 6. An open / closed evaluation value, which is determined to be closed by the open / close threshold, the open / close evaluation value is set to be closer to the open side than the open / close threshold by a predetermined value, and the open / closed threshold is determined to be open and closed to open. 6. The face image processing apparatus according to claim 4, wherein when the state on the closing side of the determination threshold continues for a predetermined time, it is determined that the eyes are closed and the opening is determined.