JP5349350B2 - Eye opening degree determination device and eye opening degree determination method - Google Patents

Description

  The present invention relates to an eye-opening degree determination device and an eye-opening degree determination method, and more particularly to an eye-opening degree determination apparatus and an eye-opening degree determination method that determine the degree of eye opening of a subject based on a captured image of the eye of the subject.

  2. Description of the Related Art Conventionally, there has been proposed an apparatus that detects a degree of eye opening of a subject based on a photographed image of the eye of the subject, and estimates a living state such as sleepiness or arousal level of the subject based on data relating to the detected degree of eye opening. For example, Patent Document 1 discloses an eye opening degree determination device that can accurately determine the degree of eye opening even when sunlight is reflected on glasses or the like and a part of the upper eyelid cannot be detected. Has been. In the apparatus of Patent Document 1, a virtual upper eyelid curve is estimated using a part of the upper eyelid of the subject, and a virtual eye corner point and a virtual eye top point are estimated based on intersections of a plurality of virtual upper eyelid curves having different eyelid opening degrees. To do. In the apparatus of Patent Literature 1, the degree of eye opening is determined using the estimated virtual eye corner point and virtual eye top point as a reference position. As a result, the positions of the corners of the eyes and the eyes can be estimated, so that the degree of eye opening can be accurately determined even when sunlight is reflected on glasses or the like and a part of the upper eyelid cannot be detected. .

JP 2009-3644 A

  By the way, when the subject is a driver of the vehicle without glasses, the degree of eye opening of the subject is erroneously detected due to a disturbance such as a sudden change of sunlight hitting the driver's face, and the biological state such as sleepiness is determined. May be inaccurate. Here, it is also conceivable to smooth the noise due to the disturbance by using a smoothing filter or the like on the detection signal of the sensor or the like that has detected the eye opening degree. However, only by smoothing the waveform of the eye-opening degree detection signal in the past fixed time, only the high-frequency component of noise can be removed as shown in FIG. 10, and therefore, the low-frequency component of noise due to disturbance cannot be removed.

  The present invention has been made in consideration of such circumstances, and its purpose is to further eliminate the influence of disturbance and to detect the degree of eye opening of a subject with higher accuracy and an eye opening degree determination method. Is to provide.

  The present invention relates to an eye opening degree data acquisition unit that acquires eye opening degree data relating to the degree of eye opening of the subject based on a photographed image of the eye of the subject, and the eye opening degree of the subject based on the eye opening degree data acquired by the eye opening degree data acquisition unit. Eye opening degree determination means, and the eye opening degree determination means selects the eye opening degree data acquired by the eye opening degree data acquisition means according to the difference between the eye opening degrees of the left and right eyes of the subject indicated by the eye opening degree data. It is an eye opening degree determination apparatus which determines a test subject's eye opening degree based on the opened eye degree data.

  According to this configuration, the eye opening degree data acquisition unit that acquires the eye opening degree data of the subject based on the photographed image of the eye of the subject, and the eye opening degree of the subject is determined based on the eye opening degree data acquired by the eye opening degree data acquisition unit In the eye opening degree determination device provided with the eye opening degree determination means, the eye opening degree determination means determines the eye opening degree of the left and right eyes of the subject indicated by the eye opening degree data among the eye opening degree data acquired by the eye opening degree data acquisition means. The eye opening degree of the subject is determined based on the eye opening degree data selected according to the difference. The degree of openness of both eyes is rarely different, and the difference in the degree of openness between both eyes is closely related to the disturbance, so the subject was based on the data on the degree of openness selected according to the difference in the degree of openness of the left and right eyes By detecting the degree of eye opening, it is possible to further eliminate the influence of disturbance and detect the degree of eye opening of the subject with higher accuracy.

  In this case, the eye opening degree determination means converts the eye opening degree data obtained by the eye opening degree data acquisition means into eye opening degree data in which the difference between the eye opening degrees of the left and right eyes of the subject indicated by the eye opening degree data is less than a predetermined threshold value. It is preferable to determine the eye opening degree of the subject based on the above.

  According to this configuration, the eye-opening degree determining means has an eye opening degree in which the difference between the eye opening degrees of the left and right eyes of the subject indicated by the eye-opening degree data is less than a predetermined threshold among the eye-opening degree data acquired by the eye opening degree data acquiring means. The degree of eye opening of the subject is determined based on the degree data. Since the degree of eye opening between both eyes is rarely different, use the eye opening degree data when the difference in the degree of opening between the left and right eyes is less than the predetermined threshold, and the difference in the degree of opening between the left and right eyes exceeds the predetermined threshold. If the eye opening degree data is not used, the eye opening degree of the subject can be detected with higher accuracy by further eliminating the influence of disturbance.

  Further, the eye opening degree determination means has the same eye opening or closing state of the subject's eye indicated by the eye opening degree data in the eye opening degree data acquired by the eye opening degree data acquisition means. It is preferable to determine the eye opening degree of the subject based on the eye opening degree data.

  Since only one of the left and right eyes is rarely open or closed, use the eye-opening degree data when either the left or right eye of the subject is the same. By not using the eye opening degree data when the state is different between the left and right eyes, it is possible to further eliminate the influence of disturbance and detect the eye opening degree of the subject with higher accuracy.

  Alternatively, the eye opening degree determining means is based on the eye opening degree data of the eye having the higher likelihood of the eye opening degree of the left and right eyes of the subject indicated by the eye opening degree data among the eye opening degree data acquired by the eye opening degree data acquiring means. It is preferable to determine the eye opening degree of the subject.

  According to this configuration, the eye opening degree determination means opens the eye of the eye with the higher likelihood of the eye opening degree of the left and right eyes of the subject indicated by the eye opening degree data among the eye opening degree data acquired by the eye opening degree data acquisition means. The degree of eye opening of the subject is determined based on the degree data. Since the degree of eye opening is rarely different, the effect of disturbance is further eliminated by using the eye opening degree data of the eye with the higher likelihood of opening degree in either the left or right eye. The degree of eye opening can be detected with higher accuracy.

  The eye opening degree determination means can estimate the subject's biological state based on the determined degree of eye opening of the subject.

  According to this configuration, the eye opening degree determination means estimates the biological state of the subject based on the determined degree of eye opening of the subject. The degree of eye opening is closely related to the biological state of the subject. In the present invention, the degree of eye opening can be detected with higher accuracy, and thus the biological state of the subject can be detected with higher accuracy.

  In this case, it is preferable that the eye opening degree determination means estimates the subject's sleepiness as the biological state of the subject.

  Even in a biological state, drowsiness is closely related to the degree of eye opening. In the present invention, the degree of eye opening can be detected with higher accuracy, so that the sleepiness of the subject can be detected with higher accuracy.

On the other hand, the present invention, the biological condition detection device, the eye opening degree data acquisition step of acquiring the eye opening degree data about the eye opening degree of the subject based on the captured image of the subject's eye, the biological condition detection device, the eye opening degree data acquisition step The eye opening degree determination step of determining the eye opening degree of the subject based on the eye opening degree data acquired in step, and the eye opening degree determination step is indicated by the eye opening degree data among the eye opening degree data acquired in the eye opening degree data acquisition step. This is an eye opening degree determination method for determining an eye opening degree of a subject based on eye opening degree data selected in accordance with a difference in eye opening degree between the left and right eyes of the subject.

  In this case, the eye opening degree determination step converts the eye opening degree data obtained by the eye opening degree data acquisition step into eye opening degree data in which the difference between the eye opening degrees of the left and right eyes of the subject indicated by the eye opening degree data is less than a predetermined threshold value. It is preferable to determine the eye opening degree of the subject based on the above.

  Further, in the eye opening degree determination step, in the eye opening degree data acquired in the eye opening degree data acquisition step, either the eye opening state or the eye closing state of the subject eye indicated by the eye opening degree data is the same for the left and right eyes of the subject. It is preferable to determine the eye opening degree of the subject based on the eye opening degree data.

  Alternatively, the eye opening degree determination step is based on the eye opening degree data of the eye having the higher likelihood of the eye opening degree of the left and right eyes of the subject indicated by the eye opening degree data among the eye opening degree data acquired in the eye opening degree data acquisition step. It is preferable to determine the eye opening degree of the subject.

  Further, the eye opening degree determination step can estimate the biological state of the subject based on the determined degree of eye opening of the subject.

  In this case, it is preferable that the eye opening degree determination step estimates the sleepiness of the subject as the biological state of the subject.

  According to the eye opening degree determination device and the eye opening degree determination method of the present invention, it is possible to further eliminate the influence of disturbance and detect the eye opening degree of the subject more accurately.

It is a block diagram which shows the structure of the biological condition detection apparatus which concerns on embodiment. It is a flowchart which shows operation | movement of the biological condition detection apparatus which concerns on embodiment. It is a figure which shows the example of the input face image. It is a figure which shows the example of the feature point in the face image of FIG. It is a perspective view which shows the example of a three-dimensional face model. It is a figure which shows the Bezier curve projected centering on the corner of the eye of the eye edge image, and the eye head. It is a graph which shows the relationship between time and an eye opening degree. It is a graph which shows the relationship between time and an eye opening degree. It is a graph which shows the probability distribution of the feature-value in the case where a sleepiness degree is low and the case where it is high. It is a figure which shows the noise component in the waveform of time and the eye-opening degree detected.

  Hereinafter, a living body state detection device according to an embodiment of the present invention will be described with reference to the drawings. The living body state detection device of this embodiment is mounted on a vehicle or the like and used for determining a driver's arousal level or the like. As shown in FIG. 1, the biological state detection device 10 according to the present embodiment includes an image sensor 12, a face position detection unit 14, a face feature point detection unit 16, a face posture estimation unit 18, an eye opening degree detection unit 20, and a false detection section. An estimation unit 22, an eye-closed / open-eye determination unit 24, a blink feature amount calculation unit 26, and a drowsiness estimation unit 28 are provided.

  The image sensor 12 is for capturing a driver's face image. The image sensor 12 is installed on, for example, the upper surface of a column cover of a vehicle and acquires a driver's face image. For example, an image of 30 frames per second can be acquired.

  The face position detection unit 14 is a part for finding out the face position of the driver from the captured image by boosting, neural network, or the like. The face feature point detection unit 16 is a part for finding feature points such as the corner of the driver's face, the corners of the eyes, the center of the nasal cavity, and the mouth edge from the face image by using a neural network or the like. The face posture estimation unit 18 is a part that fits the 3D model of the face to the face feature points detected by the face feature point detection unit 16 and calculates the position and orientation of the driver's face.

  The eye-opening degree detection unit 20 projects a plurality of curves on the edge image around the driver's eye corners and eyes detected by the face feature point detection unit 16, and determines the positions of the upper and lower eyelids from the intensity of the edges on the curves. This is the part to be detected. Furthermore, the eye-opening degree detection unit 20 calculates the eye-opening degree of the driver from the difference between the Y coordinates of the upper and lower eyelid positions.

  The false detection interval estimation unit 22 analyzes the eye opening degree of both eyes of the driver detected by the eye opening degree detection unit 20, and extracts a false detection interval in which the data of the eye opening degree is a time zone due to false detection such as disturbance. It is a part for.

  The closed / open eye determination unit 24 is a part for performing threshold processing on the opening degree of the driver detected by the opening degree detection unit 20 and determining whether the driver's eye is in the open state or the closed state.

  The blink feature value calculation unit 26 is a part for calculating a blink feature value such as a time during which the eyes are closed, which correlates with sleepiness, using data other than the erroneously detected eye opening degree data. The sleepiness estimation unit 28 is a part for estimating the sleepiness level, which is the driver's sleepiness level, using the relationship between the sleepiness level learned in advance and the blink feature amount.

  Hereinafter, the operation of the biological state detection device 10 of the present embodiment will be described. As shown in FIG. 2, the face position detection unit 14 of the biological state detection device 10 scans the entire range of the image by boosting, a neural network, or the like using the driver's face image taken by the image sensor 12 as an input. The driver's face position is found from the captured image (S01). The input face image is, for example, as shown in FIG. Note that the method for detecting the position of the face is not limited to neural network or boosting, and other methods may be used.

  The face feature point detection unit 16 uses a neural network or the like for the range where the face detected in S01 exists, and the right eye head, right eye corner, left eye head, left eye corner, nasal cavity center and The positions of feature points P such as the left and right mouth edges are detected (S2). Note that the method of detecting the feature point P is not limited to the neural network, and other methods may be used.

  The face posture estimation unit 18 fits the 3D face model as shown in FIG. 5 to the feature point P detected in S02, and sets the position and orientation when they are the best match as the face posture at that time (S03). As shown in FIG. 5, the 3D face model can hold the distances in the Xm, Ym, and Zm directions orthogonal to each other from the center of rotation of the head for each feature point P. Note that the estimation of the face orientation may be calculated by other methods.

  The eye opening degree detection unit 20 detects the eye opening degree of the left and right eyes by the following method. In the following description, a method for detecting the degree of opening of the left eye is shown as an example. The same applies to the right eye detection method. (1) As shown in FIG. 6, the eye-opening degree detection unit 20 applies a lateral Sobel filter to the fixed region including the corner of the eye and the head detected in S02, and creates an edge image in which the edge E is emphasized. To do. (2) The eye-opening degree detection unit 20 projects a plurality of curves such as a Bezier curve B on the edge image around the corners and the eyes detected in S02, and the strength of the edge on the Bezier curve B, that is, The pixel value of the edge image is calculated. (3) The eye-opening degree detection unit 20 selects the one having the strongest edge strength among the plurality of Bezier curves B as the upper and lower eyelid curves. The eye opening degree detection unit 20 uses the edge strength at this time as the likelihood (probability) of the data. (4) The eye opening degree detection unit 20 calculates the eye opening degree [pix] from the difference of the Y coordinates at the midpoint of the upper and lower eyelid curves. Note that other methods may be used as a method for detecting the degree of eye opening.

  If the difference in the degree of openness between the left and right eyes is greater than or equal to a certain threshold, the error detection interval estimation unit 22 sets the interval as an error detection interval (S05). This is because the degree of eye opening of both eyes is rarely different. However, when the face direction angle from the image sensor 12 is as large as 30 degrees or more, for example, the false detection interval estimation unit 22 does not set the false detection interval even if the eye opening degree outputs of the left and right eyes are different. Note that a method other than this may be used as a method for estimating the erroneous detection section.

  For example, when the states such as the closed eyes, the open eyes, and the intermediate state of the left and right eyes are different from each other, the erroneous detection interval estimation unit 22 can also set the interval as an erroneous detection interval. This is because the state of the left and right eyes is very different. Alternatively, the false detection interval estimation unit 22 confirms the likelihood (probability) of the output values of the degree of opening of each of the left and right eyes, and if the likelihood of the degree of opening of any eye is low, the likelihood The opening degree of the lower eye may be treated as rejection (undetected), and the subsequent process may be performed based on the opening degree of the eye with the higher likelihood. The likelihood in this case can be calculated according to a probability density distribution model of the degree of eye opening according to various situations obtained by prior learning.

  The closed / open-eye determination unit 24 closes the eyes with the eyes closed and opens the eyes with respect to the waveform of the degree of opening as shown in FIG. Is determined based on a predetermined threshold (S06). The closed / open eye determination unit 24 sets the states other than the closed eyes and the opened eyes to an intermediate state, such as the section p1 in which the eyes are changed from being opened to being closed and the section p2 in which the eyes are being changed from being closed to being opened. Note that a threshold value other than a predetermined value may be used. When the degree of eye opening of both eyes is detected, the eye closing / opening determination unit 24 determines eye closing and opening using the average value of the degree of eye opening of both eyes. The closed / open eye determination unit 24 uses the degree of opening of the left eye when the degree of opening of the right eye is not detected, and uses the degree of opening of the right eye when the degree of opening of the left eye is not detected. Determine closed eyes and open eyes.

  As shown in FIG. 8, the blinking feature amount calculation unit 26 excludes the section determined as an erroneous detection section in S05, for example, as a feature quantity correlated with sleepiness, for example, in a fixed time of 10 seconds. A blink feature amount that is a ratio of the closed eye state indicated by hatching, that is, a ratio of the number of frames in the closed eye state for 10 seconds is calculated (S07). Note that the blink feature amount calculation unit 26 may use a feature amount other than the blink feature amount as a feature amount correlated with sleepiness.

  The sleepiness estimation unit 28 estimates the sleepiness level of the driver using the relationship between the sleepiness degree learned in advance and the blink feature amount (S08). For example, the sleepiness estimation unit 28 is extracted by the statistical distribution (probability density) of the feature amount when the sleepiness degree obtained by the prior learning by offline processing as illustrated in FIG. 9 is high and when the sleepiness degree is low, and by measurement. The probability density is calculated from the feature amount, and the current sleepiness level is estimated from the magnitude relationship. In the example of FIG. 9, the probability density of the feature amount when the sleepiness level is low is calculated. The sleepiness estimation unit 28 may obtain the driver's sleepiness level by a method other than the above.

  According to the present embodiment, the eye opening degree detection unit 20 that acquires data of the eye opening degree of the subject based on the captured image of the eye of the subject, and the eye opening degree of the subject based on the data of the eye opening degree acquired by the eye opening degree detection unit 20 In the living body state detection device 10 including the eye closure / opening determination unit 24 that determines the degree, the eye closure / opening determination unit 24 includes the data of the eye opening degree among the data of the eye opening degree acquired by the eye opening degree detection unit 20. The eye opening degree of the subject is determined based on the eye opening degree data selected by the erroneous detection section estimation unit 22 according to the difference between the eye opening degrees of the left and right eyes of the subject. The degree of eye opening between both eyes is rarely different, and the difference in eye opening between both eyes is closely related to the disturbance, so it is based on the eye opening degree data selected according to the difference in eye opening between the left and right eyes. By detecting the eye opening degree of the subject, it is possible to further eliminate the influence of disturbance and detect the eye opening degree of the subject more accurately.

  In addition, according to the present embodiment, the eye-closed / open-eye determination unit 24 determines that the difference between the eye opening degrees of the right and left eyes of the subject indicated by the eye opening degree data among the eye opening degree data acquired by the eye opening degree detecting unit 20 is the same. The eye opening degree of the subject is determined based on the eye opening degree data that is less than the predetermined threshold. Since the degree of eye opening between both eyes is rarely different, use the data on the degree of opening of the eyes when the difference between the degree of opening of the left and right eyes is less than a predetermined threshold. By not using the data on the degree of eye opening when it exceeds the maximum, it is possible to further eliminate the influence of disturbance and detect the degree of eye opening of the subject with higher accuracy.

  In this embodiment, since only one of the left and right eyes is rarely open or closed, data on the degree of opening when either the open or closed eye state is the same for the left and right eyes of the subject is used. By not using the data on the degree of opening when either the eye opening state or the eye closing state is different between the left and right eyes, it is possible to further eliminate the influence of disturbance and detect the eye opening degree of the subject with higher accuracy.

  Alternatively, in the present embodiment, the eye closing / opening determination unit 24 has a high likelihood of the eye opening degree of the left and right eyes of the subject indicated by the eye opening degree data among the eye opening degree data acquired by the eye opening degree detection unit 20. The eye opening degree of the subject is determined based on the eye opening degree data of the other eye. Since the degree of openness of the eyes is rarely different, subjects using the data on the degree of openness of the eye with the higher likelihood of openness in either of the left and right eyes further eliminates the effects of disturbance The degree of eye opening can be detected with higher accuracy.

  Furthermore, in this embodiment, the sleepiness estimation unit 28 estimates the sleepiness level of the subject based on the determined degree of eye opening of the subject. The degree of eye opening is closely related to the biological state such as the degree of sleepiness of the subject. In this embodiment, the degree of eye opening can be detected with higher accuracy, and thus the biological state such as the degree of sleepiness of the subject can be detected with higher accuracy. It becomes. In particular, drowsiness is closely related to the degree of eye opening even in a biological state, and in this embodiment, the degree of eye opening can be detected with higher accuracy, so that the sleepiness of the subject can be detected with higher accuracy.

  It should be noted that the present invention is not limited to the above-described embodiment, and it is needless to say that various modifications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Living body state detection apparatus, 12 ... Image sensor, 14 ... Face position detection part, 16 ... Face feature point detection part, 18 ... Face posture estimation part, 20 ... Eye opening degree detection part, 22 ... False detection area estimation part, 24 ... eyes closed / open eye determination unit, 26 ... blink feature amount calculation unit, 28 ... drowsiness estimation unit.

Claims (12)

Eye opening degree data acquisition means for acquiring eye opening degree data relating to the eye opening degree of the subject based on a photographed image of the eye of the subject;
Eye opening degree determination means for determining the eye opening degree of the subject based on the eye opening degree data acquired by the eye opening degree data acquisition means;
With
The eye opening degree determination means is selected based on the difference between the eye opening degrees of the left and right eyes of the subject indicated by the eye opening degree data among the eye opening degree data acquired by the eye opening degree data acquisition means. An eye opening degree determination device that determines the eye opening degree of the subject based on degree data.   The eye-opening degree determining means includes a difference between the eye-opening degrees of the left and right eyes of the subject indicated by the eye-opening degree data in the eye-opening degree data acquired by the eye opening degree data acquiring means is less than a predetermined threshold value. The eye opening degree determination apparatus according to claim 1, wherein the eye opening degree of the subject is determined based on eye opening degree data.   The eye-opening degree determination unit is configured to determine whether the eye opening degree or the eye closing state of the subject indicated by the eye opening degree data among the eye opening degree data acquired by the eye opening degree data acquisition unit is the left and right eyes of the subject. The eye opening degree determination device according to claim 1, wherein the eye opening degree of the subject is determined based on the eye opening degree data in the case of the same.   The eye-opening degree determination means is the eye of the eye with the higher likelihood of the eye opening degree of the left and right eyes of the subject indicated by the eye-opening degree data in the eye-opening degree data acquired by the eye opening degree data acquiring means. The eye opening degree determination apparatus according to claim 1, wherein the eye opening degree of the subject is determined based on eye opening degree data. The eye opening degree determination device according to any one of claims 1 to 4 , wherein the eye opening degree determination means estimates the biological state of the subject based on the determined eye opening degree of the subject.   The eye opening degree determination device according to claim 5, wherein the eye opening degree determination means estimates sleepiness of the subject as the biological state of the subject. An eye opening degree data acquisition step in which the biological state detection device acquires eye opening degree data related to the eye opening degree of the subject based on a photographed image of the eye of the subject;
An eye-opening degree determination step in which the biological state detection device determines the eye-opening degree of the subject based on the eye-opening degree data acquired in the eye opening degree data acquisition step;
Including
In the eye opening degree determination step, the eye opening selected according to the difference between the eye opening degrees of the left and right eyes of the subject indicated by the eye opening degree data among the eye opening degree data acquired in the eye opening degree data acquisition step An eye opening degree determination method for determining the eye opening degree of the subject based on degree data.   In the eye opening degree determination step, the difference between the eye opening degrees of the left and right eyes of the subject indicated by the eye opening degree data among the eye opening degree data acquired in the eye opening degree data acquisition step is less than a predetermined threshold value. The eye opening degree determination method according to claim 7, wherein the eye opening degree of the subject is determined based on the eye opening degree data.   In the eye opening degree determination step, among the eye opening degree data acquired in the eye opening degree data acquisition step, one of the eye opening and closing states of the eye of the subject indicated by the eye opening degree data is the left and right eyes of the subject The eye opening degree determination method according to claim 7 or 8, wherein the eye opening degree of the subject is determined based on the eye opening degree data in the case of the same.   In the eye opening degree determination step, the eye of the eye having the higher likelihood of the eye opening degree of the left and right eyes of the subject indicated by the eye opening degree data among the eye opening degree data acquired in the eye opening degree data acquisition step. The eye opening degree determination method according to claim 7, wherein the eye opening degree of the subject is determined based on the eye opening degree data.   The eye opening degree determination method according to any one of claims 7 to 10, wherein the eye opening degree determination step estimates the biological state of the subject based on the determined eye opening degree of the subject.   The eye opening degree determination method according to claim 11, wherein the eye opening degree determination step estimates drowsiness of the subject as a biological state of the subject.

Images