JP6762794B2 - Eyelid opening / closing detection device and eyelid opening / closing detection method - Google Patents

Description

An embodiment of the present invention relates to an eyelid opening / closing detection device and an eyelid opening / closing detection method.

Recently, an increasing number of vehicles such as automobiles are equipped with a camera (hereinafter referred to as an interior camera) for photographing the driver's face in the vehicle interior. The image of the driver's face generated based on the output of the vehicle interior camera can be used, for example, to determine the opening / closing of the driver's eyelids and detect doze.

As a method of determining the opening / closing of the eyelid of the subject from the image captured by the driver as the subject, for example, there is a method of using the distance between the eyebrows and the upper eyelid and the distance between the upper eyelid and the lower eyelid. Further, as another method, there is also a method of detecting the upper eyelid and the lower eyelid using images obtained continuously in time series and determining the opening / closing of the eyelid based on the distance between the upper eyelid and the lower eyelid.

However, in the method using eyebrows, when the subject wears spectacles, the frame of the spectacles and its shadow may be mistaken for eyebrows. In addition, in the method using images obtained continuously in time series, it is necessary to use a high-performance processor in order to process images input at a predetermined frame rate, and it is difficult to use an inexpensive microcomputer or the like. .. Further, in the method using the distance between the upper eyelid and the lower eyelid, an erroneous determination may be made when the subject is facing downward or when the eyelashes of the subject are long.

Japanese Unexamined Patent Publication No. 2008-220424 Japanese Unexamined Patent Publication No. 2008-11302

The present invention has been made in consideration of the above-mentioned circumstances, and is an eyelid opening / closing detection device and an eyelid opening / closing detection method capable of determining the opening / closing of the eyelid of a subject with high accuracy without using continuous images in time series. The purpose is to provide.

The eyelid opening / closing detection device according to the embodiment of the present invention has an initial position detection dictionary from the face image of the subject generated based on the output of the image pickup element that images the face of the subject in order to solve the above-mentioned problems. The first determination result of opening and closing of the eyelid of the subject included in the eye area detected by the initial detection unit using the initial detection unit that detects the eye region using the first detection unit and the first opening / closing determination dictionary. The first discriminator that outputs the eyelid and the second open / close determination dictionary created by using an input image different from the first open / close determination dictionary are used to output the second determination result of opening / closing the eyelid. It is provided with two classifiers and an integrated determination unit that integrally evaluates the determination results of the first classifier and the second classifier and outputs the final determination result of opening and closing of the eyelids.

FIG. 6 is an external view showing an example of a vehicle in which a vehicle interior camera including an eyelid opening / closing detection device according to an embodiment of the present invention is installed. (A) is a side view showing an example of a configuration of a vehicle interior camera, and (b) is a front view. The schematic block diagram which shows the example of the function realized by the processor of the eyelid opening / closing detection device which concerns on this embodiment. FIG. 3 is a flowchart showing a procedure for determining the opening / closing of the eyelid of a subject with high accuracy by using the processor of the eyelid opening / closing detection device shown in FIG. 3 without using continuous images in time series. The figure for demonstrating the initial position detection dictionary used by the initial detection unit, the eye opening position detection dictionary and the eye closing position detection dictionary used by a second classifier. The figure for demonstrating the final judgment method by the integrated judgment part. FIG. 6 is a subroutine flowchart showing an example of a detailed procedure of the eyelid opening / closing determination process executed by the first classifier in step S4 of FIG. The figure for demonstrating the method of obtaining the CoHOG feature quantity. FIG. 6 is a subroutine flowchart showing an example of a detailed procedure of the eyelid opening / closing determination process executed by the second classifier in step S5 of FIG. The functional block diagram which shows one configuration example of the 2nd classifier. The figure for demonstrating the feature amount calculation process by the 2nd classifier. The figure for demonstrating the temporary eye opening position detection process and the temporary eye closing position detection process by a 2nd classifier. FIG. 6 is a subroutine flowchart showing an example of a detailed procedure of the eyelid opening / closing determination process using the edge strength executed by the auxiliary determination unit in step S6 of FIG. The functional block diagram which shows one configuration example of an auxiliary determination part. (A) is an explanatory diagram showing an example of a kernel used for edge intensity image generation, and (b) is an explanatory diagram showing another example of the kernel. FIG. 6 is a subroutine flowchart showing an example of a procedure of eyelid candidate point extraction processing and their pairing processing executed by the group extraction unit in step S604 of FIG. Explanatory drawing which shows an example of how the upper eyelid candidate point and the lower eyelid candidate point are extracted. Explanatory drawing which shows an example of the upper eyelid candidate point and the lower eyelid candidate point extracted on one vertical line. Explanatory drawing which shows an example of the pairing method of the upper eyelid candidate point and the lower eyelid candidate point. Explanatory drawing which shows an example of a grouping result. (A) is a diagram for explaining the average value of the length of the group and the width near the center of gravity of the group, and (b) is the approximate straight line of the upper eyelid candidate point on the inner side of the eye and the approximate straight line of the upper and lower eyelids on the inner side of the eye. The figure for explaining the angle to make. Explanatory drawing which shows an example of the rule of opening / closing judgment using a predetermined feature amount. (A) is a diagram for explaining the first rule shown in FIG. 22, (b) is a diagram for explaining the third rule shown in FIG. 22, and (c) is a diagram for explaining the third rule shown in FIG. 22. The figure for demonstrating.

An embodiment of the eyelid opening / closing detection device and the eyelid opening / closing detection method according to the present invention will be described with reference to the accompanying drawings.

FIG. 1 is an external view showing an example of a vehicle 1 in which a vehicle interior camera 4 including an eyelid opening / closing detection device 10 according to an embodiment of the present invention is installed. FIG. 1 shows an example in which the optical axis of the camera is the z-axis and the horizontal axis is the x-axis. The eyelid opening / closing detection device 10 may detect the opening / closing of the eyelids of the subject based on the face image of the subject, and may not be provided as a part of the camera as long as the face image can be acquired. In the following description, an example will be described in which the eyelid opening / closing detection device 10 is provided as a part of the vehicle interior camera 4.

In this case, as shown in FIG. 1, the vehicle interior camera 4 may be provided at a position where the face F of the driver D sitting in the driver's seat of the vehicle 1 can be photographed. Although FIG. 1 shows an example in which the vehicle interior camera 4 is provided on the upper surface of the steering column cover 2 of the vehicle 1, it may be provided on the upper surface of the dashboard 3, the instrument panel, or the like. More preferably, the vehicle interior camera 4 is provided so that the optical axis of the camera intersects the midline of the driver D's face F so that the driver D's face F can be photographed from the front. It is preferable that the camera optical axis is located on the yz plane passing through the rotation axis of the steering wheel).

FIG. 2A is a side view showing a configuration example of the vehicle interior camera 4, and FIG. 2B is a front view. The vehicle interior camera 4 includes an eyelid opening / closing detection device 10 equipped with a processor, a light source 11, a lens 12, a filter 13, and an image sensor 14.

A plurality of light sources 11 are provided, for example. In the following description, an example will be described in which the light source 11 is a near-infrared light source that emits near-infrared light having a peak wavelength of 880 nm or 940 nm, for example. As the light source 11, a visible light source or an ultraviolet light source may be used instead of the near-infrared light source.

The lens 12 is used to focus on the image sensor 14 and connect the image of the driver D's face F to the image sensor 14. The filter 13 transmits the light emitted by the light source 11. The image sensor 14 has a sensitivity capable of detecting at least the emission wavelength of the light source 11, and is composed of a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 14 is controlled by the eyelid opening / closing detection device 10 to take an image of the subject through the filter 13 to generate image data of the face image of the subject, and gives the image data to the eyelid opening / closing detection device 10.

As the image sensor 14, an image sensor 14 having sensitivity in both the visible light region and the near infrared region may be used. In this case, the filter 13 is configured to be able to be positioned at either a first position for blocking visible light and transmitting near-infrared light and a second position for not blocking visible light, for example. It is good. In the eyelid opening / closing detection technique, it is not preferable that the subject feels dazzling because it induces blinking of the subject. In this regard, by using near-infrared light, a clear face image can be obtained without the subject feeling dazzling even when the illuminance of the face F is insufficient such as at night or in a tunnel.

Subsequently, the configuration and operation of the function realization unit by the processor of the eyelid opening / closing detection device 10 will be described.

FIG. 3 is a schematic block diagram showing an example of a function realized by the processor of the eyelid opening / closing detection device 10 according to the present embodiment.

The eyelid opening / closing detection device 10 is composed of, for example, a processor, RAM, and a storage medium such as a ROM. The processor of the eyelid opening / closing detection device 10 loads the eyelid opening / closing detection program stored in the storage medium such as the ROM and the data necessary for executing this program into the RAM, and continuously in chronological order according to this program. The process of determining the opening / closing of the eyelid of the subject with high accuracy is executed without using the image.

The RAM of the eyelid opening / closing detection device 10 provides a work area for temporarily storing programs and data executed by the processor. The storage medium such as the ROM of the eyelid opening / closing detection device 10 stores, for example, an activation program of the vehicle interior camera 4, an eyelid opening / closing detection program, and various data necessary for executing these programs. A storage medium such as a ROM has a configuration including a recording circuit that can be read by a processor, such as a magnetic or optical recording medium or a semiconductor memory, and is a part of programs and data in these storage media. Alternatively, all may be updated via a network or via a portable storage medium such as an optical disc.

In the present embodiment, the term "processor" refers to, for example, a dedicated or general-purpose CPU (Central Processing Unit), GPU (Graphics Processing Unit), or application specific integrated circuit (ASIC). It is intended to mean a circuit such as a programmable logic device (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and an FPGA). The processor realizes various functions by reading and executing a program stored in a storage medium.

Further, in the present embodiment, an example in which a single processor of the eyelid opening / closing detection device 10 realizes each function has been shown, but a plurality of independent processors are combined to form the eyelid opening / closing detection device 10, and each processor Each function may be realized. When a plurality of processors are provided, storage media for storing programs may be provided individually for each processor, or one storage medium may collectively store programs corresponding to the functions of all processors. Good.

As shown in FIG. 3, the processor of the eyelid opening / closing detection device 10 has at least an image generation unit 21, an initial detection unit 22, and a first classifier 23 by means of an eyelid opening / closing detection program stored in a storage medium such as a ROM. It functions as a second classifier 24, an auxiliary determination unit 25, and an integrated determination unit 26. Each of these function realization units 21-26 is stored in a storage medium in the form of a program.

FIG. 4 is a flowchart showing a procedure for determining the opening / closing of the eyelid of a subject with high accuracy by the processor of the eyelid opening / closing detection device 10 shown in FIG. 3 without using continuous images in time series. In FIG. 4, reference numerals with numbers attached to S indicate each step of the flowchart.

The image sensor 14 is controlled by the image generation unit 21 to image the subject, generates image data of the face image of the subject, and gives the image data to the image generation unit 21 (step S1).

The image generation unit 21 controls the image sensor 14 and generates a face image based on the image data output by the image sensor 14 (step S2).

The initial detection unit 22 is a classifier that performs processing (hereinafter referred to as CoHOG processing) using, for example, a Co-occurrence Histograms of Oriented Gradients (Co-HOG) from the face image of the subject, and is an initial position. The region of the eye is detected using the detection dictionary (step S3).

FIG. 5 is a diagram for explaining the initial position detection dictionary used by the initial detection unit 22, the eye opening position detection dictionary and the eye closing position detection dictionary used by the second classifier 24.

As shown in FIG. 5, the initial position detection dictionary is a dictionary in which the image of the eye is learned as a correct answer (positive data) and the peripheral portion of the eye is used as an incorrect answer (negative data). That is, the initial position detection dictionary is a dictionary in which both the image with the eyelids open and the image with the eyelids closed are learned as correct answers. On the other hand, the eye-opening position detection dictionary and the eye-closing position detection dictionary are created with input images having a higher resolution than the initial position detection dictionary. Further, the eye-opening position detection dictionary is a dictionary in which only the image in which the eyelids are opened is regarded as the correct answer, and the peripheral portion of the eye is learned as the incorrect answer. The eye-closing position detection dictionary is a dictionary in which only the image with the closed eyelids is regarded as the correct answer and the peripheral part of the eye is learned as the incorrect answer.

Therefore, the initial position detection dictionary is more resistant to misalignment than the eye-opening position detection dictionary and the eye-closing position detection dictionary, but is inferior in accuracy. Therefore, the initial position detection dictionary is suitable for roughly detecting the eye area from a wide area such as a face image. On the other hand, the eye-opening position detection dictionary and the eye-closing position detection dictionary are vulnerable to misalignment, but are suitable for detecting the eye region from a narrow region with high accuracy. Therefore, in the present embodiment, first, the initial detection unit 22 uses the initial position detection dictionary to roughly detect the eye region from the face image. Then, with respect to the region including the detected eye region, the eye region is detected with high accuracy by the second classifier 24 using the eye opening position detection dictionary and the eye closing position detection dictionary.

Therefore, in step S3, the initial detection unit 22 detects the face by normalizing the entire image according to the face dictionary. Then, the initial detection unit 22 normalizes the detected face area according to the dictionary size of the eyes, and detects the eye area using the initial position detection dictionary.

In addition, all the dictionaries in this embodiment may learn positive and negative in reverse. On the contrary, when learning is performed, it is only necessary to invert the judgment result based on the likelihood.

By the above steps S1-S3, the eye region can be roughly detected from the face image. Next, regarding the detected eye region, the opening / closing of the eyelids is determined for each of the left eye and the right eye.

The first classifier 23 uses the first open / close determination dictionary to determine the opening / closing of the eyelids for the eye region detected by the initial detection unit 22, and outputs the determination result (hereinafter referred to as the first determination result). (Step S4). The first classifier 23 can be configured by a classifier that has learned correct and incorrect answers, and can be configured by, for example, a CoHOG classifier.

The second classifier 24 cuts out a scanning area including the eye area detected by the initial detection unit 22, and uses the eye opening position detection dictionary and the eye closing position detection dictionary from this scanning area to obtain the eye area with high accuracy. Is detected. Then, the second classifier 24 uses a second open / close determination dictionary created by using an input image different from the first open / close determination dictionary to determine whether the eyelids are open / closed with respect to the eye region detected with high accuracy. Is performed, and the determination result (hereinafter referred to as the second determination result) is output (step S5). The second classifier 24 can be configured by a classifier that has learned correct and incorrect answers, and can be configured by, for example, a CoHOG classifier like the first classifier 23.

The auxiliary determination unit 25 determines whether or not the eyelids are open / closed in the eye region detected by the initial detection unit 22 using a feature amount different from that of both the first classifier 23 and the second classifier 24, and the determination result ( Hereinafter, the third determination result) is output (step S6).

As the feature amount used by the auxiliary determination unit 25, for example, edge strength, circular separation (filter), gradient direction, and the like can be used. In the following description, an example will be described in which the auxiliary determination unit 25 determines whether to open or close the eyelid using the edge strength.

The integrated determination unit 26 integrates and evaluates the determination results of the first classifier 23, the second classifier 24, and the auxiliary determination unit 25, and makes a final determination of opening and closing of the eyelids of the subject (step S7).

FIG. 6 is a diagram for explaining a final determination method by the integrated determination unit 26.

As shown in FIG. 6, the integrated determination unit 26 makes a final determination by, for example, taking a majority vote of the determination results of the first classifier 23, the second classifier 24, and the auxiliary determination unit 25. In addition, since there are three judgment results output in this example, if even one of them includes "undecidable", it becomes impossible to take a majority vote. Therefore, the integrated determination unit 26 determines in the order in which highly accurate determination can be expected, that is, the second determination result of the second classifier 24, the first determination result of the first classifier 23, and the third determination result of the auxiliary determination unit 25. Prioritize in order. Then, if a majority vote cannot be obtained due to the inclusion of "undecidable", the final judgment is made in the order of this priority (see FIG. 6).

When the final judgment is made for both eyes, the series of procedures is completed. On the other hand, if the final determination of one eye has not been made, the process returns to step S4.

By the above procedure, the eyelid opening / closing determination results of a plurality of classifiers can be evaluated in an integrated manner. Therefore, it is possible to determine the opening / closing of the eyelids of the subject with high accuracy without using continuous images in time series.

Next, the eyelid opening / closing determination process by the first classifier 23 will be described in detail.

FIG. 7 is a subroutine flowchart showing an example of a detailed procedure of the eyelid opening / closing determination process executed by the first classifier 23 in step S4 of FIG. Further, FIG. 8 is a diagram for explaining how to obtain the CoHOG feature amount.

Unlike the second classifier 24, the first classifier 23 does not search for the eye area again, and uses the eye area detected by the initial detection unit 22 as it is. Further, the first open / close determination dictionary used by the first classifier 23 for the eyelid open / close determination may shift the position of the same eye image so that the likelihood does not change suddenly even if the eye position shifts. Since it is created using data that has been enlarged or reduced and inflated, it is resistant to misalignment. In addition, the brightness gradient threshold value when creating a dictionary is set low. Therefore, if there is a change in the brightness of the input image, this change is also learned by the first open / close determination dictionary as a feature of the image. Therefore, it is not necessary to perform a luminance correction process such as flattening the luminance histogram for the image in the eye region.

In step S401, the first classifier 23 cuts out the eye region detected by the initial detection unit 22 as it is, and normalizes it again according to the learned eye image size of the first open / close determination dictionary. This is because the size of the normalization is different from that of step S3 in FIG.

Next, in step S402, the first classifier 23 obtains the feature amount from the normalized image. As the feature amount, for example, CoHOG feature amount can be used (see FIG. 8).

Next, in step S403, the first classifier 23 collates the obtained feature amount with the first open / close determination dictionary using an SVM (support vector machine) or the like, and calculates the likelihood (score).

For example, when the first open / close judgment dictionary is a dictionary in which the image data of the closed eyelid is trained as positive and the image data of the open eyelid is trained as negative, when the likelihood is equal to or higher than the first threshold value (step). YES in S404), it is determined that the eyes are closed (step S405). On the other hand, when the likelihood is smaller than the first threshold value, it is determined that the eyes are open. At this time, as shown in FIG. 7, when the likelihood is smaller than the first threshold (NO in step S404) so as to avoid fluttering of the determination result near the threshold, the likelihood is further equal to or less than the second threshold. If (YES in step S406), it is determined that the eyes are open (step S407), while if the likelihood is greater than the second threshold value (NO in step S406), it is determined that the determination is impossible (step S408). ).

In addition, the first classifier 23 outputs the undecidable even when the eye is not detected. Even if the image protrudes when normalized in step S401, undetermination is output.

Next, the eyelid opening / closing determination process by the second classifier 24 will be described in detail.

FIG. 9 is a subroutine flowchart showing an example of a detailed procedure of the eyelid opening / closing determination process executed by the second classifier 24 in step S5 of FIG. Further, FIG. 10 is a functional block diagram showing a configuration example of the second classifier 24. As shown in FIG. 10, the second classifier 24 includes a brightness correction unit 241, a feature amount calculation unit 242, a temporary eye opening position classifier 243, a temporary eye closing position classifier 244, a first open / close classifier 245, and a second open / close identification device. It has a vessel 246 and an open / close determination unit 247.

In step S501, the luminance correction unit 241 performs a luminance correction process for flattening the luminance histogram with respect to the eye region detected by the initial detection unit 22. By performing the brightness correction processing, the robustness against ambient light is enhanced.

FIG. 11 is a diagram for explaining the feature amount calculation process by the second classifier 24.

In step S502, the feature amount calculation unit 242 cuts out a scanning area including the eye area detected by the initial detection unit 22, and normalizes the size according to the dictionary size of the second classifier 24 (FIG. 11). See top).

Next, in step S503, the feature amount calculation unit 242 scans the cut out scanning area while shifting the scanning frame, and obtains the feature amount at each scanning position. For example, if the feature amount is CoHOG feature amount, it can be obtained in the same manner as the method shown in FIG. 8, but a plurality of feature amounts are calculated for the number of scanning positions (see the lower side of FIG. 11).

FIG. 12 is a diagram for explaining a temporary eye opening position detection process and a temporary eye closure position detection process by the second classifier 24.

In step S504, the temporary eye-opening position classifier 243 collates the feature amount of each scanning position with the eye-opening position detection dictionary to obtain the likelihood at each scanning position, and assumes that the eyelids of the subject are open. The temporary eye opening position, which is the position, is detected.

Next, in step S505, the temporary closed eye position classifier 244 collates the feature amount of each scanning position with the eye closing position detection dictionary to obtain the likelihood at each scanning position, and assumes that the eyelids of the subject are closed. The temporary closed eye position, which is the position of the eye, is detected.

As shown in FIG. 5, the eye-opening position detection dictionary was created using a high-definition (high-resolution) open-eye dataset as compared to the initial position detection dictionary, and it was assumed that the eyelids were open. It is used to detect a more accurate position (temporary eye opening position) of the case eye. Similarly, the eye closure position detection dictionary is created using a higher definition (higher resolution), closed eye dataset than the initial position detection dictionary, assuming that the eyelids are closed. , Used to detect a more accurate position (temporary closed eye position).

Further, as a method of detecting the temporarily opened eye position and the temporarily closed eye position, for example, a method of adopting coordinates having a high likelihood when the feature amount and a dictionary are collated, or a method of weighting coordinates with a likelihood of a threshold value or more by a likelihood. A method such as adopting the averaged coordinates can be used.

By the above procedure of steps S501-505, the temporary eye opening position and the temporary eye closing position can be detected with high accuracy.

Subsequently, a process of outputting the second determination result of opening / closing the eyelid is performed using the second opening / closing determination dictionary created by using the input image different from the first opening / closing determination dictionary. The second opening / closing determination dictionary includes an opening / closing determination dictionary for opening eyes and an opening / closing determination dictionary for closing eyes.

In step S506, the first open / close classifier 245 collates with the open / close determination dictionary for eye opening at the temporary eye opening position to obtain the eye opening likelihood. Further, in step S507, the second opening / closing classifier 246 collates with the opening / closing determination dictionary for closing the eyes at the temporary closing position to obtain the likelihood of closing the eyes.

Next, in step S508, the open / close determination unit 247 obtains the final likelihood based on the eye-opening likelihood and the eye-closing likelihood. Then, the opening / closing of the eyelid is determined using this final likelihood. For example, when the opening / closing determination is performed as in FIG. 7, if the final likelihood is equal to or greater than the first threshold value (YES in step S509), it is determined that the eyes are closed (step S510). On the other hand, if the final likelihood is smaller than the first threshold value (NO in step S509), and if the final likelihood is equal to or less than the second threshold value (YES in step S511), it is determined that the eyes are open (NO). In step S512), if the final likelihood is greater than the second threshold value (NO in step S511), it may be determined that the determination is impossible (step S513).

As with the first classifier 23, the second classifier 24 makes a determination impossible even when no eyes are detected or when neither closed eyes nor open eyes are found by high-precision position detection. Output.

Here, the difference between the first classifier 23 and the second classifier 24 will be described.

The first classifier 23 and the second classifier 24 are both classifiers that detect the opening and closing of the eyelids, but the first classifier 23 and the second classifier 24 are different data sets (input images used for creating a dictionary). It is made using.

The first classifier 23 uses inflated data by shifting or scaling the position as a data set used for creating the first open / close determination dictionary, and positively sets the closed eyelid data and opens the closed eyelid data. To be learned as negative. By shifting the position and padding, the input data of the dictionary increases. The positive and negative may be reversed.

On the other hand, the second classifier 24 uses data that is not inflated by shifting the position as a data set used for creating the second open / close determination dictionary. As described with reference to FIGS. 9-12, the second classifier 24 is a temporary eye opening position classifier 243 and a temporary eye classifier 243, which are higher-definition eye classifiers, based on the eye region detected by the initial detection unit 22. The position accuracy is improved by detecting the eye position (temporary eye opening position when it is assumed to be open, temporary eye closing position when it is assumed to be closed) again by using the eye closure position classifier 244. Then, at each of these positions, the first open / close classifier 245 and the second open / close classifier 246, which are eyelid open / close classifiers, perform eyelid open / close determination using their respective eye-opening / closing determination dictionaries and eye-closing / opening / closing determination dictionaries. ..

In the detection of the eye region by the initial detection unit 22, since the initial position detection dictionary in which the closed eyes and the open eyes are mixed is used, the position detection accuracy becomes low. On the other hand, in the method using the second classifier 24, the detection accuracy of the eye position can be improved, so that the reliability of the score when collated with the eyelid opening / closing judgment dictionary is improved, and the reliability of the eyelid opening / closing judgment result is improved. Improves sex.

In addition, the integrated determination unit 26 can perform an integrated evaluation of a plurality of determination results by a plurality of classifiers having different data sets and identification methods, and perform a final eyelid opening / closing determination by a majority vote or the like.

For example, the likelihood of the second classifier 24 changes abruptly if the position shifts even a little. In addition, since the eye position is re-searched, the frame of the spectacles may be detected as a closed eye, and the image may be dragged by a locally confusing image. On the other hand, the first classifier 23 is resistant to misalignment and is less likely to be dragged by this type of locally confusing image.

According to the eyelid opening / closing detection device 10 according to the present embodiment, as shown in FIG. 6, even if the classifier outputs undecidable in a situation where the classifier 1 is not good at it, it is supplemented by another classifier. As a result, the final eyelid opening / closing determination can be performed with high accuracy.

Next, the eyelid opening / closing determination process using the edge strength by the auxiliary determination unit 25 will be described in detail.

FIG. 13 is a subroutine flowchart showing an example of a detailed procedure of the eyelid opening / closing determination process using the edge strength executed by the auxiliary determination unit 25 in step S6 of FIG. Further, FIG. 14 is a functional block diagram showing a configuration example of the auxiliary determination unit 25. As shown in FIG. 14, the auxiliary determination unit 25 includes a brightness correction unit 251, an edge intensity image generation unit 252, a group extraction unit 253, a grouping unit 254, and an open / close determination unit 255.

First, in step S601, the luminance correction unit 251 normalizes the image size and the inclination of the eye region detected by the initial detection unit 22.

Next, in step S602, the luminance correction unit 251 performs a luminance correction process for flattening the luminance histogram with respect to the eye region detected by the initial detection unit 22. By performing the brightness correction processing, the robustness against ambient light is enhanced.

FIG. 15A is an explanatory diagram showing an example of a kernel used for edge intensity image generation, and FIG. 15B is an explanatory diagram showing another example of the kernel.

In step S603, the edge intensity image generation unit 252 generates an edge intensity image by convolving the kernel shown in FIGS. 15A and 15B, for example.

Next, in step S604, the group extraction unit 253 extracts the upper eyelid candidate points and the lower eyelid candidate points, and extracts a set of upper and lower eyelid candidate points from these candidate points.

FIG. 16 is a subroutine flowchart showing an example of the procedure of the eyelid candidate point extraction process and the pairing process of the eyelid candidate point extraction process executed by the set extraction unit 253 in step S604 of FIG. In this procedure, it is utilized that the brightness value of the eyes is lower than the brightness value of the skin around the eyes.

FIG. 17 is an explanatory diagram showing an example of how the upper eyelid candidate point and the lower eyelid candidate point are extracted.

In step S6401, the group extraction unit 253 starts searching the edge intensity image from top to bottom for each vertical line.

When there is a position showing the maximum value of the edge strength (YES in step S6402), the group extraction unit 253 extracts this position as an upper eyelid candidate point and registers the coordinates and edge strength of the position (step S6403). This is because the position showing the maximum value of the edge strength is the position where the brightness value changes from light to dark, and is likely to be the boundary position between the skin and the eye around the eye. If there is a position indicating the minimum value of the edge strength (YES in step S6404), the group extraction unit 253 extracts this position as a lower eyelid candidate point and registers the coordinates and edge strength of the position (step S6405). ). Then, it reaches the bottom of the vertical line currently being searched (step S6406). At this time, if there is no lower eyelid candidate point in the searched line (YES in step S6407), the group extraction unit 253 extracts the position showing the minimum value of the edge strength as the lower eyelid candidate point, and sets the coordinates of this position. Register the edge strength.

FIG. 18 is an explanatory diagram showing an example of upper eyelid candidate points and lower eyelid candidate points extracted on one vertical line. As shown in FIG. 18, if the area of the eyes detected by the initial detection unit 22 includes something other than the eyes, such as the frame of eyeglasses, the upper eyelid candidate point and the lower eyelid candidate point are erroneously detected. Sometimes.

Next, the group extraction unit 253 attempts to pair the upper eyelid candidate points and the lower eyelid candidate points extracted from the searched line.

FIG. 19 is an explanatory diagram showing an example of a pairing method of the upper eyelid candidate point and the lower eyelid candidate point. The group extraction unit 253 sets the closest lower eyelid candidate point on the lower side in the vertical direction as a pair candidate for each of the upper eyelid candidate points extracted by the searched line (see the arrows from top to bottom in step S6409, FIG. 19). ). Further, for each of the lower eyelid candidate points, the closest upper eyelid candidate point on the upper side in the vertical direction is set as a pair candidate (see step S6410, arrow from bottom to top in FIG. 19).

Then, if there are pair candidates that are pair candidates and the difference in the amount of change in the brightness value at each candidate point is equal to or less than the threshold value (YES in step S6411), the pair candidates are regarded as the upper eyelid candidate point and the lower eyelid candidate point. Pair as a pair with (step S6412). The difference in the amount of change in the luminance value at each candidate point is not less than the threshold value, for example, in the example shown in FIG. 17, it means that the absolute value of the difference between ΔA and ΔB is not more than the threshold value. The procedure from step S6401 to S6412 is repeated on all vertical lines (NO in step S6413, step S6414). Note that FIG. 16 shows an example of going to the right by one line in step S6414 to show an example of processing all the lines, but in order to reduce the processing amount, the processing is performed by skipping one line or skipping two lines. You may go. In this case, skip one line or skip two lines in S6414 to move to the right. In the case of skipping one line or skipping two lines, the number of lines to be processed is reduced.

When the eyelid candidate point extraction process and their pairing process are performed on all the processing target lines (YES in step S6413), the process proceeds to step S605 in FIG. 13, and the grouping unit 254 sets the extracted eyelid candidate points. For each of the above, the grouping unit 254 groups the pairs in which the distance between the upper eyelid candidate points and the distance between the lower eyelid candidate points are both within the threshold value so that they belong to the same group.

FIG. 20 is an explanatory diagram showing an example of grouping results. In FIG. 20, candidate points belonging to the same group are shown with the same shape (circle, quadrangle, hexagon).

Next, in step S606, the open / close determination unit 255 determines, for each group, whether or not each group belongs to a pair constituting the upper and lower eyelids, and a predetermined feature amount used for determining whether to open / close the eyelids. Ask. Next, in step S607, the open / close determination unit 255 determines the opening / closing of the eyelids from the group having the highest priority for each group, and proceeds to step S7 of FIG. 4 as soon as the opening / closing determination of the eyelids is made.

FIG. 21A is a diagram for explaining the average value of the length of the group and the width near the center of gravity of the group, and FIG. 21B is an approximation straight line of the upper eyelid candidate point on the inner side of the eye and an approximation of the upper and lower eyelids on the inner side of the eye. It is a figure for demonstrating the angle formed by a straight line.

Further, FIG. 22 is an explanatory diagram showing an example of a rule for opening / closing determination using a predetermined feature amount. 23 (a) is a diagram for explaining the first rule shown in FIG. 22, (b) is a diagram for explaining the third rule shown in FIG. 22, and (c) is a diagram for explaining the third rule shown in FIG. 22. It is a figure for demonstrating the 4th rule shown in. The rule shown in FIG. 22 is based on the number of pairs constituting the group (group length), the average width near the center of gravity of the group, and the slope of an approximate straight line connecting a plurality of upper eyelid candidate points near the inner corner of the eye of the group. This is a rule for determining the opening / closing of the eyelids of a subject based on at least two selected feature quantities.

The predetermined feature quantities obtained by the open / close determination unit 255 include, for example, the length of the group, the width near the center of gravity of the group, the approximate straight line of the upper eyelid on the inner side of the eye, and the angle formed by the approximate straight line of the upper and lower eyelids on the inner side of the eye. Can be mentioned.

The length of the group can be used to determine whether or not each group belongs to a pair that constitutes the upper and lower eyelids. If the length of the group is longer than a predetermined length, there is a high possibility that the group is not a mere false positive but a group to which the pair constituting the upper and lower eyelids belongs. The length of the group may be the number of pairs constituting the group. When the number of pairs is used instead of the length, it is possible to exclude a group having a long group length but a low density of constituent pairs and a high possibility of false detection.

Further, the farther the position of the center of gravity of the group is from the center of the image, the higher the possibility of false detection, and the closer the position of the center of gravity is to the center of the image, the higher the possibility that the group belongs to the pair constituting the upper and lower eyelids.

In addition, the average value of the width of the pair (distance between the upper and lower eyelid candidate points) near the center of gravity of the group in the image (for example, the width of about 20% to 40% of the group length from the center of gravity) can be used for opening / closing determination. it can.

Further, since the approximate straight line of the upper eyelid on the inner side of the eye of the group to which the pair constituting the upper and lower eyelids belongs is downward toward the center of the group when the eyes are closed, it can be used for opening / closing determination.

Further, since the angle formed by the approximate straight line of the upper and lower eyelids on the inner side of the eye naturally changes according to the opening and closing of the eyelid, it can be used for the opening / closing determination.

In addition to these, the open / close determination unit 255 may obtain the maximum value, the average value, the dispersion, the position of the center of gravity of the group, and the like as a predetermined feature amount.

After calculating the feature amount, the open / close determination unit 255 gives a high priority to the group to which the pair constituting the upper and lower eyelids is likely to belong, depending on the length of the group and the position of the center of gravity. Sort the groups in descending order of. At this time, the group with a high possibility of false detection may be filtered out according to the length of the group and the position of the center of gravity.

Then, in the sorted order, it is confirmed whether or not the eyelid opening / closing determination rule shown in FIG. 22, for example, is met. When the rule is met, the judgment process for the eye ends. If all the groups do not meet any of the rules, the open / close determination unit 255 outputs the inability to determine.

The auxiliary determination unit 25 creates an edge intensity image of the eye region detected by the initial detection unit 22, extracts a plurality of pairs of upper and lower eyelid candidate points from the edge intensity, and groups the extracted pairs according to the positional relationship. Then, the opening and closing of the eyelid is determined from the feature quantities of each group such as the length of the group, the width near the center of gravity, and the approximate straight line of the upper eyelid on the inner side of the eye. Therefore, even if the frame of the spectacles or its shadow is reflected, even if the subject is facing downward or the subject has long eyelashes, it is high without using continuous images in chronological order. It is possible to accurately determine the opening and closing of the eyelashes of the subject. Further, in the feature amount calculation process by the auxiliary determination unit 25, the processing load is about a straight line approximation, and it is much easier to determine the opening / closing of the eyelid as compared with the conventional technique of curve fitting the contour of the eyelid. It can be performed.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

4 ... Vehicle interior camera 10 ... Eyelid opening / closing detection device 14 ... Image sensor 21 ... Image generation unit 22 ... Initial detection unit 252 ... Edge strength image generation unit 253 ... Group extraction unit 254 ... Grouping unit 255 ... Open / close determination unit

Claims (11)

An initial detection unit that detects the eye area using an initial position detection dictionary from the face image of the subject generated based on the output of the image sensor that images the face of the subject.
Using the first open / close determination dictionary, a first classifier that outputs the first determination result of opening / closing of the eyelid of the subject included in the eye region detected by the initial detection unit, and
A second classifier that outputs a second determination result of opening / closing of the eyelid using a second opening / closing determination dictionary created by using an input image different from the first opening / closing determination dictionary.
An integrated determination unit that comprehensively evaluates the determination results of the first classifier and the second classifier and outputs the final determination result of opening and closing of the eyelids.
Eyelid opening / closing detection device equipped with. The second classifier is
From the region including the eye region detected by the initial detection unit, the eye region is detected again by using the position detection dictionary generated by using the input image having a higher definition than the initial position detection dictionary. , The second determination result of opening / closing of the eyelid included in the re-detected eye area is output by using the second opening / closing determination dictionary.
The eyelid opening / closing detection device according to claim 1. The position detection dictionary of the second classifier is
The eye opening position detection dictionary for detecting the temporary eye opening position, which is the position of the eyes when the eyelids of the subject are assumed to be open, and the eye positions when the eyelids of the subject are assumed to be closed. Includes an eye closure position detection dictionary for detecting a certain temporary eye closure position,
The second classifier is
From the region including the eye region detected by the initial detection unit, the temporary eye opening position is detected by using the eye opening position detection dictionary, and the temporary eye closing position is detected by using the eye closing position detection dictionary. Then, the second determination result is output by using the second opening / closing determination dictionary for each of the temporary eye opening position and the temporary eye closing position.
The eyelid opening / closing detection device according to claim 2. The first classifier and the second classifier are
Each is a classifier using a co-occurrence histogram in the luminance gradient direction.
The eyelid opening / closing detection device according to any one of claims 1 to 3. The initial position detection dictionary of the initial detection unit is
Generated using images of eyes with open eyes and images of eyes with closed eyes,
The eyelid opening / closing detection device according to any one of claims 1 to 4. Auxiliary determination to output a third determination result of opening / closing of the eyelid included in the eye region detected by the initial detection unit using a feature amount different from that of the first classifier and the second classifier. Department,
The eyelid opening / closing detection device according to any one of claims 1 to 5, further comprising. The auxiliary determination unit
Using the edge strength as the feature amount, the opening / closing of the eyelid is determined from the eye region detected by the initial detection unit based on the edge strength, and the third determination result is generated.
The eyelid opening / closing detection device according to claim 6. The integrated determination unit
The final determination of opening and closing of the eyelid is performed by a majority vote of the determination results of the first classifier, the second classifier, and the auxiliary determination unit.
The eyelid opening / closing detection device according to claim 6 or 7. The image sensor is
Imaging the face of the subject illuminated by near-infrared light,
The eyelid opening / closing detection device according to any one of claims 1 to 8. The eyelid opening / closing detection device according to any one of claims 1 to 9.
With the image sensor
In-vehicle camera equipped with. A step of generating a face image of the subject based on the output of an image sensor that captures the face of the subject, and
A step of detecting an eye area from the face image using an initial position detection dictionary,
Using the first open / close determination dictionary, a step of outputting the first determination result of opening / closing the eyelids of the subject included in the eye area, and
A step of outputting a second determination result of opening / closing of the eyelid using a second opening / closing determination dictionary created by using an input image different from the first opening / closing determination dictionary.
A step of integrating and evaluating the first determination result and the second determination result and outputting the final determination result of opening and closing of the eyelid.
A method for detecting eyelid opening / closing.