JP2018018399A - Eyelid opening/closing detector and eyelid opening/closing detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an eyelid opening/closing detector capable of accurately determining opening/closing of eyelids of a subject without using sequential images in time series and an eyelid opening/closing detection method.SOLUTION: The eyelid opening/closing detector includes: an initial detection part for detecting an area of eyes from a face image of a subject generated on the basis of an output of an imaging device which has picked up the face of the subject; an edge intensity image generation part for generating an edge intensity image from the area of the eyes; a pair extraction part for extracting a plurality of pairs of an upper eyelid candidate point and a lower eyelid candidate point on the basis of the edge intensity image; a grouping part for grouping each of the plurality of pairs in such a manner that a pair adjacent to each other on right and left sides having a closer distance between each other belong to the same group; and an opening/closing determination part for determining opening/closing of the eyelids of the subject on the basis of a feature amount of the group.SELECTED DRAWING: Figure 3

Description

  Embodiments described herein relate generally to a heel open / close detection device and a heel open / close detection method.

  Recently, an increasing number of vehicles such as automobiles are equipped with a camera for photographing a driver's face in a vehicle interior (hereinafter referred to as a vehicle interior camera). The image of the driver's face generated based on the output of the vehicle interior camera can be used, for example, to determine whether the driver's bag is opened or closed and to detect dozing.

  As a method for determining whether the eyelid of the subject is opened or closed from an image obtained by imaging the driver as a subject, for example, there is a method using the distance between the eyebrows and the upper eyelid, and the distance between the upper eyelid and the lower eyelid. As another method, there is a method of detecting upper eyelid and lower eyelid using images obtained continuously in time series and determining whether the eyelid is opened or closed based on the distance between the upper eyelid and the lower eyelid.

  However, in the method using eyebrows, when the subject wears spectacles, the frame of the spectacles and the shade thereof may be mistaken for eyebrows. Further, in the method using images obtained continuously in time series, it is necessary to use a high-performance processor to process images input at a predetermined frame rate, and it is difficult to use an inexpensive microcomputer or the like. . In addition, in the method using the distance between the upper eyelid and the lower eyelid, an erroneous determination may be made when the subject is facing down or the subject's eyelashes are long.

JP 2008-220424 A JP 2008-113902 A

  The present invention has been made in consideration of the above-described circumstances, and the eyelid opening / closing detection device and the eyelid opening / closing detection method capable of performing eyelid opening / closing determination with high accuracy without using time-series continuous images. The purpose is to provide.

  In order to solve the above-described problem, the eyelid open / close detection device according to an embodiment of the present invention detects an eye region from a face image of the subject generated based on an output of an imaging device that images the face of the subject. An initial detection unit that performs an edge intensity image generation unit that generates an edge intensity image from the eye region, and a group extraction unit that extracts a plurality of sets of upper eyelid candidate points and lower eyelid candidate points based on the edge intensity image A grouping unit for grouping each of the plurality of sets so that pairs whose adjacent distances between left and right adjacent groups belong to the same group, and opening / closing the eyelids of the subject based on the feature amount of the group An open / close determination unit for determining

1 is an external view showing an example of a vehicle in which a vehicle interior camera including a heel opening / closing detection device according to an embodiment of the present invention is installed. (A) is a side view which shows one structural example of the camera for vehicle interior, (b) is a front view. The schematic block diagram which shows the example of an implementation | achievement function by the processor of the eyelid opening / closing detection apparatus which concerns on this embodiment. The flowchart which shows the procedure at the time of performing the opening / closing determination of a to-be-photographed object with high precision, without using the time series continuous image by the processor of the eyelid opening / closing detection apparatus shown in FIG. The figure for demonstrating the initial position detection dictionary which an initial detection part uses, the position detection dictionary for eye opening used by a 2nd discriminator, and the position detection dictionary for closed eyes. The figure for demonstrating the final determination method by an integrated determination part. FIG. 5 is a subroutine flowchart showing an example of a detailed procedure of eyelid open / close determination processing executed by the first discriminator in step S4 of FIG. 4. FIG. The figure for demonstrating how to obtain | require a CoHOG feature-value. FIG. 5 is a subroutine flowchart showing an example of a detailed procedure of eyelid open / close determination processing executed by the second discriminator in step S5 of FIG. 4. FIG. The functional block diagram which shows one structural example of a 2nd discriminator. The figure for demonstrating the feature-value calculation process by a 2nd discriminator. The figure for demonstrating the temporary open eye position detection process and temporary closed eye position detection process by a 2nd discriminator. FIG. 5 is a subroutine flowchart illustrating an example of a detailed procedure of a heel open / close determination process using edge strength executed by an auxiliary determination unit in step S <b> 6 of FIG. 4. The functional block diagram which shows one structural example of an auxiliary | assistant determination part. (A) is explanatory drawing which shows an example of the kernel used for edge intensity | strength image generation, (b) is explanatory drawing which shows the other example of a kernel. FIG. 14 is a subroutine flowchart showing an example of a candidate eyelid point extraction process and a pairing process executed by the pair extraction unit in step S604 of FIG. Explanatory drawing which shows an example of a mode that an upper eyelid candidate point and a lower eyelid candidate point are extracted. Explanatory drawing which shows an example of the upper eyelid candidate point and lower eyelid candidate point extracted on 1 vertical line. Explanatory drawing which shows an example of the pairing method of an upper eyelid candidate point and a lower eyelid candidate point. Explanatory drawing which shows an example of a grouping result. (A) is a figure for demonstrating the average value of the length of a group, and the width | variety vicinity of the center of gravity of a group, (b) is an approximate straight line of the upper eyelid candidate point on the eye side, and an approximate straight line of the upper and lower eyelids on the eye side. The figure for demonstrating the angle | corner made. Explanatory drawing which shows an example of the rule of the open / close determination using a predetermined feature-value. (A) is a diagram for explaining the first rule shown in FIG. 22, (b) is a diagram for explaining the third rule shown in FIG. 22, and (c) is a fourth rule shown in FIG. The figure for demonstrating.

  DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of a heel open / close detection device and a heel open / close detection method according to the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is an external view showing an example of a vehicle 1 in which a vehicle interior camera 4 including a heel opening / closing detection device 10 according to an embodiment of the present invention is installed. FIG. 1 shows an example in which the camera optical axis is the z-axis and the horizontal axis is the x-axis. Note that the eyelid opening / closing detection device 10 only needs to detect the eyelid opening / closing of the subject based on the face image of the subject, and may not be provided as a part of the camera as long as the face image can be acquired. In the following description, an example in which the heel opening / closing detection device 10 is provided as a part of the vehicle interior camera 4 will be described.

  In this case, as shown in FIG. 1, the vehicle interior camera 4 may be provided at a position where the face F of the driver D sitting in the driver's seat of the vehicle 1 can be photographed. Although FIG. 1 shows an example in which the vehicle interior camera 4 is provided on the upper surface of the steering column cover 2 of the vehicle 1, the vehicle interior camera 4 may be provided on the upper surface of the dashboard 3 or in the instrument panel. More preferably, the vehicle interior camera 4 is preferably provided so that the camera optical axis intersects the midline of the driver F's face F so that the driver's D's face F can be photographed from the front. It is preferable that the camera optical axis is located on a yz plane passing through the rotation axis of the steering wheel.

  FIG. 2A is a side view showing a configuration example of the vehicle interior camera 4, and FIG. 2B is a front view. The vehicle interior camera 4 includes a bag opening / closing detection device 10 including a processor, a light source 11, a lens 12, a filter 13, and an image sensor 14.

  For example, a plurality of light sources 11 are provided. In the following description, an example in which the light source 11 is a near infrared light source that emits near infrared light having a peak wavelength of, for example, 880 nm or 940 nm will be described. As the light source 11, a visible light source or an ultraviolet light source may be used instead of the near infrared light source.

  The lens 12 is focused on the image sensor 14 and used to connect the image of the face F of the driver D to the image sensor 14. The filter 13 transmits light emitted from the light source 11. The imaging element 14 has a sensitivity capable of detecting at least the emission wavelength of the light source 11, and is configured by a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 14 is controlled by the eyelid opening / closing detection device 10 to image the subject via the filter 13 to generate image data of the face image of the subject, and supplies the image data to the eyelid opening / closing detection device 10.

  Note that the image sensor 14 may have a sensitivity in both the visible light region and the near infrared region. In this case, the filter 13 is configured to be positioned at one of two positions, for example, a first position for blocking visible light and transmitting near-infrared light, and a second position for blocking visible light. Good. In the eyelid opening / closing detection technique, it is not preferable that the subject feels dazzling because it induces blinking of the subject. In this regard, by using near-infrared light, a clear face image can be acquired without the subject feeling dazzling even when the illuminance of the face F is insufficient, such as at night or in a tunnel.

  Next, the configuration and operation of the function realization unit by the processor of the eyelid opening / closing detection device 10 will be described.

  FIG. 3 is a schematic block diagram showing an example of functions realized by the processor of the eyelid opening / closing detection device 10 according to the present embodiment.

  The bag open / close detection device 10 is constituted by a storage medium such as a processor, a RAM, and a ROM, for example. The processor of the eyelid opening / closing detection device 10 loads the eyelid opening / closing detection program stored in a storage medium such as a ROM and data necessary for executing the program into the RAM, and continues in time series according to the program. A process for performing eyelid opening / closing determination of the subject with high accuracy without using the obtained image is executed.

  The RAM of the eyelid opening / closing detection device 10 provides a work area for temporarily storing programs and data executed by the processor. The storage medium including the ROM of the heel opening / closing detection device 10 stores, for example, a startup program for the vehicle interior camera 4, a heel opening / closing detection program, and various data necessary for executing these programs. Note that a storage medium such as a ROM has a configuration including a recording circuit readable by a processor, such as a magnetic or optical recording medium or a semiconductor memory, and a part of programs and data in the storage medium. Alternatively, the whole may be updated via a network or a portable storage medium such as an optical disk.

  In the present embodiment, the term “processor” means, for example, a dedicated or general-purpose CPU (Central Processing Unit), GPU (Graphics Processing Unit), or an application specific integrated circuit (ASIC). It shall mean a circuit such as a programmable logic device (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and an FPGA). The processor implements various functions by reading and executing a program stored in the storage medium.

  In the present embodiment, an example in which a single processor of the eyelid opening / closing detection device 10 realizes each function has been described. However, the eyelid opening / closing detection device 10 is configured by combining a plurality of independent processors, and each processor Each function may be realized. Further, when a plurality of processors are provided, a storage medium for storing the program may be provided for each processor individually, or a single storage medium may store programs corresponding to the functions of all the processors in a lump. Good.

  As shown in FIG. 3, the processor of the eyelid opening / closing detection device 10 uses at least an image generation unit 21, an initial detection unit 22, a first discriminator 23, according to an eyelid opening / closing detection program stored in a storage medium such as a ROM. It functions as the second discriminator 24, the auxiliary determination unit 25, and the integration determination unit 26. Each of these function realization units 21-26 is stored in a storage medium in the form of a program.

  FIG. 4 is a flowchart showing a procedure when the eyelid opening / closing detection apparatus 10 shown in FIG. 3 performs the eyelid opening / closing determination of the subject with high accuracy without using time-series continuous images. In FIG. 4, reference numerals with numbers added to S indicate steps in the flowchart.

  The image sensor 14 is controlled by the image generation unit 21 to image a subject, generates image data of a face image of the subject, and supplies the image data to the image generation unit 21 (step S1).

  The image generation unit 21 controls the image sensor 14 and generates a face image based on the image data output from the image sensor 14 (step S2).

  The initial detection unit 22 is a discriminator that performs processing (hereinafter referred to as CoHOG processing) using, for example, a luminance gradient direction co-occurrence histogram (CoHOG: Co-occurrence Histograms of Oriented Gradients) from the face image of the subject. An eye region is detected using the detection dictionary (step S3).

  FIG. 5 is a diagram for explaining an initial position detection dictionary used by the initial detection unit 22, and an eye opening position detection dictionary and an eye closing position detection dictionary used by the second discriminator 24.

  As shown in FIG. 5, the initial position detection dictionary is a dictionary in which an eye image is learned as a correct answer (positive data) and a peripheral portion of the eye is learned as an incorrect answer (negative data). That is, the initial position detection dictionary is a dictionary in which both the image with the eyelids opened and the image with the eyes closed are learned as correct answers. On the other hand, the position detection dictionary for eye opening and the position detection dictionary for eye closing are created from input images with higher resolution than the initial position detection dictionary. Further, the position detection dictionary for eye opening is a dictionary in which only an image with an eyelid opened is learned as a correct answer and a peripheral part of the eye is learned as an incorrect answer. The closed eye position detection dictionary is a dictionary in which only an image with a closed eyelid is taken as a correct answer and a peripheral part of the eye is learned as an incorrect answer.

  For this reason, the initial position detection dictionary is more resistant to misalignment but less accurate than the position detection dictionary for eye opening and the position detection dictionary for eye closing. For this reason, the initial position detection dictionary is suitable for roughly detecting the eye area from a wide area such as a face image. On the other hand, the eye-opening position detection dictionary and the eye-closing position detection dictionary are suitable for detecting an eye region from a narrow region with high accuracy while being vulnerable to positional deviation. Therefore, in this embodiment, first, the initial detection unit 22 detects an eye area roughly from the face image using the initial position detection dictionary. Then, for the region including the detected eye region, the second discriminator 24 causes the eye region to be detected with high accuracy using the eye opening position detection dictionary and the eye closing position detection dictionary.

  Therefore, in step S3, the initial detection unit 22 detects the face by normalizing the entire image according to the face dictionary. Then, the initial detection unit 22 normalizes the detected face area according to the eye dictionary size, and detects the eye area using the initial position detection dictionary.

  Note that all dictionaries in the present embodiment may learn positive and negative in reverse. Conversely, when learning is performed, it is only necessary to invert the determination result based on the likelihood.

  The eye region can be roughly detected from the face image by the above steps S1 to S3. Next, with respect to the detected eye region, eyelid opening / closing determination is performed for each of the left eye and the right eye.

  The first discriminator 23 uses the first open / close determination dictionary to determine whether to open / close the eyelid for the eye area detected by the initial detection unit 22 and outputs a determination result (hereinafter referred to as a first determination result). (Step S4). The first classifier 23 can be configured by a classifier that has learned a correct answer and an incorrect answer, and can be configured by, for example, a CoHOG classifier.

  The second discriminator 24 cuts out a scanning region including the eye region detected by the initial detection unit 22, and uses the eye opening position detection dictionary and the eye closing position detection dictionary to accurately detect the eye region. Is detected. Then, the second discriminator 24 uses the second open / close determination dictionary created using an input image different from the first open / close determination dictionary to determine whether to open / close the eyelid with respect to the eye area detected with high accuracy. And a determination result (hereinafter referred to as a second determination result) is output (step S5). The second discriminator 24 can be configured by a discriminator that has learned a correct answer and an incorrect answer, and can be configured by, for example, a CoHOG discriminator as in the first discriminator 23.

  The auxiliary determination unit 25 determines whether to open or close the eyelid for the eye area detected by the initial detection unit 22 using a feature amount different from that of each of the first discriminator 23 and the second discriminator 24. Hereinafter, the third determination result is output (step S6).

  As the feature amount used by the auxiliary determination unit 25, for example, edge strength, circular separation (filter), gradient direction, and the like can be used. In the following description, an example in which the auxiliary determination unit 25 performs the eyelid open / close determination using the edge strength will be described.

  The integrated determination unit 26 performs integrated evaluation of the determination results of the first discriminator 23, the second discriminator 24, and the auxiliary determination unit 25, and performs final determination of opening / closing of the eyelid of the subject (step S7).

  FIG. 6 is a diagram for explaining a final determination method by the integrated determination unit 26.

  As shown in FIG. 6, the integrated determination unit 26 performs final determination by taking a majority decision of the determination results of the first discriminator 23, the second discriminator 24, and the auxiliary determination unit 25, for example. In this example, since three determination results are output, it is not possible to take a majority vote if even one of them includes “determination impossible”. Therefore, the integrated determination unit 26 calculates the second determination result of the second discriminator 24, the first determination result of the first discriminator 23, and the third determination result of the auxiliary determination unit 25 in the order in which highly accurate determination can be expected. Prioritize in order. If a majority decision cannot be made due to the inclusion of “impossibility of determination”, final determination is performed in the order of priority (see FIG. 6).

  If the final determination is made for both eyes, the series of procedures is completed. On the other hand, if the final determination of one eye has not been performed, the process returns to step S4.

  With the above procedure, it is possible to integrally evaluate the eyelid open / close determination results of a plurality of discriminators. For this reason, it is possible to determine the eyelid opening / closing of the subject with high accuracy without using time-series continuous images.

  Next, the eyelid open / close determination process by the first discriminator 23 will be described in detail.

  FIG. 7 is a subroutine flowchart showing an example of a detailed procedure of the eyelid open / close determination process executed by the first discriminator 23 in step S4 of FIG. FIG. 8 is a diagram for explaining how to obtain the CoHOG feature amount.

  Unlike the second discriminator 24, the first discriminator 23 uses the eye region detected by the initial detection unit 22 as it is without re-searching the eye region. Further, the first open / close determination dictionary used by the first discriminator 23 for the eyelid open / close determination shifts the position of the same eye image so that the likelihood does not change suddenly even if the eye position shifts. Because it is created using data that has been scaled up or down, it is resistant to misalignment. In addition, the brightness gradient threshold when creating the dictionary is set to be low. Therefore, if there is a change in luminance in the input image, the first open / close determination dictionary learns this change as a feature of the image. For this reason, luminance correction processing such as luminance histogram flattening is unnecessary for the image of the eye region.

  In step S401, the first discriminator 23 cuts out the eye area detected by the initial detection unit 22 as it is and normalizes it again according to the eye image size learned in the first open / close determination dictionary. This is because the normalization size is different from step S3 in FIG.

  Next, in step S402, the first discriminator 23 obtains a feature amount from the normalized image. As the feature amount, for example, a CoHOG feature amount can be used (see FIG. 8).

  Next, in step S403, the first discriminator 23 collates the obtained feature quantity with the first open / close determination dictionary and SVM (support vector machine) or the like, and calculates a likelihood (score).

  For example, when the first open / close determination dictionary is a dictionary learned by using closed eyelid image data as positive and open eyelid image data as negative, if the likelihood is greater than or equal to the first threshold (step It is determined that the eyes are closed (YES in S404) (step S405). On the other hand, when the likelihood is smaller than the first threshold, it is determined that the eyes are open. At this time, as shown in FIG. 7, when the likelihood is smaller than the first threshold value (NO in step S <b> 404) so as to avoid the determination result fluttering near the threshold value, the likelihood is further equal to or smaller than the second threshold value. If so (YES in step S406), it is determined that the eyes are open (step S407), but if the likelihood is greater than the second threshold (NO in step S406), it may be determined that the determination is impossible (step S408). ).

  Note that the first discriminator 23 also outputs a determination impossibility even when no eyes are detected. Inability to determine is also output when the image protrudes when normalized in step S401.

  Next, the eyelid open / close determination process by the second discriminator 24 will be described in detail.

  FIG. 9 is a subroutine flowchart showing an example of a detailed procedure of the eyelid opening / closing determination process executed by the second discriminator 24 in step S5 of FIG. FIG. 10 is a functional block diagram illustrating a configuration example of the second discriminator 24. As shown in FIG. 10, the second discriminator 24 includes a luminance correction unit 241, a feature amount calculation unit 242, a temporary open eye position discriminator 243, a temporary close eye position discriminator 244, a first open / close discriminator 245, and a second open / close discriminator. And an open / close determination unit 247.

  In step S <b> 501, the luminance correction unit 241 performs luminance correction processing for flattening the luminance histogram on the eye area detected by the initial detection unit 22. By performing the brightness correction process, the robustness with respect to the ambient light is increased.

  FIG. 11 is a diagram for explaining the feature amount calculation processing by the second discriminator 24.

  In step S502, the feature amount calculation unit 242 cuts out a scanning region including the eye region detected by the initial detection unit 22, and normalizes the size according to the dictionary size of the second discriminator 24 (FIG. 11). See above).

  Next, in step S503, the feature amount calculation unit 242 performs scanning while shifting the scanning frame in the cut out scanning region, and obtains the feature amount at each scanning position. For example, if the feature amount is a CoHOG feature amount, it is obtained in the same manner as the method shown in FIG. 8, but a plurality of feature amounts are calculated by the number of scanning positions (see the lower side of FIG. 11).

  FIG. 12 is a diagram for explaining the temporary open eye position detection process and the temporary closed eye position detection process by the second discriminator 24.

  In step S504, the provisional eye opening position discriminator 243 collates the feature value of each scanning position with the position detection dictionary for eye opening to obtain the likelihood at each scanning position, and the eye when it is assumed that the eyelid of the subject is open. A temporary open eye position that is a position is detected.

  Next, in step S505, the temporary closed eye position discriminator 244 determines the likelihood at each scanning position by comparing the feature amount of each scanning position with the position detection dictionary for the closed eye, and assumes that the subject's eyelid is closed The temporary closed eye position which is the position of the eye is detected.

  As shown in FIG. 5, it is assumed that the position detection dictionary for eye opening is created using a high-definition (high resolution), open-eye data set compared to the initial position detection dictionary, and the eyelids are open. This is used to detect a higher-precision position (temporary eye opening position) of the case eye. Similarly, the position detection dictionary for closed eyes is created using a high-definition (high-resolution) closed-eye data set compared to the initial position detection dictionary, and the eye is assumed to be closed. It is used to detect a position with higher accuracy (temporary closed eye position).

  In addition, as a method of detecting the temporary open eye position and the temporary closed eye position, for example, a method that employs coordinates having a high likelihood when a feature amount is matched with a dictionary, or a weight having a likelihood equal to or greater than a threshold is weighted with the likelihood. A method of adopting averaged coordinates can be used.

  Through the procedure of steps S501 to 505 described above, the temporarily opened eye position and the temporarily closed eye position can be detected with high accuracy.

  Subsequently, using a second opening / closing determination dictionary created using an input image different from the first opening / closing determination dictionary, processing for outputting a second determination result of opening / closing of the bag is performed. The second opening / closing determination dictionary includes an eye opening / closing determination dictionary and an eye opening / closing determination dictionary.

  In step S506, the first opening / closing discriminator 245 collates with the opening / closing determination dictionary for opening eyes at the temporary opening position to obtain the eye opening likelihood. In step S507, the second open / close discriminator 246 checks the closed eye open / close determination dictionary at the temporary closed eye position to obtain the closed eye likelihood.

  Next, in step S508, the open / close determination unit 247 obtains a final likelihood based on the open eye likelihood and the closed eye likelihood. The final likelihood is used to determine whether to open or close the eyelid. For example, when performing open / close determination as in FIG. 7, if the final likelihood is equal to or greater than the first threshold (YES in step S509), it is determined that the eyes are closed (step S510). On the other hand, if the final likelihood is smaller than the first threshold (NO in step S509), and if the final likelihood is equal to or smaller than the second threshold (YES in step S511), it is determined that the eye is open ( In step S512), if the final likelihood is larger than the second threshold (NO in step S511), it may be determined that the determination is impossible (step S513).

  Similar to the first discriminator 23, the second discriminator 24 makes the determination impossible even when eyes are not detected or when closed eyes or open eyes are not detected by high-precision position detection. Output.

  Here, the difference between the first discriminator 23 and the second discriminator 24 will be described.

  The first discriminator 23 and the second discriminator 24 are both discriminators that detect the opening and closing of the bag, but the first discriminator 23 and the second discriminator 24 are different data sets (input images used for creating a dictionary). It is made using.

  The first discriminator 23 uses, as a data set used to create the first open / close determination dictionary, data that is padded by shifting the position or expanding / reducing the data, and closed heel data is positive and open heel data. To learn as a negative. By shifting the position and padding, the dictionary input data increases. Positive and negative may be reversed.

  On the other hand, the second discriminator 24 uses, as a data set used for creating the second open / close determination dictionary, data that is not subjected to padding with a shifted position. As described with reference to FIGS. 9-12, the second discriminator 24 is based on the eye area detected by the initial detection unit 22, and the provisional eye opening position discriminator 243 and the provisional eye position discriminator 243, which are higher-definition eye discriminators. By using the closed eye position identifier 244, the position accuracy is improved by detecting the eye position again (a temporary open position when it is assumed to be open, and a temporary closed position when it is assumed to be closed). At each of these positions, the first open / close discriminator 245 and the second open / close discriminator 246, which are discriminators for eyelid opening / closing, perform eyelid opening / closing determination using the eye opening / closing determination dictionary and eye opening / closing determination dictionary, respectively. .

  The detection of the eye area by the initial detection unit 22 uses an initial position detection dictionary in which closed eyes and open eyes are mixed, so that the position detection accuracy is lowered. On the other hand, in the method using the second discriminator 24, the detection accuracy of the eye position can be increased, so the reliability of the score when collated with the eyelid opening / closing determination dictionary is improved, and the reliability of the eyelid opening / closing determination result is reliable. Improves.

  Further, the integrated determination unit 26 can perform integrated evaluation of a plurality of determination results by a plurality of discriminators having different data sets and identification methods to be used, and perform final eyelid open / close determination by majority vote or the like.

  For example, the likelihood of the second discriminator 24 changes abruptly when the position is slightly shifted. Further, since the eye position is searched again, the eyeglass frame may be dragged to a locally confusing image, such as detecting the eye as a closed eyeglass frame. On the other hand, the first discriminator 23 is highly resistant to misalignment and is unlikely to be dragged by this type of local misleading image.

  According to the eyelid opening / closing detection device 10 according to the present embodiment, as shown in FIG. 6, even if the classifier outputs an indeterminate judgment in a scene in which one classifier is not good, it is compensated by another classifier. As a result, the final eyelid opening / closing determination can be performed with high accuracy.

  Next, the eyelid opening / closing determination process using the edge strength by the auxiliary determination unit 25 will be described in detail.

  FIG. 13 is a subroutine flowchart illustrating an example of a detailed procedure of the eyelid opening / closing determination process using the edge strength executed by the auxiliary determination unit 25 in step S6 of FIG. FIG. 14 is a functional block diagram illustrating a configuration example of the auxiliary determination unit 25. As illustrated in FIG. 14, the auxiliary determination unit 25 includes a luminance correction unit 251, an edge intensity image generation unit 252, a set extraction unit 253, a grouping unit 254, and an open / close determination unit 255.

  First, in step S <b> 601, the luminance correction unit 251 normalizes the image size and inclination of the eye area detected by the initial detection unit 22.

  Next, in step S <b> 602, the luminance correction unit 251 performs luminance correction processing for flattening the luminance histogram on the eye area detected by the initial detection unit 22. By performing the brightness correction process, the robustness with respect to the ambient light is increased.

  FIG. 15A is an explanatory diagram illustrating an example of a kernel used for generating an edge intensity image, and FIG. 15B is an explanatory diagram illustrating another example of the kernel.

  In step S603, the edge intensity image generation unit 252 generates an edge intensity image by convolving the kernel shown in FIGS. 15A and 15B, for example.

  Next, in step S604, the set extraction unit 253 extracts the upper eyelid candidate point and the lower eyelid candidate point, and extracts a set of upper and lower eyelid candidate points from these candidate points.

  FIG. 16 is a sub-routine flowchart showing an example of the eyelid candidate point extraction process and the pairing process performed by the pair extraction unit 253 in step S604 of FIG. This procedure uses the fact that the luminance value of the eye is lower than the luminance value of the skin around the eye.

  FIG. 17 is an explanatory diagram showing an example of how the upper eyelid candidate point and the lower eyelid candidate point are extracted.

  In step S6401, the set extraction unit 253 starts searching the edge intensity image from top to bottom for each vertical line.

  If there is a position indicating the maximum value of the edge strength (YES in step S6402), the set extraction unit 253 extracts this position as an upper eyelid candidate point, and registers the coordinates of the position and the edge strength (step S6403). This is because the position showing the maximum value of the edge intensity is a position where the luminance value changes from light to dark, and is likely to be a boundary position between the skin around the eyes and the eyes. If there is a position indicating the minimum value of the edge strength (YES in step S6404), the set extraction unit 253 extracts this position as a lower eyelid candidate point and registers the coordinates and edge strength of the position (step S6405). ). Then, it reaches the bottom of the vertical line currently being searched (step S6406). At this time, if there is no lower eyelid candidate point in the searched line (YES in step S6407), the set extraction unit 253 extracts the position indicating the minimum value of the edge strength as the lower eyelid candidate point, and the coordinates of this position and Register edge strength.

  FIG. 18 is an explanatory diagram showing an example of the upper eyelid candidate point and the lower eyelid candidate point extracted on one vertical line. As shown in FIG. 18, if the eye area detected by the initial detection unit 22 includes something other than eyes such as a frame of glasses, the upper eyelid candidate point and the lower eyelid candidate point are erroneously detected. Sometimes.

  Next, the pair extraction unit 253 attempts to pair the upper eyelid candidate point and the lower eyelid candidate point extracted with the searched line.

  FIG. 19 is an explanatory diagram illustrating an example of a pairing method of the upper eyelid candidate point and the lower eyelid candidate point. The pair extraction unit 253 sets, for each of the upper eyelid candidate points extracted from the searched line, the closest lower eyelid candidate point on the lower side in the vertical direction as a pair candidate (step S6409, see arrow from top to bottom in FIG. 19) ). For each lower eyelid candidate point, the closest upper eyelid candidate point on the upper side in the vertical direction is set as a pair candidate (see the arrow from the bottom to the top in FIG. 19).

  If there is a pair candidate that is a pair candidate and the difference in the luminance value variation at each candidate point is equal to or less than the threshold (YES in step S6411), the pair candidate is identified as an upper candidate point and a lower candidate point. Pairing as a pair (step S6412). For example, in the example illustrated in FIG. 17, the difference in luminance value change amount between the candidate points means that the absolute value of the difference between ΔA and ΔB is less than or equal to the threshold. The procedure from step S6401 to S6412 is repeated for all vertical lines (NO in step S6413, step S6414). Note that FIG. 16 shows an example in which all lines are processed, and an example in which the line goes to the right in step S6414 is shown. However, in order to reduce the processing amount, processing is performed by skipping one line or skipping two lines. In this case, in S6414, one line is skipped or two lines are skipped to move right. In the case of skipping one line or skipping two lines, the number of processing target lines is reduced.

  When the eyelid candidate point extraction processing and the pairing processing are performed on all the processing target lines (YES in step S6413), the process proceeds to step S605 in FIG. 13, and the grouping unit 254 extracts the pair of the eyelid candidate points extracted. The grouping unit 254 performs grouping so that pairs in which the distance between the upper eyelid candidate points and the distance between the lower eyelid candidate points are both within the threshold value belong to the same group.

  FIG. 20 is an explanatory diagram illustrating an example of a grouping result. In FIG. 20, candidate points belonging to the same group are indicated by the same shape (circular, square, hexagonal).

  Next, in step S606, the open / close determination unit 255 determines, for each of the groups, a predetermined feature amount used for determining whether each group is a group to which a pair that constitutes the upper and lower eyelids belongs and for determining whether the eyelids are open / closed. Ask. Next, in step S607, the open / close determining unit 255 determines whether to open / close the bag from the group having a high priority for each group, and proceeds to step S7 in FIG.

  FIG. 21A is a diagram for explaining the average value of the length of the group and the width near the center of gravity of the group, and FIG. 21B is an approximation of the approximate straight line of the upper eyelid candidate point and the approximate upper and lower eyelids of the eye side. It is a figure for demonstrating the angle | corner which a straight line makes.

  FIG. 22 is an explanatory diagram showing an example of an open / close determination rule using a predetermined feature amount. FIG. 23 (a) is a diagram for explaining the first rule shown in FIG. 22, (b) is a diagram for explaining the third rule shown in FIG. 22, and (c) is a diagram for explaining FIG. It is a figure for demonstrating the 4th rule shown to. The rule shown in FIG. 22 is based on the number of pairs constituting the group (group length), the average width in the vicinity of the center of gravity of the group, and the slope of the approximate straight line connecting a plurality of upper eyelid candidate points in the vicinity of the group's eye. This is a rule for determining the opening / closing of the eyelid of the subject based on at least two selected feature values.

  Examples of the predetermined feature amount calculated by the open / close determination unit 255 include the length of the group, the width near the center of gravity of the group, the approximate straight line of the upper eyelid of the group, the angle formed by the approximate straight line of the upper and lower eyelids of the eye side, and the like. Can be mentioned.

  The length of the group can be used to determine whether or not each group is a group to which a pair constituting upper and lower sides belongs. If the length of the group is equal to or longer than a predetermined length, the possibility that the group is a group to which the pair that constitutes the upper and lower eyelids belongs rather than a mere erroneous detection. Note that the length of the group may be the number of pairs constituting the group. When the number of pairs is used instead of the length, it is possible to eliminate a group having a long group length but having a low density of pairs and a high possibility of erroneous detection.

  In addition, the possibility of erroneous detection increases as the position of the center of gravity of the group is further away from the center of the image, and the closer to the center of the image, the higher the possibility that the group is a group to which a pair constituting the upper and lower eyelids belongs.

  The average value of the pair widths (distance between upper and lower eyelid candidate points) in the vicinity of the center of gravity of the group in the image (for example, the lateral width of about 20% to 40% of the group length from the center of gravity) can be used for open / close determination. it can.

  Moreover, the approximate straight line of the upper eyelid of the group to which the pair constituting the upper and lower eyelids belongs is downward toward the center of the group when the eyes are closed, and therefore can be used for open / close determination.

  Further, the angle formed by the approximate straight line of the upper and lower eyelids changes naturally according to the opening and closing of the eyelids, so that it can be used for opening / closing determination.

  In addition to these, the open / close determination unit 255 may obtain the maximum value, average value, variance, group centroid position, and the like of the pair width as the predetermined feature amount.

  After calculating the feature amount, the open / close determination unit 255 gives a high priority to the group that is highly likely to be a group to which the pair that constitutes the upper and lower eyelids belongs, based on the group length and the center of gravity position. Sort groups in descending order. At this time, according to the length of the group and the position of the center of gravity, a group with a high possibility of erroneous detection may be filtered out.

  Then, in the sorted order, for example, it is confirmed whether or not it matches the determination rule of the eyelid opening / closing shown in FIG. When the rule is met, the determination process for that eye ends. When all the groups do not match any rule, the open / close determination unit 255 outputs a determination impossible.

  The auxiliary determination unit 25 creates an edge strength image of the eye region detected by the initial detection unit 22, extracts a plurality of pairs of upper and lower eyelid candidate points from the edge strength, and groups the extracted pairs according to the positional relationship. Then, the eyelid opening / closing is determined from the group features such as the length of the group, the width near the center of gravity, and the approximate straight line of the upper eyelid. For this reason, even if the frame of the eyeglasses or its shadow is reflected, the subject is facing down, or the subject's eyelashes are long, it is not necessary to use a continuous image in time series. It is possible to accurately determine whether the subject's eyelid is opened or closed. In addition, the processing load involved in the feature amount calculation processing by the auxiliary determination unit 25 is about linear approximation, and it is very easy to determine the opening / closing of the eyelids compared to the conventional technique in which the contour of the eyelids is curve-fitted. It can be performed.

  In addition, although some embodiment of this invention was described, these embodiment is shown as an example and is not intending limiting the range of invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 4 ... Vehicle interior camera 10 ... Open / close detection device 14 ... Image sensor 21 ... Image generation unit 22 ... Initial detection unit 252 ... Edge intensity image generation unit 253 ... Set extraction unit 254 ... Grouping unit 255 ... Open / close determination unit

Claims (11)

An initial detection unit for detecting an eye region from the face image of the subject generated based on an output of an image sensor that images the face of the subject;
An edge strength image generation unit for generating an edge strength image from the eye region;
A set extraction unit for extracting a plurality of sets of upper eyelid candidate points and lower eyelid candidate points based on the edge intensity image;
A grouping unit that groups each of the plurality of sets such that a set in which a distance between adjacent sets on the left and right is close belongs to the same group;
An open / close determination unit that determines open / close of the eyelid of the subject based on the feature amount of the group;
A hull open / close detection device comprising: The feature amount of the group is
The number of the groups constituting the group, the average distance between the upper eyelid candidate points and the lower eyelid candidate points constituting the group in the vicinity of the center of gravity of the group, and a plurality of upper eyelid candidate points in the vicinity of the head of the group Including at least one of the slopes of the approximate straight lines to be joined,
The wrinkle opening / closing detection device according to claim 1. The set extraction unit
The edge intensity image is searched from top to bottom, and a position where the luminance value changes from light to dark is extracted as an upper eyelid candidate point, and a position where the luminance value changes from dark to light is extracted as a lower eyelid candidate point. A plurality of vertical lines of the edge intensity image.
The wrinkle opening / closing detection apparatus according to claim 1 or 2. The set extraction unit
For each of the upper eyelid candidate points, the closest lower eyelid candidate point on the lower side in the vertical direction is set as a pair candidate, and for each of the lower eyelid candidate points, the closest upper eyelid candidate point on the upper side in the vertical direction is set as a set candidate. A pair of candidates that are mutually paired candidates and whose difference in luminance value variation at each candidate point is equal to or less than a threshold value are extracted as the pair of the upper eyelid candidate point and the lower eyelid candidate point
The wrinkle opening / closing detection device according to claim 3. The grouping unit
Each of the plurality of sets is grouped so that the distances between the upper eyelid candidate points and the distance between the lower eyelid candidate points are both within a threshold value belong to the same group,
The scissor opening / closing detection apparatus of any one of Claims 1 thru | or 4. The grouping unit
Grouping each of the plurality of sets into a plurality of groups;
The opening / closing determination unit
Prioritizing each of the plurality of groups based on at least one of the number of the groups constituting the group and the position in the image of the center of gravity of the group, and opening / closing the subject's eyelids in order from the highest priority group When the determination result of opening or closing comes out, the determination is ended, and when the determination is impossible, the opening / closing determination is repeated in the next priority group.
The scissor opening / closing detection apparatus of any one of Claims 1 thru | or 5. The opening / closing determination unit
(A) The average of the distances between the upper eyelid candidate points and the lower eyelid candidate points constituting the group in the vicinity of the center of gravity of the group is greater than a first width threshold value when the number of the groups constituting the group is equal to or greater than a threshold value. When the approximate straight line that is narrow and connects a plurality of upper eyelid candidate points in the vicinity of the group's eye is within a predetermined angle or downward from the horizontal toward the center of the group from the eye, or (b) constitutes the group The average of the distance between the upper eyelid candidate point and the lower eyelid candidate point constituting the group in the vicinity of the center of gravity of the group is equal to or greater than a threshold value, and the third width threshold value is wider than the first width threshold value. When the approximate straight line connecting the plurality of upper eyelid candidate points in the vicinity of the group's eye is narrow and downward from the eye toward the center of the group, the eyelid of the subject is closed. That the judges,
The heel opening / closing detection device according to claim 6. The opening / closing determination unit
(A) The number of sets constituting the group is equal to or greater than a threshold, and the average distance between the upper eyelid candidate points and the lower eyelid candidate points constituting the group near the center of gravity of the group is greater than the second width threshold value. If it is wide, or (b) the average number of distances between the upper eyelid candidate point and the lower eyelid candidate point constituting the group in the vicinity of the center of gravity of the group is equal to or greater than a threshold value. An angle formed between an approximate straight line connecting a plurality of upper eyelid candidate points and an approximate straight line connecting a plurality of lower eyelid candidate points in the vicinity of the head of the group which is narrower than the fourth width threshold is smaller than the threshold. If there is, it is determined that the eyelid of the subject is open.
The heel opening / closing detection device according to claim 6 or 7. The image sensor is
Image the face of the subject illuminated with near infrared light;
The scissor opening / closing detection apparatus of any one of Claims 1 thru | or 8. A wrinkle opening / closing detection device according to any one of claims 1 to 9,
The imaging element;
Car interior camera equipped with. Generating a face image of the subject based on an output of an imaging device that images the face of the subject;
Detecting an eye region from the face image;
Generating an edge intensity image from the eye region;
Extracting a plurality of sets of upper eyelid candidate points and lower eyelid candidate points based on the edge intensity image;
Grouping each of the plurality of sets such that pairs in which the distance between adjacent sets on the left and right are close to each other belong to the same group;
Determining whether to open or close the eyelid of the subject based on the feature amount of the group;
A wrinkle opening / closing detection method comprising: