JP2008246004A - Pupil detection method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately detect the corneal reflection and the image of a facial region without depending on the optical environment in the periphery, and to improve the accuracy in detection of a pupil by preventing the irregularlity of the level of brightness in other areas than the pupil region in a difference image. <P>SOLUTION: The pupil detection method comprises: a fist step of transmitting the charge from a photoelectric conversion element to a first charge transmission part of an image sensor after the image sensor is exposed to the light in a first period while a subject is irradiated with a light source 21 by using image sensors C1 and C2; a second step of transmitting the charge from the photoelectric conversion element to a second charge transmission part after the image sensor is exposed to light while the light source is turned off in a second period continued from the first period; a third step of outputting the charge transmitted to the first and second charge transmission parts by using a shift register; and a fourth step of detecting the pupil by executing an image differencing process after obtaining a bright pupil image, an unilluminated image and a dark pupil image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a pupil detection method.

  The pupil detection technique is useful for pupil mouse, gaze detection, face direction detection, drowsiness detection, and the like, and has been proposed as various techniques so far. In those applications, the reason for using the pupil detection technique is that the pupil can be detected with high robustness and high accuracy by irradiating the face with near infrared rays. Specifically, the light source for obtaining the bright pupil image and the light source for obtaining the dark pupil image are alternately turned on in synchronization with the video frame, and the background is obtained by subtracting the light pupil image and the dark pupil image obtained alternately. This is because the pupil can be easily detected by extinction and only the pupil is highlighted.

  When the pupil is detected, the bright pupil image and the dark pupil image are obtained alternately, and the difference between the images is used to detect the pupil after highlighting the pupil. The pupil detection is performed while the surrounding light environment changes. Is effective and promising. To set the light source, arrange the light source close to the camera opening to obtain the bright pupil image, and arrange the LEDs at a position away from the camera opening to obtain the dark pupil image, Prepare a method of alternately lighting in synchronization with the field (see Patent Document 1 below) and two types of light sources with different amounts of retinal reflection, and lighting them alternately in synchronization with the video field. There is a method of detecting the pupil from the difference image (see Non-Patent Document 1 below).

As a method that can be easily realized so far, the inventors have obtained a bright pupil image and a dark pupil image in the odd field and the even field using an interlace camera such as the NTSC system, and those images are stored in a personal computer. Since taking in, difference and pupil detection have been carried out. Further, corneal reflection has been detected from a bright pupil image or a dark pupil image. Here, since there is a 1/60 second gap in the acquisition time, if the face moves during that time (between fields) and the pupil is displaced, the effect of the image difference is diminished and the detection of the pupil is difficult. Therefore, a method of compensating for this by image processing has also been proposed (Non-Patent Document 2 and Non-Patent Document 3).
JP 2005-182247 A Aya Nakajima, Yoshinobu Ebizawa, Junji Kombe, “Pupil detection with two-wavelength light source”, Transactions of the Institute of Image Information and Television Engineers, Vol.60, No. 12, pp.130-136 Showa Tonishi, Shinji Nakabe, Yoshinobu Ebizawa, “High-precision pupil position detection using corneal reflection”, 2006 Annual Conference of the Institute of Image Information and Television Engineers, 20-3, 2006.9.1 Yoshinobu Ebisawa, Shinji Nakabe, Shinichi Shibata, “Face Direction Detection Device Based on Pupil and Nostril Detection”, Proc. Of the 12th Image Sensing Symposium, pp.358-363, 2006.6.9

  However, the conventional techniques as described above may have the following disadvantages. That is, when the face moves, the background portion does not disappear completely, and the pupil is difficult to detect. In particular, when the surroundings are bright, the pupil becomes smaller, so the amount of near-infrared light incident on the pupil is reduced, and since the image is not saturated, AGC (Automatic Gain Control) is used, resulting in a differential image. Since the level of the pupil part at becomes low, the pupil is difficult to detect. Even in the method of compensating for the image difference, the effect depends strongly on the detection accuracy of the nostril and corneal reflection, and it is difficult to achieve a fundamental solution.

  In addition, since the light sources for obtaining the bright pupil and the dark pupil are generally mounted at different positions, even if they are the same, it is difficult to make the irradiation powers of both light sources the same in the entire image. In other words, due to the difference in the position of the light source, a level difference appears in the image other than the pupil portion in the bright pupil image and the dark pupil image. Therefore, if the pupil is dark in the difference image, pupil detection is performed by this level difference. It becomes difficult. Also, the difference in the position of the light source appears as a positional shift in the glasses, and if the difference is simply made, it will not cancel each other, and the reflection of the glasses tends to cause erroneous detection of pupil detection. The spectacle reflection reflects the front and back surfaces of the ophthalmic lens several times, tends to be a spot whose luminance level is not so different from the pupil, and may be difficult to distinguish from the pupil. In addition, since the edge portion of the spectacle frame or the lens is an acute angle portion, the influence due to the difference in the position of the two light sources is likely to be noticeable, and an image is likely to remain even if the difference is made. Therefore, even if the head does not move, when the background remains and the pupil level is low, pupil detection is difficult. Such a situation occurs similarly when the illumination intensity of both light sources is not balanced.

  In addition, when the eyes are closed and the pupil cannot be detected, and when it is necessary to distinguish whether the face is out of the camera frame and the pupil cannot be detected, it is difficult to distinguish simply by detecting the pupil. Accordingly, the approximate position of the face should be detected. However, since the camera that detects the pupil is a black and white camera, it is difficult to detect the face area.

  Furthermore, the cornea reflection image that is generated when the light source is reflected by the cornea is used for line-of-sight detection, etc., but when a bright object such as the sun is reflected in the cornea or in the spectacle lens, Corneal reflection is difficult to detect.

  In addition, it is conceivable that two cameras are driven synchronously, light sources of two wavelengths are turned on simultaneously, wavelength separation is performed using a dichroic mirror, and a bright pupil image and a dark pupil image are obtained with each camera. In the method, since both images can be captured at the same time, there is no problem of the shift between the images. There is almost no problem with spectacle reflection because the positional deviation of the light source of two wavelengths is small. On the other hand, when the AGC function of the camera is used, there is a problem in that the difference effect is not achieved well because the gain of each camera changes arbitrarily.

  Further, a method of attaching a band pass filter having two wavelengths having different retinal reflection amounts as the center wavelength for each line of the image sensor and simultaneously turning on the light sources of two wavelengths to keep the simultaneity can be considered. In this case, a position where the bright pupil image and the dark pupil image are shifted due to a physical shift of the line is photographed, and eyeglass frame reflection or the like tends to remain in the difference image, which makes it difficult to detect the pupil.

  Therefore, the present invention has been made in view of such a problem, and can accurately detect a corneal reflection and a face region image regardless of the surrounding light environment, and can also detect unevenness in luminance levels other than the pupil portion in the difference image. An object of the present invention is to provide a pupil detection method capable of increasing the accuracy of pupil detection by suppressing the pupil.

  In order to solve the above problems, the pupil detection method of the present invention is a pupil detection method for detecting the pupil of the subject, using an image sensor having one photoelectric conversion element corresponding to one pixel, In a state in which the subject is irradiated with a light source for obtaining a bright pupil image or a dark pupil image in the first period, after the image sensor is exposed in the first period, the charge accumulated by the exposure is transferred from the photoelectric conversion element to the photoelectric conversion element. The first step of transferring to the first charge transfer portion of the image sensor corresponding to the photoelectric conversion element, and the image sensor in the second period with the light source turned off in the second period continuous to the first period A second step of transferring the charge accumulated by the exposure from the photoelectric conversion element to a second charge transfer unit different from the first charge transfer unit corresponding to the photoelectric conversion element; 2 charge transfer part A third step of outputting the transmitted charge using a shift register connected to the first and second charge transfer units; and a bright pupil image, an unilluminated image, and a dark pupil image based on the output of the shift register And a fourth step of detecting the pupil by performing image difference processing after obtaining.

  In the first step, the first period is subdivided into a plurality of periods repeated a predetermined number of times in a predetermined cycle, and charges are transferred to the first charge transfer unit for each of the plurality of periods. In the second step, The second period is subdivided into a plurality of periods repeated a predetermined number of times in a predetermined period, and charges are transferred to the second charge transfer unit for each of the plurality of periods. In the third step, the first period and It is preferable to output the charges transferred to the first and second charge transfer units after the predetermined number of repetitions of the second period are completed.

  In the fourth step, corneal reflection is detected from a difference image obtained by subtracting an unilluminated image from a bright pupil image or a difference image obtained by subtracting an unilluminated image from a dark pupil image. It is also preferable to do.

  In the fourth step, the cornea is obtained by multiplying the difference image obtained by image difference of the non-illuminated image from the bright pupil image and the difference image obtained by image difference of the non-illuminated image from the dark pupil image. It is also preferable to detect reflection.

  Further, in the fourth step, a post-difference bright pupil image obtained by subtracting the non-illuminated image from the bright pupil image and a post-difference dark pupil image obtained by subtracting the non-illuminated image from the dark pupil image. It is also preferable to detect the pupil from a re-difference image obtained by subtracting the two images.

  A re-difference image is obtained by correcting the luminance balance between the post-difference bright pupil image and the post-difference dark pupil image in a partial area of the image and then subtracting the post-difference dark pupil image from the post-difference bright pupil image. It is also preferable to be a difference image after luminance balance correction.

  The difference image after luminance balance correction is preferably a difference image after correction for luminance balance for spectacle reflection obtained by subtracting the dark pupil image after difference from the bright pupil image after difference in consideration of spectacle reflection.

  Also, in the fourth step, in order to obtain a difference image after correcting the luminance balance for spectacle reflection, a coefficient for balancing the luminance is obtained by excluding the spectacle reflection region in the post-difference bright pupil image and the post-difference dark pupil image. It is also preferable.

  It is also preferable to further adjust the light emission amount of the light source by estimating the ambient light intensity from the ratio of the luminance level of the post-difference bright pupil image or the post-difference dark pupil image and the non-illuminated image.

  Further, in the fourth step, the object is detected by a template matching method using a predetermined target person's part as a tracking feature point or a wire frame method based on the template matching method from the post-difference bright pupil image and the post-difference dark pupil image. It is also preferable to detect a person's feature point.

  It is also preferable to detect the pupil from the re-difference image by specifying the position of the eye of the subject by the template matching method or the wire frame method based on the template matching method.

  According to the pupil detection method of the present invention, it is possible to accurately detect a corneal reflection and a face area image regardless of the surrounding light environment, and to suppress unevenness in luminance levels other than the pupil portion in the difference image, thereby improving the accuracy of pupil detection. Can be increased.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of a pupil detection method and a pupil detection device according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant description is omitted.

(First embodiment)
The present embodiment relates to a method and apparatus for highly robustly detecting facial feature points such as pupils, corneal reflections, eyes and eyes, and the like with a video camera, and particularly when the surroundings are bright and the pupils are small. It is related with the technique which enables it to detect. FIG. 1 is an optical path diagram showing an arrangement example of the pupil detection device according to the first embodiment of the present invention. The objective lens 19, the lens 18, and the mirror M1 form an optical path forming unit. The first camera includes a bandpass filter F1 having a center wavelength near the short wavelength (850 nm), the lens 13, and the image sensor C1, and the second camera has a center wavelength near the long wavelength (950 nm). The bandpass filter F2, the lens 14, and the image sensor C2 are included. The mirror M1 is a half mirror or a dichroic mirror, and is arranged on the optical axes of the two cameras, thereby allowing the optical axes of the two cameras to substantially coincide with each other. An opening common to the two cameras is formed on the front surface of the lens 19, and a ring-shaped light source 21 is attached in the vicinity of the opening so as to surround the opening.

FIG. 2 shows a plan view of the light source 21 of FIG. 1 viewed from the outside. The light source 21 is for irradiating illumination light toward the subject's face, and has a structure in which a plurality of two types of light emitting elements 23 a and 23 b are embedded in a ring-shaped pedestal portion 22. The pedestal 22 of the light source 21 is attached so as to be positioned outside the opening 24 along the edge of the opening 24. The light emitting element 23 a is a semiconductor light emitting element (LED) having a center wavelength of output light of 850 nm, and is arranged in a ring shape at equal intervals along the edge of the opening 24 on the pedestal portion 22. The light emitting element 23b is a semiconductor light emitting element having a center wavelength of output light of 950 nm, and is arranged in a double ring shape at equal intervals on the pedestal portion 22 adjacent to the outside of the light emitting element 23a. It is provided with more than the number. At this time, the number of light emitting elements 23a and 23b and the supply current are set to appropriate numbers and values so that the illuminance on the face of the subject who is the subject of photographing is the same. Further, the light emitting elements 23a and 23b can be independently controlled in light emission timing by a control signal from a camera control device (not shown).

Light of two types of wavelength components from the light source 21 is reflected toward the subject's face in a state in which the common optical axis is kept substantially coincident with the optical axis of the camera in the opening of the camera. That is, when viewed from the face, the light from the light source 21 is irradiated from within the opening including the common optical axis of the camera. The face image including the bright pupil image and the face image including the dark pupil image by the illumination are separated by the mirror M 1 after passing through the lens 19 and the lens 18. Then, the light including the short wavelength component passes through the band pass filter F1 and is imaged on the short wavelength image sensor C1 by the lens 13. On the other hand, the light including the long wavelength component reflected and separated by the mirror M1 is
The light passes through the bandpass filter F2 and is imaged on the long wavelength image sensor C2 by the lens 14.

  Here, the configuration and function of the image sensors C1 and C2 will be described below. FIG. 3 is a timing chart showing the timing of camera shutter, charge transfer, and luminance level output, and FIGS. 4 to 7 are diagrams showing a schematic configuration of the image sensor.

  In the image sensors C1 and C2, as shown in FIG. 4, photoelectric conversion elements and CCD (charge coupled device) units (charge transfer units) are two-dimensionally arranged, and a shift register of one column is arranged at the bottom row of the CCD units. Connected and configured. Specifically, the CCD unit includes a photoelectric conversion element that is two-dimensionally arranged and accumulates charges for each pixel, and a plurality of CCD units that are arranged in a line corresponding to each line of the photoelectric conversion element. The photoelectric conversion elements of the corresponding pixels are connected to the odd-numbered units in each line. The even-numbered CCD units in each line are provided corresponding to the same photoelectric conversion elements as the preceding CCD unit, and charges corresponding to the photoelectric conversion elements are transferred from the preceding unit. Further, one shift register is connected to each CCD unit in the last row of each line. The image sensors C1 and C2 are configured to be controlled as follows. That is, in the image sensors C1 and C2, the shutter open period is shortened (for example, 100 μsec), and two frame images are taken continuously (FIG. 3). During the period (a) when the first shutter is open, the light source 21 is turned on and the charge from the photoelectric conversion element is transferred to the corresponding CCD unit (FIG. 4). During the next shutter opening period (b), the light source is not turned on, and charges are transferred from the same photoelectric conversion element to the same CCD unit. Between these two exposures and charge transfer, charge is transferred down by one unit in the CCD row. Thereafter, each time the same CCD row is transferred in one unit in the lower direction, the last (bottom) shift register row is transferred in the right direction, and one line of image information is output. As a result, the image obtained in the period (a) and the image on the same line obtained in the period (b) are output alternately. Thus, since image information on the same line can be obtained even with image information for two lines, it is not difficult to detect a pupil due to a line shift unlike NTSC. Image difference may be performed immediately after these outputs. Thereby, the luminance level component due to the ambient light is removed, and only the luminance component due to the light of the light source can be extracted. In this embodiment, the charge is transferred from one photoelectric conversion element to one CCD unit (FIG. 5), but the charge obtained in the period (a) in each of the separate CCD units arranged vertically. And the charge obtained during the period (b) may be transferred (FIG. 6).

  By preparing two sets of cameras C1 and C2 having such an image sensor and activating them in synchronization with the light source 21, the light sources of the respective wavelengths are turned on as described above in each camera. An image of the light source using only the light source is obtained. If wavelength separation is not possible here, the synchronization of both cameras may be shifted so that the exposure times do not overlap. In particular, with one camera, if the light source 21 is not turned on at the timing (a) but the light source 21 is turned on at the timing (b), the amount of delay in synchronization is only about 100 μsec in this example. Mu

  As described above, in the conventional device using two cameras, even if the light intensity of the two light sources is balanced at a certain ambient brightness, if the ambient brightness changes, the AGC action Will have different gains. For this reason, the luminance level of each light source is also different, and the background portion cannot be made zero even if the luminance level components of only the light sources of the respective wavelengths are differentiated. Even if AGC is not used, the same thing occurs if the wavelength component contained in the ambient light changes (which differs greatly between a light bulb and a fluorescent lamp).

To solve this, the image obtained when the light source of the camera that produces the bright pupil image is turned on is Ib (x, y), and the image obtained when the light source is turned off is I01 (x , y). Also, the image obtained when the light source of the camera that produces the dark pupil image is turned on is Id (x, y), and the image obtained when the light source is turned off is I02 (x, y) . Then, distributions I1 (x, y) and I2 (x, y) of image luminance levels generated only by light from the light source in each camera are represented by the following equations (1) and (2):
I1 (x, y) = Ib (x, y) -I01 (x, y)… (1)
I2 (x, y) = Id (x, y) -I02 (x, y)… (2)
It can be calculated by As described above, since the balance of I1 (x, y) and I2 (x, y) generally does not match, the luminance balance is corrected as follows. However, since these are images having a similar luminance distribution, they may be converted so that the average and standard deviation of the luminance level histogram are substantially the same. After this histogram equalization processing is performed on both, the subsequent processing may be continued. Briefly, an average value <I1> of I1 (x, y) and an average value <I2> of I2 (x, y) are calculated in advance, and the following formula (3);
<I1> = k ・ <I2>… (3)
A value k is obtained. Thereafter, the final difference image Is (x, y) is obtained by the following equation (4), and the pupil can be extracted by binarizing the difference image by providing an appropriate threshold value.
Is (x, y) = I1 (x, y) -k ・ I2 (x, y)… (4)

Or, the divided image Idiv (x, y) is expressed by the following equation (5);
Idiv (x, y) = I1 (x, y) / (k · I2 (x, y))… (5)
It is also possible to divide the luminance levels of the images. At this time, since the background image other than the face region is farther than the face, the luminance level is low for both I1 (x, y) and I2 (x, y). For this reason, when dividing (5), the background image is subjected to strong sesame salt noise, and only the pupil portion is lifted while the face area is almost filled with zero. Since there is sesame salt noise in the background, it takes image processing costs to detect only the pupil from this image.

Therefore, the face area is obtained by the following method. As mentioned above, the background is dark because it is far away. In order to emphasize this, an image Im (x, y) obtained by multiplying I1 (x, y) and I2 (x, y) is obtained by the following equation (6).
Im (x, y) = I1 (x, y) ・ I2 (x, y) (6)
If you find the first derivative histogram of this image, you will get two peaks. If the valley is a threshold, the face area will be highlighted. When the logical product operation of this face region binarized image and the image obtained by the equation (5) is performed, the background portion disappears almost completely. Further, a histogram of the obtained image is obtained, and the luminance is examined in order from 0 to 0. The luminance level when the frequency becomes 0 for the first time is obtained, and binarization is performed using this as a threshold value, thereby extracting a pupil image. it can.

  Figure 8 shows the NTSC camera that outputs an image frame with a brightness level of 0 to 255 and an image size of 320 x 240. A two-wavelength light source of the type shown in Fig. 2 is attached to each light source in the odd and even fields under darkness. It is a simulation image when it makes it light. The lighting time of the shutter and the light source was 100 μsec. Here, (a): I1 (x, y), (b): I2 (x, y) are expressed in a pseudo manner, (e) is Im (x, y), and (f) is Idiv ( x, y) are represented, and the image after the AND operation and the finally obtained pupil image are (h) and (i), respectively. Also, the graphs (j) and (k) are the first derivative histogram and the histogram of the image (h), respectively. In the upper right of the figure, the specific calculation formula corresponding to the formula (6) (the formula for generating the image (e)) is (e) = {(c) × (d)} × 0.001. The specific calculation formula (formula for generating the image (f)) corresponding to the formula (5) is (f) = {(c) × 255 ÷ (d)} − 255.

Here, it is possible to determine whether or not the face is in the camera frame by obtaining the area, the center of gravity, and the range of the image showing the face image area of (g). Knowing whether a face is in a frame is important in two ways. (1) It is important to check the opening and closing of eyes with an in-vehicle drowsiness detection device or the like, and pupil detection can be used. When the pupil cannot be formed due to various problems, it is necessary to determine whether it is caused by blinking or whether the face is out of the camera frame, and to deal with whether or not there is a warning to the driver. (2) Since pupils and corneal reflection are present in the face area, their false detection in the face area can be greatly reduced.
In addition, in the multiplication image obtained by the equation (6), bright portions are emphasized by both I1 (x, y) and I2 (x, y), and therefore, corneal reflection can be detected from this image. However, since the face sometimes shines, the maximum value may not be obtained correctly. Therefore, it is detected as a bright spot from the vicinity of the previously detected pupil. FIG. 9 shows that two corneal reflection images can be detected from the entire image. (A) is an image of (e) and (b) of FIG. 8, and (a) is an image after smoothing (5 × 5 pixels). (C) is an operation of ((a) −10− (b)) × 10. I did.

  In this method, when the bright pupil becomes very bright in the dark, the pupil portion is extremely saturated in the bright pupil image shown in FIGS. If corneal reflection is present, it disappears. However, since the corneal reflection remains in the dark pupil image of FIG. 8B or 8D, it is effective to detect the corneal reflection from the integrated image of FIG.

  In this embodiment, since the shutter is set to 100 μsec, the difference in acquisition time between the image when the light source is lit and the image when the light source is not lit is 100 μsec. If the sensitivity of the camera is not sufficient, it is necessary to lengthen the shutter time, but it is better that the shutter time is short, and if this is long, the difference in acquisition time also becomes long, so the face image between the bright pupil image and the dark pupil image When the lens moves, not only the pupil moves, but also where the brightness level changes drastically, such as the outline of the face, it appears as an edge in the difference image, making it difficult to detect the pupil. Therefore, in FIG. 3, the light source is turned on only once for 100 μs and then turned off for the next 100 μs, and an image is not taken. For example, a predetermined period is blinked alternately every 10 μs as many times as necessary. It is also possible to subdivide into a plurality of periods in which the shutters are repeated and to repeatedly open the shutter in accordance with the alternately flashing periods. At this time, the charge may be transferred to separate CCD units in the subdivided 10 μsec period when the light source is turned on and when the light source is turned off. Finally, when the necessary number of flashing periods is completed, the charge is output as a luminance level signal from the CCD unit via the shift register. However, in this case, it is necessary to transfer charges from the photoelectric conversion element to the two CCDs in parallel at the timing of alternately repeating the connection configuration between the photoelectric conversion element and the CCD in the image sensor as shown in FIG. is there.

  Taking such a method has the following advantages with respect to detection of corneal reflection. In order to detect the line of sight with high accuracy, an approximate line of sight can be detected from the relative positional relationship between the pupil center reflected in the camera and the corneal reflection center of the light source. A proposal has already been filed for a method for strictly implementing the method (Japanese Patent Laid-Open No. 2005-185431). At this time, it is needless to say that the method using the image difference method described so far is preferably used for pupil extraction. However, for corneal reflection, a bright pupil image and a dark pupil image are obtained. If the positions of the two sets of light sources are not significantly deviated, they will overlap and cancel each other, so the corneal reflection center cannot be determined accurately. If the two sets of light sources are largely shifted, the entire apparatus becomes large, and as described above, the brightness level is shifted between the bright pupil image and the dark pupil image, so that it is not suitable for pupil detection. Therefore, it is better to detect the corneal reflection not from the difference image but from the bright pupil image or the dark pupil image. However, since these images are so-called raw images, they are affected by ambient light. Particularly noticeable is that when the driver's line of sight is detected directly on the face of the sun, the sun itself may appear exactly the same as the corneal reflection of the light source. This makes it impossible to distinguish between corneal reflection and the sun, and correct gaze detection cannot be performed. In addition, the sun or bright front scenery on the spectacle lens interferes with corneal reflection detection. As described above, merely by subtracting the dark pupil image from the bright pupil image, the sun image disappears and at the same time the corneal reflection disappears. Therefore, the cornea reflection image is detected from the difference image obtained by subtracting the non-illuminated image from the bright pupil image or the difference image obtained by subtracting the non-illuminated image from the dark pupil image. In these images, only images from the respective light sources can be obtained, so that images of ambient light such as the sun can be removed, so that corneal reflection can be easily detected. In addition, (a) and (b) of FIG. 8 represent images after the difference between the unilluminated images.

  Hereinafter, the pupil detection method and apparatus according to the present embodiment will be described with a focus on advantages compared with the prior art.

  Even in the case of the conventional pupil detection using the difference in the position of the light source and the conventional pupil detection using the two wavelengths, the light intensity balance of the two light sources is substantially matched while looking at the luminance level on the image. In this way, the light quantity is adjusted (FIG. 10A). Thereby, in the difference image, the luminance level of the background portion other than the pupil basically shows 0. Therefore, if the threshold value is set to a value larger than 0 in consideration of the noise of the background portion, the binarization can be easily performed. Pupil image can be extracted. On the other hand, if the dark pupil image is brighter due to an imbalance in the light amount balance (here, the light source for the bright pupil image has reached its limit and it is difficult to increase the amount of light, so the light source for the dark pupil image As shown in FIG. 10C, most of the background portion turns negative, and the threshold region for extracting the pupil image becomes narrower, so that the pupil detection tends to be difficult. is there. On the other hand, if the bright pupil image is brighter (in this example, the light amount of the light source for the bright pupil image has reached its limit and it is difficult to increase the light amount, an example in which the light amount of the light source for the dark pupil image is reduced) As shown in FIG. 10B, only the pupil part appears at first glance, but the luminance level of the background part is not known, so it is difficult to set the threshold value low. As shown in this example, in the difference image, when the luminance level of the pupil is extremely high with respect to the background portion, it is not a problem, but when it is low (when the pupil is extremely small), it becomes a big problem and the most appropriate. It can be said that such a setting method is a balanced state as shown in FIG.

  In the case of a detection method using a two-wavelength light source, automatic gain adjustment (AGC) is performed on the camera side in order to detect pupils in the range of ambient brightness from 0.0 lux (in the dark) to tens of thousands of lux. It is necessary to use the function. When the surroundings are bright, the gain of the camera is small, so the pupil level in the difference image is small. In addition, when the surroundings are bright, the pupil contracts and the incident light from the light source decreases, so that the pupil is not only small but also dark. In general, the sensitivity of an ordinary camera varies depending on the wavelength. When two cameras are used, the gain of both changes automatically. Therefore, even if the light intensity balance of two light sources is used at a certain brightness (at the wavelength used), the device is moved to a bright location. Then, the balance between the bright pupil image and the dark pupil image is greatly deviated, and it is difficult to detect the pupil only by making a difference.

  In the method of obtaining the bright pupil image and the dark pupil image by the difference in the position of the light source, since the light source for obtaining the bright pupil image and the dark pupil image is different, there are inevitably different irregularities and tendencies on the image. Therefore, even if the image difference is performed, the luminance level generally does not become zero even in the background portion, and it is buried in it and it is difficult to detect the pupil. Here, the irradiation intensity on the face of two light sources having different positions generally does not match as the light source becomes smaller. Therefore, even if the difference value becomes zero in some parts of the difference image, it does not become zero in other parts. This is especially true if the position of the head changes.

  On the other hand, according to the pupil detection method and apparatus according to the present embodiment, unlike conventional image sensors, the same photoelectric conversion element is used for each pixel, and light is received several times at slightly different timings. Basically, a bright pupil image, a non-illuminated image, and a dark pupil image with almost the same acquisition timing are obtained by performing photoelectric conversion and outputting the same to the outside. By subtracting the unilluminated image from the bright pupil image and dark pupil image, only the bright and dark components generated by the light source can be extracted, and the corneal reflection can be accurately detected regardless of the surrounding light environment, and the face area image is also accurate. Can be extracted. Furthermore, the method of detecting the pupil from the re-difference image obtained from the difference bright pupil image and the difference dark pupil image is particularly useful in a method using two cameras that perform wavelength separation using a two-wavelength light source. is there.

  In addition, by using a light source as shown in FIG. 2, the position of the LED is different, but when the outer LED is lit, the light is transferred to the inner LED, and the reflected image of the glasses tends to cancel out in the difference image. Yes, there is almost no problem of spectacle reflection, and the pupil is strongly embossed in the difference image compared to the two-wavelength single ring. Here, as a light source, a double ring is used as shown in FIG. 11, and an LED for obtaining a bright pupil image is arranged on the inner side, and an LED for obtaining a dark pupil image is arranged on the outer side. The difference may work synergistically. However, if the dark pupil image is darker than the bright pupil image due to differences in camera sensitivity and LED power, the number of LEDs for the dark pupil image is preferably increased as shown in FIG. Furthermore, as shown in FIG. 12, the light source may have a configuration in which LEDs are arranged in a ring shape by placing LED chips of two types of wavelengths in one mold. In this case, it can be configured in a small size, and the deviation of the position of the light source can be considerably reduced, so that the spectacle reflection can be almost completely eliminated when the image difference is made. However, in the method that relies only on the difference in wavelength, only a small level is placed between the bright pupil and the dark pupil, and therefore, when the pupil is dark, it is difficult to detect the pupil by relying only on the level difference. In this case, the configuration as shown in FIG. 2 or FIG. 11 is preferable.

  FIG. 18 shows an example of an image generated by the pupil detection method according to this embodiment. FIG. 18A shows a dark pupil image (raw image) without glasses, a non-illuminated image, and a difference image thereof. FIG. 18B shows a dark pupil image (raw image) with glasses, no image. Illumination images and their difference images are shown.

(Second Embodiment)
A second embodiment of the present invention will be described. The difference from the first embodiment in the pupil detection method according to the second embodiment is that the pupil is detected by one camera.

  Specifically, in the case of one camera, as one method, the light source for the bright pupil is turned on in FIG. 3A using the camera shown in FIG. ), The dark pupil light source is turned on, and the pupil can be detected from the obtained difference image. In this case, the method can be carried out in substantially the same manner even when the light source position is different or when the two-wavelength light source is used. Unlike the case of using a two-wavelength light source and using two cameras, the same amount of light is incident on the ambient light in both cases (a) and (b) even if AGC is used. Therefore, if the light quantity of the two-wavelength light source is balanced once in advance, the balance is not basically lost thereafter. However, if you need to detect corneal reflections in strong ambient light, you want an image with both light sources turned off. If there is, the luminance component due to the ambient light disappears and the corneal reflection can be easily detected by subtracting the image when not lit from the image when the light source is lit. In addition, even when there is strong unevenness in the brightness level on the face due to ambient light, it can be offset and an image can be obtained only by illumination of the light source, in which case the image does not change significantly. The face area can always be detected correctly, whereby it can be determined whether or not the face is included in the camera image, and there is no false detection when there is a pupil or corneal reflection other than the face.

  In order to implement this, the image sensor which expanded FIG. 3-7 can respond (FIGS. 13 and 14). Here, if the image obtained in the period of FIG. 13B when the light source is turned off is subtracted from the image obtained in the period of FIG. 13A where the light source for the bright pupil is turned on, only the light source for the bright pupil is obtained. The image by is obtained. Further, if the image obtained in the period of FIG. 13B when the light source is turned off is subtracted from the image obtained in the period of FIG. 13C where the dark pupil light source is turned on, only the dark pupil light source is obtained. An image is obtained. In this case, an image sensor is used in which three CCD units that can transfer charges from one photoelectric conversion element in different periods corresponding to one photoelectric conversion element are provided in a line (FIG. 14).

  According to such a pupil detection method, it is possible to suppress unevenness in the brightness level of the part other than the pupil part, which is generated on the difference image obtained by simply subtracting the dark pupil image from the bright pupil image due to the difference in the irradiation conditions of the two light sources. In particular, when the pupil is small, it is easy to detect the pupil even if the luminance level of the pupil is low.

(Third embodiment)
A third embodiment of the present invention will be described. The difference in the pupil detection method according to the third embodiment from the second embodiment is that a filter for a two-wavelength light source provided on the image sensor is used.

  That is, when one camera and a two-wavelength light source are used, as one method, there is a method of alternately attaching a filter that selectively transmits each image of the two-wavelength light source on the image sensor for each pixel. At this time, if the gain is the same for all the pixels, the light source of two wavelengths may be turned on simultaneously and all the pixels may be simultaneously exposed for the purpose of detecting the pupil. Then, the difference between adjacent ones is calculated, and the pupil may be detected from the difference image. However, in order to easily detect corneal reflection and cancel out even when the brightness level is strongly uneven on the face due to ambient light, an image sensor as shown in FIG. Use both light sources to turn on and expose (Fig. 15 (a)), transfer charges to the CCD unit, turn off both light sources and expose (Fig. 15 (b)), and transfer charges to the CCD unit To do. In this method, the image when the light source for the bright pupil is lit through the filter for the light source for the bright pupil, the image when the light source is not lit, and the light source for the dark pupil is lit through the filter for the light source for the dark pupil A total of four images can be obtained: an image when the light is not lit and an image when the light is not lit. The rest can be considered as in the case of two cameras. In the image sensor used at this time, the photoelectric conversion elements for the bright pupil and the photoelectric conversion elements for the dark pupil are alternately provided on the line, and the CCD units corresponding to the respective photoelectric conversion elements are arranged on the line. It is provided one by one.

  In addition, this invention is not limited to embodiment mentioned above. For example, the following processing may be added.

(Modification 1 for image unevenness)
If the position of the light source is different, generally each light source has a different brightness level gradient on the face. If the light sources are physically arranged vertically, the luminance level often varies in the vertical direction in the image. In that case, the method described in the first embodiment may be performed for each line in the horizontal direction of the image. If the light sources are arranged on the left and right, it may be performed for each line in the vertical direction. The same applies to an oblique case. In addition, even in the case of a two-wavelength light source, even in the case of a two-wavelength light source as shown in FIGS. 2, 11, and 12, unevenness occurs at each wavelength (for example, the bright pupil image is bright in the center of the image and the dark pupil is brighter in the surroundings). Therefore, the difference image does not become zero over the entire image. Again, there is not much problem when the pupil luminance level in the difference image is high, but when the pupil is dark, the presence of unevenness in the background reduces the robustness of pupil detection. In order to solve this, the method described in the first embodiment may be performed for each part of the image. That is, the analysis is performed by regarding the entire area enclosed by a square in FIG. 17 as an entire image. Then, it is only necessary to move this region within the entire image by one pixel at a minimum and obtain a result for each region to determine whether a pixel at the center of the region is a pupil portion or not.

(Modification 2 for image unevenness)
Eyeglass reflection In many cases, it appears as an intense big spot. Moreover, the bright pupil image and the dark pupil image do not always appear exactly the same. Therefore, the spectacle reflection has different effects on the average values I1 and I2 of I1 (x, y) and I2 (x, y), respectively. It has a great influence on the conversion process. Therefore, in the above-described processing in (Modification 1 for image unevenness), in each of the images I1 (x, y) and I2 (x, y), the same is applied from the higher luminance level, for example, 20%. % Pixels are extracted and the corresponding pixels are removed from both I1 (x, y) and I2 (x, y) (this number of pixels generally exceeds 20 + 20 in this example) In addition, the part other than the spectacle reflection part is frequently removed), and k obtains the conversion parameter of the histogram equalization process from the remaining pixels. By doing so, correct conversion can be performed, and a dark and small pupil can be easily extracted. However, since the spectacle reflection remains, it is distinguished from the pupil from the size and shape and removed after the final binarization in FIG. Note that this method is effective even when the bright pupil becomes very bright in the dark.

(Adjusting the amount of light emitted from the light source and measuring the intensity of facial irradiation with infrared rays)
In a dark place, the pupil is generally very large, and the AGC gain is also large, so the pupil may be saturated. In this case, at least in the bright pupil image, the corneal reflection may be crushed and cannot be detected, which may cause a problem.

  As described above, since AGC is used, the ambient brightness is difficult to understand from the image when the light source is turned on. However, if the non-illuminated image is used, the ratio of the average luminance in the face area of the bright pupil image or the dark pupil image after the difference between the average luminance in the face area of the non-illuminated image and the non-illuminated image is calculated. The irradiation intensity of sunlight (including infrared rays) can be estimated approximately. When it is determined that the surrounding is dark, the light emission amount can be automatically adjusted by adjusting the current amount of the light source. Just adjusting in two steps can change the current between night and day, and is well worth it.

(Feature point detection using wireframe method)
Using the template matching method, the corners of the eyes, the head of the nose, the left and right edges of the mouth, etc. are detected as feature points, and the stereo camera is used to obtain the three-dimensional structure of the face. This structure always fits the face. There is a wire frame method in which the structure is tracked in such a manner that the face direction and position are detected from the position and direction of the structure. A device that detects not only the face direction but also the boundary of the black and white eyes, the pupil and black eye edges, and the corneal reflection has been commercialized (SeeingMachine, faceLAB, http: //www.seeingmachines.com/). However, since the wire frame method uses the template matching method, the wire frame method is vulnerable to changes in the ambient light environment and cannot be correctly fitted. Even in this apparatus, it is necessary to illuminate the face with near infrared rays in order to use it at night and without distinction in the daytime. Such an environment does not greatly change the pupil detection method described so far. Therefore, by using an image sensor as shown in FIG. 4, when the light source is turned on and when the light source is turned off, images are continuously acquired, and the latter image is subtracted from the former image. An image with only the removed light source is obtained. If the image is made uniform in histogram, the same image as usual can be obtained even if AGC is used. Feature points such as the corners of the eyes, the head of the eyes, the head of the nose, and the left and right edges of the mouth are detected from this image, and the structure of the face is captured using the wire frame method. As described above, it is possible to detect feature points with high robustness by removing the influence of ambient light, thereby correctly providing a wire frame. Further, the pupil detection method using the image difference described so far Can easily detect pupils and corneal reflections. In other words, a bright pupil image, a dark pupil image, and an unilluminated image are obtained as described above by using two light sources that have not been used in the wire frame method until now, and an unilluminated image is obtained from the bright pupil image. From the difference bright pupil image obtained by subtracting the difference and the dark pupil image obtained by subtracting the unilluminated image from the dark pupil image, the feature points such as the corner of the eye, the head of the eye, the head of the nose, the left and right edges of the mouth are captured, Fit the face with the wire frame method. As a result, the eye position is captured robustly, and a re-difference image (or a divided image) obtained by subtracting the difference dark pupil image from the difference bright pupil image is obtained for the eye portion, and the pupil portion is extracted therefrom. Further, a corneal reflection is obtained from an integrated image (or an added image) of the differential bright pupil image and the differential dark pupil image. As described above, the wireframe method is applied to the image obtained by subtracting the non-illuminated image, and the eye position is detected with high robustness, and the pupil and corneal reflection are detected. It was possible to realize high-precision gaze direction detection that was difficult.

It is an optical path diagram which shows the example of arrangement | positioning of the pupil detection apparatus which is 1st Embodiment of this invention. It is a top view which shows the structure of the light source of FIG. It is a timing chart figure which shows the timing of the shutter of a camera, charge transfer, and a luminance level output. It is a figure which shows schematic structure of the image sensor of FIG. It is a figure which shows schematic structure of the image sensor of FIG. It is a figure which shows schematic structure of the image sensor of FIG. It is a figure which shows schematic structure of the image sensor of FIG. FIG. 3 is a diagram showing an image processing result when the light source shown in FIG. 2 is attached to an NTSC camera and each light source is turned on in an odd field and an even field. It is a figure which shows a mode that two corneal reflection images are detected from the image shown in FIG. It is a graph which shows the level of a dark pupil image, a bright pupil image, and a difference image, and (a) shows a case where the light intensity is adjusted so that the luminance levels on the images of the two light sources are substantially matched. When the light amount of the light source for the dark pupil image is decreased, (c) is a graph when the light amount of the light source for the dark pupil image is increased. It is a top view which shows the modification of the light source of FIG. It is a top view which shows the modification of the light source of FIG. In the pupil detection method concerning 2nd Embodiment, it is a timing chart figure which shows the timing of the shutter of a camera, charge transfer, and a luminance level output. It is a figure which shows schematic structure of the image sensor concerning 2nd Embodiment. In the pupil detection method concerning 3rd Embodiment, it is a timing chart figure which shows the timing of the shutter of a camera, charge transfer, and a luminance level output. It is a figure which shows schematic structure of the image sensor concerning 3rd Embodiment. It is an image figure which shows the area | region of the process target in an image in the modification of this invention. It is a figure which shows an example of the image produced | generated in the pupil detection method of this invention, (a) is a dark pupil image (raw image) in the case of without glasses, an unilluminated image, those difference images, (b) is It is a figure which shows the dark pupil image (raw image) in the case with glasses, a non-illuminated image, and those difference images.

Explanation of symbols

C1, C2 ... image sensor, 21 ... light source, F1, F2 ... band pass filter, 23a, 23b ... light emitting element.

Claims (11)

A pupil detection method for detecting a pupil of a subject,
Using the image sensor having one photoelectric conversion element corresponding to one pixel, the light source for obtaining a bright pupil image or a dark pupil image is irradiated to the subject in the first period, and A first step of transferring the charge accumulated by exposure to the first charge transfer section of the image sensor corresponding to the photoelectric conversion element after the image sensor is exposed to the period of 1;
In a state where the light source is turned off in a second period that is continuous with the first period, after the image sensor is exposed in the second period, charges accumulated by exposure are transferred from the photoelectric conversion element to the photoelectric conversion element. A second step of transferring to a second charge transfer unit different from the first charge transfer unit corresponding to the conversion element;
A third step of outputting the charges transferred to the first and second charge transfer units using a shift register connected to the first and second charge transfer units;
A fourth step of detecting a pupil by performing image difference processing after obtaining a bright pupil image, an unilluminated image, and a dark pupil image based on the output of the shift register;
A pupil detection method comprising: In the first step, the first period is subdivided into a plurality of periods repeated a predetermined number of times in a predetermined cycle, and charges are transferred to the first charge transfer unit for each of the plurality of periods,
In the second step, the second period is subdivided into a plurality of periods repeated the predetermined number of times in the predetermined period, and charges are transferred to the second charge transfer unit for each of the plurality of periods. ,
In the third step, after the predetermined number of repetitions of the first period and the second period are completed, the charges transferred to the first and second charge transfer units are output.
Pupil detection method.   In the fourth step, corneal reflection is detected from a difference image obtained by subtracting an unilluminated image from a bright pupil image or a difference image obtained by subtracting an unilluminated image from a dark pupil image. The pupil detection method according to claim 1 or 2.   In the fourth step, corneal reflection is obtained from an image obtained by multiplying a difference image obtained by image difference of an unilluminated image from a bright pupil image and a difference image obtained by image difference of an unilluminated image from a dark pupil image. The pupil detection method according to claim 1 or 2, wherein:   In the fourth step, a post-difference bright pupil image obtained by subtracting an unilluminated image from a bright pupil image, and a post-difference dark pupil image obtained by subtracting an unilluminated image from a dark pupil image The pupil detection method according to any one of claims 1 to 4, wherein a pupil is detected from a re-difference image obtained by image difference. The re-difference image is obtained by subtracting the post-difference dark pupil image from the post-difference bright pupil image after correcting the luminance balance between the post-difference bright pupil image and the post-difference dark pupil image in a partial region of the image. It is a difference image after luminance balance correction obtained by
The pupil detection method according to claim 5. The difference image after luminance balance correction is a difference image after correcting luminance balance for spectacle reflection obtained by subtracting the dark pupil image after difference from the bright pupil image after difference in consideration of spectacle reflection.
The pupil detection method according to claim 6.   In the fourth step, in order to obtain a difference image after correcting the spectacle reflection luminance balance, a coefficient for balancing the luminance is obtained by excluding spectacle reflection areas in the post-difference bright pupil image and the post-difference dark pupil image. The pupil detection method according to claim 7 to be obtained. From the ratio of the brightness level of the post-difference bright pupil image or the post-difference dark pupil image and the non-illuminated image, the ambient light intensity is estimated, and the light emission amount of the light source is further adjusted.
The pupil detection method according to claim 5. In the fourth step, a template matching method using a predetermined subject portion as a tracking feature point based on the post-difference bright pupil image and the post-difference dark pupil image, or a wire frame method based on the template matching method Detecting feature points of the subject by
The pupil detection method according to claim 5. The position of the subject's eyes is specified by the template matching method or the wire frame method based on the template matching method, and the pupil is detected from the re-difference image,
The pupil detection method according to claim 10.