JP4528976B2 - Pupil detection apparatus and method - Google Patents

Description

  The present invention relates to an apparatus and method for detecting a pupil of a subject based on a photographed image.

  The technology for detecting the subject's pupil is, for example, preventing a drowsy driving by detecting the subject's line of sight and changes in the line of sight while driving a car, or using a finger by the movement of the subject's pupil It can also be used for a pupil mouse that performs input to a computer. The inventor has developed a technique for detecting a pupil from an image photographed by a camera. For example, Patent Document 1 discloses a technique for detecting a subject's line of sight from a position between a pupil and a corneal reflection point. Patent Document 2 discloses a technique in which the position coordinate of the detected pupil is used as a cursor position control signal. Patent Document 3 discloses a technique for determining the direction of the head by detecting the positions of the pupil and nostril. Patent Document 4 discloses a technique for determining the three-dimensional position of the viewpoint viewed by the subject by detecting the positions of the pupil and the corneal reflection point.

  In these techniques, for detection of the pupil, a method is used in which the pupil portion is distinguished from the surrounding image by subtracting the bright pupil image and the dark pupil image captured by the camera. When a light source such as near-infrared light is provided near the camera's aperture and the subject's face is irradiated with light along the optical axis of the camera, the light reaches the retina from the pupil and is reflected, and the lens and cornea are reflected. Go back to the camera opening. In this image, the pupil is photographed brightly, and the image is called a bright pupil image. On the other hand, if the subject's face is illuminated with light from a light source away from the camera opening, the light reflected from the retina hardly enters the camera opening, so the pupil is photographed darkly, and the image is darkened. This is called a pupil image. The size of the pupil changes depending on the brightness of the surroundings, and becomes small and difficult to detect particularly in bright places. In addition, since a part of the glasses near the pupil causes reflection in a spectacle wearer, it is difficult to detect the pupil from a single image using either the bright pupil or the dark pupil. However, if the difference between the image obtained by photographing the bright pupil and the image obtained by photographing the dark pupil is obtained, the surrounding portions other than the pupil portion have almost the same brightness in both images. Only the pupil with a difference will be highlighted. Thereby, the pupil can be easily detected, and the cost of pupil detection can be reduced and the robustness can be improved.

  Depending on the surrounding brightness and other conditions, the image of the pupil part is not necessarily brighter than the surrounding image even in the case of the bright pupil. It is not always darker. The bright pupil and the dark pupil in this specification have a relative brightness difference between the pupils in the two images taken at that time, and the bright pupil is relatively brighter than the dark pupil. That's what it means.

Furthermore, the present inventor shot the face of the subject in a state in which the light sources having different wavelengths of retina are simultaneously turned on without switching on and turning on the two light sources. Patent applications 1 and 2 relating to a technique for simultaneously obtaining a bright pupil image and a dark pupil image with different wavelengths and performing a difference between the images.
[Patent Document 1] JP-A-2005-185431
[Patent Document 2] JP 2005-182247 A
[Patent Document 3] Japanese Patent Application Laid-Open No. 2005-266868
[Patent Document 4] Japanese Patent Application Laid-Open No. 2005-198743
[Patent Application 1] Japanese Patent Application No. 2005-241728
[Patent Application 2] Japanese Patent Application No. 2005-306961

However, in the techniques of Patent Documents 1 to 4, since the light source for bright pupil photography and the two light sources for dark pupil photography are alternately turned on and photographed, the photographing of the bright pupil image and the dark pupil image involves a time difference. If the subject's face moves during that time, the positions of the bright and dark pupils in both images will shift, and the position of the pupil may not be detected accurately due to the image difference. For example, when an odd field and an even field constituting one frame every 1/30 seconds in NTSC camera shooting are a bright pupil image and a dark pupil image, respectively, 1/60 seconds are required for shooting both images. If there is a time difference and the position of the pupil in both images shifts to a state where they do not overlap due to the movement of the face between them, there is no effect of differentiating the images. As shown in FIG. 5A, when the bright pupil image P1 and the dark pupil image P2 are partially shifted, the difference effect appears only in the overlapping hatched portion. If this further deviates, there will be no overlapping portion and no difference effect will be produced, making pupil detection difficult. Also, if the face moves while the shutter is open during bright pupil and dark pupil photography, the pupil image will be blurred as shown in FIG. Even in the bright pupil image P1 and the dark pupil image P2 blurred as shown in (C), if a part of them overlaps, a temporary difference effect appears in the hatched portion. However, as shown in FIG. 6, if there is a high-luminance part near the pupil, such as reflection of glasses, the images d1 and d2 are erased by taking the difference if the two images match. However, if the two images are shifted, even if the difference is made, it remains as a high-luminance portion like a shaded portion, and thus it may not be distinguished from the pupil images P1 and P2. Thus, if the bright pupil image and the dark pupil image are shifted, it becomes difficult to detect the pupil with high accuracy.
In addition, according to the techniques of Patent Applications 1 and 2, light of different wavelengths is simultaneously irradiated, and a bright pupil image and a dark pupil image are obtained at the same wavelength, so that there is no time difference, but wavelength separation is performed. Problems such as the need for equipment and two cameras for photographing at two wavelengths, and the need for work for wavelength separation arise.

  The present invention has been made to solve these problems, and the position of the pupil part due to the time difference between the image obtained by photographing the bright pupil and the image obtained by photographing the dark pupil without requiring a particularly complicated apparatus or work. An object of the present invention is to provide a highly robust pupil detection apparatus and method that eliminates the problem of misalignment.

  In order to achieve such an object, the pupil detection device of the present invention has an image sensor of a rolling shutter system, and obtains bright pupil image data and dark pupil image data for each alternate line of the image sensor. And image analysis means for detecting the pupil by taking the difference between the bright pupil image data and dark pupil image data for each line obtained by the imaging means.

  According to the present invention, since the photographing time difference between the lines of the image sensor photographed by the rolling shutter method is very short, a bright pupil image and a dark pupil image with little deviation based on the time difference are captured. In the image analysis means, the difference between the data of the bright pupil image and the dark pupil image can be used to make the pupil portion stand out and to easily detect the pupil.

  Further, in the present invention, when the imaging means has a light source that is close to the optical axis of the camera and a light source that is far from the camera, the bright pupil image can be obtained when the light sources are turned on only by changing the positions of the two light sources. Data or dark pupil image data can be obtained.

  Further, in the present invention, when the imaging unit has two light sources having different wavelengths, the bright pupil image data or the dark pupil image data is obtained when the respective light sources are turned on regardless of the installation position. Can do.

  Further, in the present invention, when the image pickup means has two types of band-pass filters corresponding to different wavelengths alternately arranged for each line of the image sensor, it is not necessary to distinguish between the light sources. A bright pupil image and a dark pupil image can be obtained for each line of the image sensor.

  Further, in the present invention, when the image analysis means detects the pupil by taking the difference between the bright pupil image data and the dark pupil image data of the adjacent line, it is adjacent without special data processing. It is possible to take the difference between the data with almost no time difference.

  Further, in the present invention, the image analyzing means takes a difference between the average of the bright pupil image data or the dark pupil image data of the nearest line and the dark pupil image data or the bright pupil image data between those lines. Thus, when detecting the pupil, it is possible to perform image analysis in which the positional deviation between the lines of the image sensor is eliminated and the resolution is increased.

  Further, in the pupil detection method of the present invention, the bright pupil and the dark pupil are alternately photographed for each line of the rolling shutter type image sensor, and after obtaining the bright pupil image data and the dark pupil image data for each line, The pupil is detected by taking the difference between the bright pupil image data and the dark pupil image data for each line.

  According to the present invention, since the photographing time difference between the lines of the image sensor photographed by the rolling shutter method is very short, it is possible to obtain bright pupil image data and dark pupil image data with almost no deviation. By making a difference between the bright pupil image data and the dark pupil image data, the pupil can be made to stand out and the pupil can be easily detected.

  Further, in the present invention, when alternately photographing bright pupils and dark pupils for each line, when exposing a plurality of lines simultaneously to obtain bright pupil image data and dark pupil image data for each line, One line can be exposed multiple times, and the exposure time can be increased as necessary.

  Further, in the present invention, when photographing the bright pupil and the dark pupil is performed by alternately switching on the light source close to the optical axis of the camera and the light source far from the camera, only the positions of the two light sources are changed and installed. Thus, bright pupil image data or dark pupil image data can be obtained when each light source is turned on.

  In addition, in the present invention, when the bright pupil and the dark pupil are photographed by alternately switching on the light sources having different wavelengths, the light sources having different wavelengths are always turned on regardless of the installation position of the light sources. Thus, bright pupil image data or dark pupil image data can be obtained when each light source is turned on.

  Further, in the present invention, when photographing the bright pupil and the dark pupil is performed through band pass filters corresponding to different wavelengths arranged for each line of the image sensor, it is not necessary to distinguish between the light sources, Bright pupil image data and dark pupil image data can be obtained for each line of the image sensor.

  In the present invention, the difference between the bright pupil image data and the dark pupil image data for each line is the special data when the difference between the bright pupil image data and the dark pupil image data of the adjacent line is taken. It is possible to take the difference between the adjacent data with almost no time difference without processing.

  Further, in the present invention, the difference between the bright pupil image data and the dark pupil image data for each line is the average of the bright pupil image data of the nearest lines or the dark pupil image data, and the dark pupil between those lines. When taking a difference from the image data or the bright pupil image data, it is possible to perform an image analysis in which the positional deviation between the lines of the image sensor is eliminated and the resolution is increased.

  According to the present invention, it is possible to solve the problem of the positional deviation of the pupil part due to the time difference between the image obtained by photographing the bright pupil and the image obtained by photographing the dark pupil without requiring a particularly complicated apparatus or work, and thereby exhibiting robustness. An apparatus and method for detecting a high pupil can be provided.

Hereinafter, preferred embodiments of an apparatus and method for detecting a pupil according to the present invention will be described in detail with reference to the drawings. FIG. 1 is a conceptual diagram showing the entire pupil detection device of the present embodiment.
The pupil detection device 1 includes an imaging unit 2, a control unit 3, and an image analysis unit 4. The imaging means 2 is for photographing a face portion including the pupil P of the pupil detection target person. The camera 21, the near-infrared light source 22 arranged at a position close to the optical axis of the camera 21, and the light of the camera 21. It has a near infrared light source 23 arranged at a position away from the axis. When the subject's pupil is photographed by turning on the light source 22 close to the camera optical axis, the light reflected from the subject's retina enters the camera 21 as described above, so that the bright pupil is photographed. When photographing is performed with the light source 23 far from the camera optical axis turned on, the light reflected by the retina hardly enters the camera 21 and thus the dark pupil is photographed. The control means 3 mainly gives commands to the camera 21 and the light sources 22 and 23. The image analysis means 4 mainly obtains image data from the camera 21 and performs data analysis for pupil detection. Then, a signal based on the result of pupil detection by the image analysis means 4 is transmitted to an output device outside the pupil detection device. Here, the output device 5 is, for example, a drowsiness warning device for a pupil detection target person who is a driver who is driving the transport device, or a computer input device that uses a pupil change as a pupil detection target person's pupil change. The control means 3 may perform various controls based on signals from the imaging means 2 and the image analysis means 4, and the control means 3 and the image analysis means 4 may be configured by a single computer. Needless to say.

  The camera 21 in the imaging means uses an image sensor such as a CMOS that can perform a rolling shutter type electronic shutter operation as an imaging element. In the rolling shutter system, the pixels in the image sensor are exposed from the top row and moved to the bottom row by row. As with exposure, reading starts from the top row and proceeds to the bottom one row at the same speed and is read sequentially. The time difference between the column where exposure is started and the column where readout is performed is the accumulation time (exposure time).

  Next, the operation of pupil detection according to this embodiment will be described. When photographing the face portion of the subject, the light source 22 and the light source 23 are alternately turned on in synchronization with the exposure for each line of the image sensor of the camera 21. Then, as shown in FIG. 2A, at the time t1, the uppermost line L1 of the image sensor is exposed. At this time, since the light source 22 is turned on, the pupil of the subject is a bright pupil. Taken. As shown in FIG. 2B, in the exposure of the second line L2 at the next time t2, since the light source 23 is turned on, the pupil is photographed as a dark pupil. At the next time t3, as shown in FIG. 2C, the light source 22 is turned on again in accordance with the exposure of the line L3, and the bright pupil is photographed. At the next time t4, as shown in FIG. 2D, the line L3 is taken of the dark pupil. In this way, bright pupil images are obtained in order from the top in odd lines of the image sensor, and dark pupil images are obtained in order from the top in even lines.

In image analysis, one odd line and the image data of the next even line are associated with each other, and the pixel at the corresponding position of the line in which the dark pupil is imaged from the luminance data in each pixel of the line in which the bright pupil is imaged. The difference data is obtained by subtracting the luminance data. Such an operation is performed for each set of an odd line and the next even line. Either the bright pupil or the dark pupil may be imaged with either the odd line or the even line, but when obtaining the difference, the dark pupil image data is subtracted from the bright pupil image data.
The coordinate along the line L is x, the coordinate perpendicular to the line is y from the top to the bottom, and the luminance of the coordinate (x, y) is S (x, y). ), The luminance at coordinates (x, y + 1) is represented as S (x, y + 1), the difference data at coordinates (x, y) is represented as D (x, y), and the y-th line is a bright pupil imaging line. If the (y + 1) th line is a dark pupil imaging line, the difference data is as shown in the following equation.
D (x, y) = S (x, y) −S (x, y + 1)

  This will be described with reference to FIG. 4. A part of the bright pupil line 1 is an upper line, and a part of the dark pupil line 2 is an intermediate line. In line 1, the bright pupil portion has a high luminance, and in line 2, the dark pupil portion has a low luminance. As described above, the brightness of the bright pupil is higher than that of the surroundings and the luminance of the dark pupil is not necessarily lower than that of the surroundings. However, the bright pupil is relatively brighter than the dark pupil. In the vicinity of the pupil, a portion having high luminance appears due to a disturbance such as reflection of glasses, and such a disturbance appears in the same way when shooting the bright pupil and when shooting the dark pupil. When the difference is taken, the disturbance disappears and only the pupil part is highlighted. Such calculation is performed on the entire image or a portion where a pupil is predicted to exist by a prediction model such as a Kalman filter. Thereby, the pupil can be detected by obtaining a difference image and binarizing the discriminant analysis method or the like.

  Here, the time difference between adjacent lines is very short. For example, if one image is taken in 1/30 seconds and the number of lines is 500, the time difference between adjacent lines is It is about 1/5000 second (about 70 μs), and there is almost no deviation or blurring of the pupil image data between the bright pupil line and the adjacent dark pupil line. In the case of the rolling shutter method, if an object that moves fast is photographed, the image may be distorted because there is a time difference between the exposure of the upper line and the exposure of the lower line. If the time difference between the two is small, the pupil can be easily detected, and thus the problem of the rolling shutter method is not inconvenient.

As described above, since the difference data is obtained by performing the calculation using the odd line and the next even line (or vice versa) as one unit, the number of lines of the obtained difference data is the odd line and the even line. Half of the combined number of lines. In addition, since the difference between a certain line and the next line is taken, in terms of the y direction, it is not exactly the difference between the data whose positions match. However, taking the average of the data of two odd lines adjacent to each other, the data represents data corresponding to the position in the y direction of the even line in the middle of the two odd lines. Therefore, by taking the difference between the average value of the data of two adjacent odd lines and the data of the even line in the middle, it is possible to obtain difference data in which the positional deviation is eliminated in the y direction. Similarly, the difference between the average of the data of two adjacent even lines and the data of the middle odd line can be taken. By doing so, it is possible to eliminate the shift in the position in the y direction. For example, the brightness of the image data of the edges of the nostrils, eyelids, etc. may vary greatly between the lines to be subtracted. Although there is a case where it cannot be erased, the average value of one line placed in this way is obtained, and the difference from the line located between them is prevented, so that disturbance can be prevented from remaining as a large value after the difference. Further, the number of lines from which difference data can be obtained is the total number of odd lines and even lines, and the resolution can be increased. Similar to the above, such a difference method is represented by (x, y) coordinates, luminance S (x, y) of the coordinates, and difference data D (x, y) as follows. The y-th and (y + 2) dark eye lines are assumed to be bright pupil imaging lines, and the (y-1) th and (y + 1) -th lines are assumed to be dark pupil imaging lines.
D (x, y) = S (x, y)-(S (x, y-1) + S (x, y + 1)) / 2
D (x, y + 1) = (S (x, y) + S (x, y + 2)) / 2−S (x, y + 1)
Specifically, the average value of the dark pupil image data of the second and fourth even lines is subtracted from the bright pupil image data of the third odd line, and the bright pupil image data of the third and fifth odd lines. The dark pupil image data of the fourth even line is subtracted from the average value.

  The above is an example in which bright pupils and dark pupils are alternately photographed for each line, but a plurality of odd lines and a plurality of even lines may be alternately exposed and photographed as shown in FIG. As shown in FIG. 3A, at the first time t1, the lines L1, L3, and L5 are exposed to photograph the bright pupil. At the next time t2, the lines L2, L4, and L6 are exposed as shown in FIG. At the next t3, as shown in FIG. 3C, the lines L3, L5, and L7 are exposed to photograph the bright pupil. Thereafter, exposure is performed every three lines in the same manner. In this case, one line is exposed three times, and the data is a combination of three exposures. In this way, the total exposure can be freely set by changing the number of lines exposed simultaneously. Therefore, for example, it is effective when it is desired to increase the exposure amount in each line.

  According to this embodiment, the photographing time difference between the lines of the image sensor photographed by the rolling shutter method is very short, and the image data between the lines is used as bright pupil image data, dark pupil image data. Therefore, it is possible to erase a disturbance other than the pupil and obtain a difference image in which the pupil is conspicuous by using the bright pupil image and the dark pupil image with little positional deviation due to the time difference.

  In addition, the pupil is detected by taking the difference between the average of the bright pupil image data or dark pupil image data of the nearest line and the dark pupil image data or the bright pupil image data between those lines. Therefore, it is possible to eliminate the positional deviation between the lines of the image sensor and to perform image analysis with improved resolution.

  In addition, a bright pupil image or a dark pupil image can be obtained simply by installing a light source close to and far from the optical axis of the camera 21 and switching lighting in accordance with exposure for each line of the image sensor.

  In the present embodiment, the two light sources 22 and 23 having different positions are used in order to obtain images of the bright pupil and the dark pupil. However, two light sources having different wavelengths may be used. For example, when light sources having different wavelengths of about 850 nm and 950 nm are used, the retinal reflectance is different, so that a bright pupil is obtained in an image photographed with a light source of about 850 nm, and a dark pupil is obtained in an image photographed with a light source of about 950 nm. Is obtained. In this case, even when the installation position is set at an equal distance from the opening of the camera 21 or when the installation position is set at a position at a different distance, by alternately turning on the respective light sources, A bright pupil image or a dark pupil image can be obtained respectively.

  Further, as a means for obtaining a bright pupil image and a dark pupil image with such two types of light sources, two types of bandpass filters may be arranged for each line of the image sensor. For example, when using a light source having a wavelength of about 850 nm and a light source having a wavelength of about 950 nm as described above, two band pass filters that allow light of one wavelength to pass and block light of the other wavelength are provided in the pixel of the image sensor. Form on each. Various band-pass filters can be used as long as they fulfill such functions. In this case, each line of the image sensor automatically selects and passes light for photographing bright pupils and light for photographing dark pupils, so that it is not necessary to control lighting of the light source in accordance with exposure. Further, the arrangement of the band-pass filters can be obtained by changing the arrangement pattern in addition to providing the arrangement of the band-pass filters in a completely alternate manner for each line.

It is a conceptual diagram of the apparatus of embodiment of this invention. It is explanatory drawing which shows exposure with the image sensor of embodiment of this invention. It is explanatory drawing which shows exposure with the image sensor of embodiment of this invention. It is explanatory drawing of the image processing of embodiment of this invention. It is explanatory drawing when the image of a pupil has shifted | deviated. It is explanatory drawing when the image of a pupil and the image of the reflection part of glasses, etc. have shifted | deviated.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Pupil detection apparatus, 2 ... Imaging means, 21 ... Camera, 22, 23 ... Light source, 3 ... Control means, 4 ... Image analysis means, 5 ... Output device

Claims (13)

An image pickup means having a rolling shutter type image sensor, and obtaining bright pupil image data and dark pupil image data for each alternate line of the image sensor;
A pupil detection device comprising image analysis means for detecting a pupil by taking a difference between the bright pupil image data and the dark pupil image data for each line obtained by the imaging means.   The pupil detection apparatus according to claim 1, wherein the imaging unit includes a light source that is close to an optical axis of the camera and a light source that is far from the optical axis.   The pupil detection device according to claim 1, wherein the imaging unit includes two light sources having different wavelengths.   The pupil according to any one of claims 1 to 3, wherein the imaging unit has two types of band-pass filters corresponding to different wavelengths alternately arranged for each line of the image sensor. Detection device.   The pupil detection device according to claim 1, wherein the image analysis unit detects a pupil by taking a difference between the bright pupil image data and the dark pupil image data of adjacent lines. .   The image analysis means takes the difference between the average of the bright pupil image data or the dark pupil image data of the nearest line and the dark pupil image data or the bright pupil image data between the lines, and determines the pupil. The pupil detection device according to claim 1, wherein detection is performed. For each line of the rolling shutter type image sensor, after taking bright pupils and dark pupils alternately, and obtaining bright pupil image data and dark pupil image data for each line,
A pupil detection method for detecting a pupil by taking a difference between the bright pupil image data and the dark pupil image data for each line.   8. The bright pupil image data and the dark pupil image data for each line are obtained by simultaneously exposing a plurality of lines when photographing bright pupils and dark pupils alternately for each line. The pupil detection method described in 1.   9. The pupil detection method according to claim 7, wherein photographing of the bright pupil and the dark pupil is performed by alternately switching on a light source close to an optical axis of a camera and a light source far from the camera.   The pupil detection method according to claim 7, wherein photographing of the bright pupil and the dark pupil is performed by alternately switching on lighting of light sources having different wavelengths. 11. The pupil according to claim 7, wherein imaging of the bright pupil and the dark pupil is performed through band-pass filters corresponding to different wavelengths arranged for the lines of the image sensor. Detection method.   The difference between the bright pupil image data and the dark pupil image data for each line is a difference between the bright pupil image data and the dark pupil image data of an adjacent line. The pupil detection method according to any one of the above.   The difference between the bright pupil image data for each line and the dark pupil image data is the average of the bright pupil image data of the nearest line or the dark pupil image data, and the dark pupil image data between those lines. Or the difference with bright pupil image data is taken, The pupil detection method in any one of Claims 7-12 characterized by the above-mentioned.

Images