JP4528980B2 - Pupil detection device and pupil detection method - Google Patents

Description

  The present invention relates to a pupil detection device and a pupil detection method that can be used in the field of eye gaze detection, and more specifically, pupil detection that enables reliable detection of the pupil even in an environment where the ambient light is strong and is not suitable for pupil detection. The present invention relates to an apparatus and a pupil detection method.

  The detection of pupils has attracted attention as an effective means for capturing information such as estimating the subject's intention by capturing facial feature points and obtaining information such as where they are looking from and where they are looking from. Various studies and inventions have been made in relation to applications (Patent Documents 1 to 4, Non-Patent Document 1). The pupil is excellent as a detection target because it is small in size and not only looks like a circle or an ellipse when viewed from any direction, but is unlikely to be hidden by the eyelid unlike black eyes. Using this, there is a proposal to make the movement of the pupil correspond to the cursor movement on the personal computer screen, and the cursor can be moved by the movement of the head.

Recently, there has been a demand for detecting the line of sight of driving of passenger cars and trucks, detecting drowsiness, and detecting unnecessary driving using pupil detection technology. At this time, it is desired that the pupil can be stably detected even under a poor situation where direct sunlight hits the face.
Japanese Patent Application No. 2004-73998 JP2004-261598 Special table 2002-513176 JP 11-56782 A "Pupil detection and tracking using multiple light sources", author H. Morimoto et al. (CH Morimoto, D. Koons, A. Amir, M. Flickner), Image and Vision Computing 18 (2000) 331-335 (Image and Vision Computing 18 (2000) 331-335)

Consider possible problems in bright pupil detection. A red eye phenomenon (bright pupil phenomenon) is known as one of the relation between the light source and the pupil in the detection of the pupil, and this phenomenon is used for pupil detection (Patent Document 1).
FIG. 1 is an optical path diagram showing an example of a positional relationship among the camera C, the light source L, and the eyeball EB where the bright pupil is not observed. The light beam incident on the pupil from the light source L is indicated by lines indicated by 1 and 2 in the figure. The light source L is located away from the opening (the effective diameter of the lens) of the camera C.
If the subject's eyes have refractive power, and now the focus of the eye coincides with the position of the light source L, the ends of the region through which the light beam incident on the pupil passes are as 1 'and 2' lines. The light source image focuses on the retina and creates a spot of dots on the retina. On the retina, the light is irregularly reflected and reflected in various directions, but a part thereof passes through the pupil and returns in the path of 1′-1 and 2′-2 (shaded portion). At this time, the light generated from the spot on the retina is focused on the position of the light source, and if it is assumed that the light source L has no size, it passes through the light source position and spreads again. Since the light returned from this eye does not enter the opening of the camera, the pupil appears dark from the camera.

FIG. 2 shows a case where the eyes are focused before the light source (far-sighted state).
In this case, since the light incident on the pupil is focused behind the retina, it appears as a large spot out of focus on the retina. The light from the spot exits the retina and is focused at the focal position of the eye, and passes through the range of 1 → 1 ′ path and 2 → 2 ′ path (shaded area). This path can be understood by considering the spot on the retina as a collection of point light sources and considering the focal position of the eye. Also in this case, the light returned from the pupil does not enter the camera.

  Conversely, as shown in FIG. 3, when the focus of the eye is located in front of the light source (myopia), the light from the light source that has passed through the retina is focused on the front of the retina and the size on the retina is reduced. Make a spot to have. Of the irregularly reflected light from the spot, the light passing through the pupil passes through the shaded area in the figure. Also in this case, since the reflected light from the retina does not enter the opening of the camera, the pupil appears as a dark part from the camera.

  If the light source is arranged near the opening of the camera as shown in FIG. 4, a part of the reflected light from the retina enters the opening of the camera, so that the pupil appears bright. However, as is clear from this figure, since more than half of the reflected light from the retina does not enter the opening of the camera, it is inefficient to detect the pupil by illuminating the pupil.

Up to now, when the pupil is large (generally, the maximum diameter is 8 mm), and when the pupil is small (generally, the minimum diameter is 2 mm), the light path is as shown in FIG. If the pupil is small, the amount of light that passes through the pupil is also small in the first place (the pupil area and the pupil luminance are proportional), and the pupil luminance is low, so that the pupil is extremely difficult to detect. In addition, since the pupil is simply small, it is difficult to distinguish from the remnants of spectacle reflection by image processing, leading to false detection. Furthermore, in reality, the light source has a size, and a certain amount of distance is absolutely necessary from the light source to the opening. Therefore, the amount of light returning to the camera opening is extremely reduced.
In other words, when the pupil is large, the pupil is likely to shine even if the light source is relatively far from the camera opening, but when the pupil is small, it is as close as possible to the camera opening, and ultimately not present in the opening. It is hard to be bright. As shown in FIG. 6, when the light source is aligned with the optical axis of the camera, a bright pupil is obtained, but this is an obstacle to imaging.

  In the aforementioned Patent Document 4, a method of installing a light source in the center of the camera opening in order to obtain a bright pupil (bright eye) image has been proposed, but in this case also, the light source itself has a size, When only the center of the light source is bright, the light source and the aperture are substantially at a distance, and the pupil is difficult to be bright. Also, the biggest problem with the method of installing the light source in the camera opening is that if the camera magnification is reduced, for example, if the camera lens is selected to an enlargement factor that allows the entire face 80 cm ahead to be fully reflected, The image appears as black and interferes with pupil detection.

  An object of the present invention is to provide a pupil detection device and a pupil detection method that can reliably detect the pupil even in an environment where the ambient light of the subject is strong and is not suitable for detection of the pupil.

In order to achieve the above object, a pupil detection device according to claim 1 of the present invention includes a camera, a light source, an optical path forming means, and a calculating means, and the optical path forming means is a light from the light source on the face of a subject. And a pupil detection device for detecting a pupil by calculating an image obtained by forming an image of the face including the pupil of the subject on the camera and forming an image on the camera The light source is provided so as to be able to irradiate the light from within the opening of the camera, and includes a first illumination light source including a first wavelength component in which the reflection in the pupil of the subject becomes a bright pupil, and the pupil of the subject The internal reflection includes a second wavelength component that becomes a dark pupil, and includes a second illumination light source that exhibits an illumination effect equivalent to that of the first illumination light source except for the pupil, and includes an independent first illumination light source and a second illumination light source. The optical path of the illumination light source Synthesized so as to keep the common optical axis by means consists further said common optical axis to change the optical path to match the optical axis of the camera unit, the camera, first by the first illumination light source 1 First image data acquisition means for obtaining the image data, and second image data acquisition means for obtaining the second image data by the second illumination light source, wherein the calculation means comprises the first image data And a means for calculating the second image data and detecting the pupil.
Moreover, the pupil detection device according to claim 2 according to the present invention includes a camera, a light source, an optical path forming unit, and an arithmetic unit, and the optical path forming unit irradiates light from the light source to the face of the subject, and A pupil detection device configured to form an image of a face including a pupil of the subject on the camera, and to detect a pupil by calculating an image obtained by being imaged on the camera, the light source Is provided so that the light can be emitted from within the opening of the camera, the first illumination light source including a first wavelength component in which the reflection in the pupil of the subject becomes a bright pupil, and the reflection in the pupil of the subject is a dark pupil And a second illumination light source that exhibits an illumination effect equivalent to that of the first illumination light source except for the pupil, and includes an independent first light emission source and a second light emission source, respectively. Lens means with optical axis arranged close to parallel Further, the optical path is changed so that the parallel optical axis close to the optical axis of the camera means coincides with the optical path forming means, and the camera receives the first image data from the first illumination light source. First image data obtaining means for obtaining, and second image data obtaining means for obtaining second image data obtained by the second illumination light source, wherein the computing means comprises the first image data and the second image data. This is means for calculating the image data and detecting the pupil.

According to a seventh aspect of the present invention, in the pupil detection method according to the present invention, the image of the face including the pupil of the subject is formed on the camera by irradiating the subject's face with light from the light source, and then formed on the camera. A pupil detection method for detecting a pupil by calculating an image obtained by the calculation means, wherein the light source causes a reflection in the pupil of the subject to be a bright pupil, and in the subject's pupil The reflection is a dark pupil, and the second wavelength component exhibiting an illumination effect equivalent to that of the first wavelength component is irradiated from the inside of the opening of the camera except for the pupil, and the light of the independent first wavelength component Combining the optical axes of the light of the second wavelength components so as to maintain a common optical axis, and further changing the optical path so that the common optical axis matches the optical axis of the camera means; According to the first wavelength component Obtaining the first image data; obtaining the second image data based on the second wavelength component by the camera; and calculating the first image data and the second image data by the computing means. And a step of calculating and detecting a pupil.
In the pupil detection method according to claim 8 of the present invention, an image of the face including the pupil of the subject is formed on the camera by irradiating the subject's face with light from a light source, and then formed on the camera. A pupil detection method for detecting a pupil by calculating an image obtained by the calculation means, wherein a first wavelength component in which the reflection in the pupil of the subject becomes a bright pupil by the light source, and in the pupil of the subject The reflection is a dark pupil, and the second wavelength component exhibiting an illumination effect equivalent to that of the first wavelength component is irradiated from the inside of the opening of the camera except for the pupil, and the light of the independent first wavelength component Condensing the light of the second wavelength component by arranging the respective optical axes in close proximity and parallel, and further changing the optical path so that the close parallel optical axis coincides with the optical axis of the camera means ; In the first wavelength component Obtaining the first image data, obtaining the second image data based on the second wavelength component by the camera, and calculating the first image data and the second image data by the computing means. And a step of calculating a pupil to detect a pupil.

The pupil detection device according to claim 3 of the present invention is the pupil detection device according to claim 1 or 2 , wherein the calculation means detects a pupil by calculating a difference between the first image data and the second image data. It is configured as follows.

According to a fourth aspect of the present invention, there is provided a pupil detection device according to the first or second aspect , wherein the camera means is a single unit of the first image data acquisition means and the second image data acquisition means. The pixel group of the same image sensor of this camera is divided and configured.

The pupil detection device according to claim 5 of the present invention is the pupil detection device according to claim 1 or 2 , wherein the camera means is configured such that the first image data acquisition means and the second image data acquisition means are different. It consists of a camera.

A pupil detection device according to a sixth aspect of the present invention is the pupil detection device according to any one of the first to fifth aspects, wherein the pupil detection device is arranged at a certain distance from the subject and at a certain distance from each other. It is a set of pupil detection devices for measuring a dimensional position.

According to the pupil detection device according to claims 1 and 2 and the pupil detection method according to claims 7 and 8 according to the present invention, it is possible to reliably detect the pupil.

According to the pupil detection device of the first aspect of the present invention, the light source keeps the optical axes of the independent first illumination light source and the second illumination light source by the optical path forming means. Furthermore, since the optical path is changed so that the common optical axis coincides with the optical axis of the camera means, a good illumination effect can be obtained.

According to the pupil detection device of the second aspect of the present invention, a good illumination effect can be obtained with a smaller illumination light source.

According to the pupil detection device of the third aspect of the present invention, the calculation means can satisfactorily extract only the pupil image by calculating the difference between the first image data and the second image data.

According to the pupil detection device of a fourth aspect of the present invention, the camera means includes a pixel group of the same image sensor in which the first image data acquisition means and the second image data acquisition means are a single camera. The apparatus is divided to reduce the size of the apparatus.

According to the pupil detection device according to claim 5 of the present invention, the camera means includes the first image data acquisition means and the second image data acquisition means configured by separate cameras, and reliable pupil detection. Is possible.

According to the pupil detection device of the sixth aspect of the present invention, a set of pupil detection devices for measuring the three-dimensional position of the pupil by a distance meter method is provided.

First, the principle of the present invention will be described with reference to the drawings and the like in comparison with the prior art.
7 and 8 are explanatory views for explaining the principle of bright pupil detection in the present invention.
As shown in FIG. 7, the light of the light source is irradiated to the eyes through the half mirror M, and the reflected light of the retina is made incident on the opening of the camera without being disturbed by the light source itself. This example shows the case of myopia. The example of FIG. 8 shows a case where the eye is focused at infinity. As in these examples, even when the pupil area is small or where the eye is in focus, the retina reflected light enters the opening of the camera.
In order to obtain a bright pupil image in a small pupil, it is desirable to install a light source in the camera opening, but in order to obtain a dark pupil image, it is more effective to install it as far as possible from the camera opening. However, on the other hand, if the light source for obtaining the bright pupil image and the light source for obtaining the dark pupil image are separated, the spectacle reflected light and the spectacle frame reflected light appear in different positions between the two images as described above. In addition, it is difficult to remove by image difference. In addition, a luminance difference is easily generated between the two on the face surface. Furthermore, at the boundary between the face and the background, there is a difference such that the illumination on one side is bright and the other side is a shadow and darkness, so that it is difficult to detect.

Therefore, in the present invention, two types of light sources having different wavelength components are used in order to cause a luminance difference only in the pupil portion even if the positions of the light sources are the same.
The first illumination light source includes a first wavelength component whose reflection in the pupil of the subject becomes a bright pupil, and the second illumination light source is a second wavelength component whose reflection in the pupil of the subject becomes a dark pupil. Except for the pupil, the same illumination effect as that of the first illumination light source is exhibited.
Note that, as described in the above-described invention described in Patent Document 3 (Japanese Patent Publication No. 2002-513176), the reflectance of retinal reflection differs between the wavelengths of 850 nm and 950 nm. Therefore, the center wavelength is basically 850 nm. Use at least two types of LEDs and LEDs having a central wavelength of 950 nm (the central wavelengths are not limited to these. 930 nm may be replaced by 970 nm, or 850 nm may be replaced by 830 nm or 880 nm) Use the fact that the reflectivity of the retina changes greatly at about 900 nm as a boundary (however, in an LED with a center wavelength of less than 850 nm, the light source itself appears to shine, which is undesirable depending on the application).

  Therefore, the bright pupil and the dark pupil are not necessarily brighter or darker than the portion other than the pupil depending on the brightness of the surroundings, but the bright pupil is relatively brighter than the dark pupil. Yes.

  The two types of LED light sources described above are installed so that they are equivalently located at the same position and so that the light source is contained in the opening as much as possible. By doing so, the spectacle reflection reflected in the camera image looks the same regardless of which wavelength of the light source is turned on. Furthermore, the pupil of the light source with a wavelength shorter than 900 nm (short wavelength light source) appears brighter than the light source with a wavelength longer than 900 nm (long wavelength light source). Of course, since the sensitivity of the camera also varies depending on the wavelength, it is ideal to balance the luminance of the portion other than the pupil by adjusting the amount of current flowing to each light source in advance. By doing in this way, in the difference image, only the pupil is lifted and it is easy to detect. The same applies to the light reflected from the front surface, the back surface, or the frame of the spectacle lens.

  Here, differences from the method described in Patent Document 3 will be briefly described. In the above method, since the light sources are arranged outside the opening of the camera, it is difficult to detect a small pupil. In the present invention, the light source is basically arranged in the opening of the camera (it does not mean that it should not protrude from the camera opening). Therefore, even a small pupil can be made a bright pupil, and the pupil can be detected. In the above method, since the physical positions of the light sources having different wavelengths are arranged separately, it is difficult to remove the specular reflection of the light source only by making a difference. In the present invention, as will be described later, since different light sources are installed at the same position, if the difference is made, the reflection of the glasses is automatically removed and the pupil is easily detected.

The method described in Patent Document 2 (Japanese Patent Application Laid-Open No. 2004-261598) also relates to detection of a bright pupil and a dark pupil. In this method, one of two light sources having the same or different wavelengths is installed near the optical axis of the camera, and the other is installed far from the optical axis.
Two images are created by time-division when the wavelengths are the same, and wavelength separation when the wavelengths are different, and the pupil is detected from the difference image between them. It is difficult to remove spectacles because of the different positions.
On the other hand, in the present invention, light sources of two wavelengths are basically installed in the opening, and equivalently, the light sources of the two wavelengths are the same so that the reflected spectacles cancel each other in the two different images. Install in position.
In this method, when it is desired to accurately detect the center of the pupil when the movement of the pupil is fast, the wavelength separation is used. When not, time division may be used.

  (Specific Example of Light Source) As shown in FIG. 9, two types of light sources (LEDs) 1 and 2 for L1 and L2 are installed so that their optical axes are as close to each other as parallel as possible. One packaged in a bullet mold (condenser lens) is used so that the directivity of light is almost the same. Thereby, the light source (mold) viewed from the light emission direction in the subject has the same luminance distribution regardless of which light source emits light.

  Therefore, since the reflected light of other parts of the subject illuminated with these light sources, for example, the reflection of the glasses of the subject is very similar reflected light, the reflection of the glasses in the image taken by the camera is the image The difference is removed very efficiently.

A light source having characteristics similar to those shown in FIG. 9 can be obtained by using another optical system. For example, a) Similar optical characteristics can be obtained even when concave reflection is used.
B) A Y-branch optical fiber can also be used. Light from one of the two types of short-wavelength LEDs is incident from one of the branches of the branch fiber, and light from a long-wavelength LED is incident from the other, thereby mixing and condensing the two types of wavelengths. It is also possible to illuminate the face with the lens according to the angle of view (view angle) of the camera.
C) Further, as shown in FIG. 16, the dichroic mirror M is used for the two types of light sources L1 and L2, for example, most of the light from the long wavelength light source is reflected, and most of the light from the short wavelength light source is reflected. By transmitting the light, it is possible to efficiently irradiate the eyes with both lights. In some cases, a half mirror M is also possible.
In that case, considering the wavelength-sensitivity curve of the image sensor of the camera C, the power of the light source, and the like, the reflection-to-transmission ratio of the half mirror M may be appropriately selected so as to balance the two. The light thus synthesized is introduced by a half mirror M or the like so as to be parallel to the optical axis of the camera C as shown in FIG.
(First embodiment of pupil detection)

FIG. 10 is an optical path diagram showing a first arrangement example of the first embodiment of pupil detection according to the present invention.
A subject's eyeball EB is irradiated with two wavelengths of near-infrared light through a half mirror M by a light source L12 having light emission sources with center wavelengths of 850 nm and 950 nm. The half mirror M and the light source L12 are arranged so that the optical axis of the light source L12 and the optical axis of the camera C substantially coincide with each other.
However, as described above, a combination of a short wavelength and a long wavelength across about 900 nm may be used, and is not limited to the specific wavelength described above.
Reflected light from the retina is incident on the camera C via a half mirror or a dichroic mirror M for wavelength separation at about 900 nm.
Here, when the half mirror M is used, band-pass filters F1 and F2 having the same center wavelength as the light source are arranged in the cameras C1 and C2, respectively. When the band pass filter is not used, a low pass filter F and a high pass filter F having a cutoff wavelength of about 900 nm are arranged in front of the camera. Two lights having different center wavelengths are separated as much as possible.

  The pupil is detected after subtracting the images obtained by the two cameras C1, C2. At this time, two video cameras are used as shown in FIG. 11 in order to cut external light (environmental light) as much as possible and to reduce blurring of the pupil image due to the movement of the pupil that moves due to the movement of the head. An effective method is to attach the shutter to one frame, open it for a short time (for example, 0.1 ms), and turn on the light source only during that time.

  One of the simplest implementations is to insert a multi-channel image input board into a personal computer and use a progressive camera as the video camera. A periodic external trigger signal is input to the image input board by an oscillator or the like. In accordance with the signal, an exposure signal is output from the image input board to the camera. The same signal is used as a signal for turning on the two-wavelength light source. By shortening the lighting time of the light source, the amount of current at the time of lighting can be extremely increased, and the ratio of the light amount by the light source to the light amount by the external light during the period when the shutter is open can be increased. The effect of (environmental light) can be greatly reduced.

FIG. 13 is an optical path diagram showing a second arrangement example of the first embodiment of pupil detection according to the present invention. This second arrangement example is composed of one camera. In that case, as shown in FIG. 15, band pass filters having different center wavelengths are covered in stripes on the image sensor used in the camera so as to fit the pixels. Although a vertical stripe filter is shown here, it may be a horizontal stripe or a lattice. In any case, an image difference is performed between pixels covered by neighboring bandpass filters having different center wavelengths, and a pupil is detected as a portion having a large luminance difference.
Here, instead of the band-pass filter, a high-pass filter and a low-pass filter having a cutoff wavelength of about 900 nm may be substituted. However, in that case, in order to reduce the influence of ambient light, it is desirable to attach a visible light blocking filter or a wide band-pass filter that passes about 800 nm to about 1000 nm to the camera opening.

FIG. 17 is an optical path diagram showing a third arrangement example of the first embodiment of pupil detection according to the present invention. The objective lens 19, the lens 18, and the mirrors M ₁ and M ₂ form optical path forming means. The first camera is composed of a bandpass filter F ₁ having a center wavelength near the short wavelength (850 nm), the lens 13 and the image sensor C ₁ , and the second camera has a center wavelength near the long wavelength (950 nm). Are constituted by a band pass filter F ₂ , a lens 14, and an image sensor C ₂ . Mirror M ₁ is a half mirror or a dichroic mirror, by being disposed on the optical axes of the two cameras, it is to allow the optical axis of apparently two cameras coincide substantially. A common opening of the two cameras is formed on the front surface of the lens 19. Further, a half mirror M ₂ for coupling an illumination light source is provided on the subject side of the objective lens 19.

  The light source 21 includes a plurality of light emitting elements 21 a to 21 d, branch fibers 23 a to 23 d, and a coupling unit 24. Of these light emitting elements 21a to 21d, two are long wavelength LEDs and the other two are short wavelength LEDs, and each is optically connected to the coupling portion 24 by branch fibers 23a to 23d. Here, as the light emitting elements 21a to 21d constituting the light source 21, generally, the short wavelength light source tends to have higher light emission power, so one is for the short wavelength and the remaining three are for the long wavelength. Also good.

When four light emitting elements 21a~21d emits light, the light of the two wavelength components are combined at the junction 24 is incident toward the half mirror M ₂ by. As a result, the light of the two wavelength components is reflected toward the subject's face by the half mirror M ₂ while keeping the common optical axis substantially coincident with the optical axis of the camera within the opening of the camera. The That is, when viewed from the face, the synthesized light is irradiated from within the opening including the common optical axis of the camera. The face image including the bright pupil image and the face image including the dark pupil image by the illumination are separated by the mirror M ₁ after passing through the half mirror M _{2, the} lens 19, and the lens 18. Then, the light including the short wavelength component passes through the band pass filter F ₁ and forms an image on the short wavelength image sensor C ₁ by the lens 13. On the other hand, the light including the long wavelength component reflected and separated by the mirror M ₁ passes through the band pass filter F ₂ and is imaged on the long wavelength image sensor C ₂ by the lens 14.

Furthermore, if for some reason you do not want to use a band-pass filter or dichroic mirror, one of the simplest implementations is to install two image input boards on one computer as shown in FIG. Insert an external trigger signal into both boards periodically, and delay the exposure time of the two cameras by providing a delay on one image input board, and at the same time, the emission timing of each light source of the two-wavelength light source A method of shifting is conceivable. If the exposure time is less than 1 ms, the difference in the exposure timing of the camera that performs the image difference is 2 ms at the maximum. Therefore, the blur due to the movement of the pupil can be almost ignored, and the center of the pupil can be detected accurately. . Of course, as described above, an external trigger signal having a time difference from the beginning may be input to the two image input boards with a time shift without providing a delay in the image input board. Note that the image difference can be performed at the time when both of the two images of the pair that performs the difference are acquired, or a method of calculating the difference as soon as possible after the video signal acquired later is transferred to the personal computer. It is done. In this way, in any case, both exposure and video output are shifted in time (asynchronous), and two images to be subjected to image difference are acquired.
(Second Embodiment of Pupil Detection)

It is not always necessary to separate the two wavelengths for applications where blurring of the pupil movement is not an issue.
As shown in FIG. 13, the light source of each wavelength of the two-wavelength light source is changed for each frame or through the half mirror M in which the light source of two wavelengths in the two-wavelength LED (L12) is placed in front of one camera. For example, in the case of an NTSC (interlace scanning) camera, alternately turn on each field, irradiate light on the eyes, and after subtracting temporally adjacent images obtained by each light source irradiation, the pupil image is Detect by image processing.
One of the simplest realization methods is to alternately switch each light source from the exposure signal shown in FIG. 11 and the light source selection signal output from the image input board using the circuit shown in FIG. A new exposure signal may be created to light up. As the light source selection signal, a video vertical synchronization signal, an exposure signal from an image input board, or the like is input to a D flip-flop, and an output thereof can be used. (The D flip-flop output alternately repeats the high level and the low level at the rising or falling timing of the input signal.)

  In this case as well, in order to reduce the influence of ambient light, it is desirable to attach a visible light blocking filter or a wide-band bandpass filter that passes about 800 nm to about 1000 nm to the camera opening.

Thus, the pupil detection device of the present invention does not necessarily require two cameras. For example, in the arrangement example shown in FIG. 17, a wide bandpass filter that passes about 800 nm to about 1000 nm or a bandpass filter that passes about 850 nm and about 950 nm is provided on the front surface of the lens 19. Instead of ₁ and C ₂ , an interlaced scan (interlaced scan) image sensor (camera) such as the NTSC system or PAL system may be fixed at an intermediate position between the lens 18 and the lens 19, or non-interlaced scanning. The image sensor (camera) may be fixed. In the case of interlaced scanning, by causing the long wavelength light source and the short wavelength light source among the light sources 21a to 21d to emit light alternately in synchronization with the generation timing of the odd field and even field in the image signal of the image sensor, The bright pupil image and the dark pupil image can be obtained separately, and the pupil is detected by subtracting the obtained odd line pixels from the even line pixels adjacent to the odd lines. In the case of non-interlaced scanning, the light source for the long wavelength and the light source for the short wavelength among the light sources 21a to 21d are alternately emitted in synchronization with the frame of the image sensor, thereby separating the bright pupil image and the dark pupil image. The pupil is detected by subtracting the corresponding pixels between the frames obtained at the obtained adjacent times. In either case, in order to increase the accuracy of pupil detection, it is preferable to set the shutter speed of the image sensor to about 0.1 ms to 1 ms and cause the light sources 21a to 21d to emit light in accordance with the shutter timing. It is.
(Third embodiment for performing pupil detection and pupil three-dimensional position measurement)

In order to detect the subject's line of sight with high accuracy, the three-dimensional position information of the pupil is important. In addition, the three-dimensional position of the pupil is an important information source in the field of human interface because it is known where the subject is looking from.
Stereo measurement is generally used to measure the three-dimensional position of the pupil. Therefore, two or more optical systems shown in FIGS. 10, 13, 17 and the like are required. Specifically, the two systems of pupil detection devices having such an optical system are arranged at a position facing the subject and spaced apart from each other by a certain distance and also from each other by a certain distance. The three-dimensional position of the pupil can be measured by stereo measurement based on the two-dimensional position of the pupil detected by each pupil detection device.

Originally, an invisible LED is used for the light source itself, and if the wavelength range has a sensitivity that the camera can withstand, the center wavelength is in the range of 850 nm to 950 nm. Under such circumstances, in order to optically separate two or more systems after using two wavelengths in each of two or more optical systems, a fairly narrow bandpass filter is required. Separation by a filter is very difficult. Therefore, it is appropriate to select a time division method. However, the light sources having two central wavelengths in each system may emit light at the same time to separate the wavelengths.
In that case, an optical system using the two cameras shown in FIGS. 10 and 17, or a band pass filter having center wavelengths of 850 nm and 950 nm, respectively, in adjacent pixels as shown in FIG. 15 in FIG. In an optical system using one camera using an attached image sensor, two types of central wavelength light sources of two wavelength light sources attached to each of two or more optical systems emit light simultaneously. It is only necessary to cause the system to emit light with a slight shift in time, and simultaneously open the shutter of the camera to which each light source is attached.

  According to the pupil detection device of the present invention, the pupil can be accurately detected. By using this, it can be used in the field of human interface development in which the amount of movement of the pupil corresponds to the movement of the cursor on the personal computer screen. In addition, because pupil detection is possible even in extremely different lighting environments, it can be used in the industrial field of equipment development for driving assistance that detects the eyes of passengers and truck drivers, detects drowsiness, and detects other driving. Can also be used.

It is explanatory drawing which shows the positional relationship of the eyeball, light source, and camera for demonstrating the reason a bright pupil is hard to be detected. It is explanatory drawing of the same concept as FIG. 1 in a hyperopic state. It is explanatory drawing of the same concept as FIG. 1 in a myopia state. It is explanatory drawing of the same concept as FIG. 1 in the state which has arrange | positioned the light source near the opening of a camera. It is explanatory drawing of the same concept as FIG. 1 for demonstrating the problem when a pupil is small. It is explanatory drawing of the same concept as FIG. 1 for demonstrating the problem in case a pupil is small and the light source is arrange | positioned in the opening of a camera. It is explanatory drawing for demonstrating the principle of the bright pupil detection in this invention. It is another explanatory view for explaining the principle of bright pupil detection in the present invention. It is a figure which shows the specific example of the 1st, 2nd illumination light source used with the pupil detection apparatus by this invention. It is an optical path diagram which shows the pupil detection apparatus by this invention. It is a time chart which shows the output timing of an illumination light source and a camera. It is another time chart which shows the output timing of an illumination light source and a camera. It is an optical path diagram which shows the other pupil detection apparatus by this invention. It is a circuit diagram which shows the timing generation circuit of a light source selection signal and an exposure period. It is a figure which shows the image sensor of a camera. It is a figure which shows the other specific example of the 1st, 2nd illumination light source used with the pupil detection apparatus by this invention. It is an optical path diagram which shows the other pupil detection apparatus by this invention.

Explanation of symbols

L (L1, L2, L12) , 21a~21d ... light source, C (C1, C2) ... camera, EB ... _{eyeball, M, M 1, M 2} ... ( dichroic half) mirror, F (F1, F2) ... filter.

Claims (8)

A camera, a light source, an optical path forming means, and a computing means, wherein the optical path forming means irradiates the subject's face with light from the light source, and forms an image of the face including the pupil of the subject on the camera A pupil detection device configured to detect a pupil by calculating an image obtained by being imaged on the camera,
The light source is
The light is provided so as to be radiated from the opening of the camera,
A first illumination light source including a first wavelength component in which the reflection in the pupil of the subject becomes a bright pupil;
The subject's intra-pupil reflection includes a second wavelength component that becomes a dark pupil, and includes a second illumination light source that exhibits an illumination effect equivalent to that of the first illumination light source except for the pupil,
The optical axes of the independent first illumination light source and the second illumination light source are combined by the optical path forming means so as to maintain a common optical axis, and the common optical axis coincides with the optical axis of the camera means. The optical path is changed to
The camera
First image data acquisition means for obtaining first image data from the first illumination light source;
Second image data acquisition means for obtaining second image data from the second illumination light source,
The pupil detection device is a means for detecting the pupil by performing arithmetic processing on the first image data and the second image data. A camera, a light source, an optical path forming means, and a computing means, wherein the optical path forming means irradiates the subject's face with light from the light source, and forms an image of the face including the pupil of the subject on the camera A pupil detection device configured to detect a pupil by calculating an image obtained by being imaged on the camera,
  The light source is
The light is provided so as to be radiated from the opening of the camera,
A first illumination light source including a first wavelength component in which the reflection in the pupil of the subject becomes a bright pupil;
The subject's intra-pupil reflection includes a second wavelength component that becomes a dark pupil, and includes a second illumination light source that exhibits an illumination effect equivalent to that of the first illumination light source except for the pupil,
The independent first light source and the second light source are arranged so that their optical axes are close to each other in parallel and are condensed by the lens means, and the near parallel optical axis coincides with the optical axis of the camera means by the optical path forming means. It is configured by changing the optical path as
  The camera
First image data acquisition means for obtaining first image data from the first illumination light source;
Second image data acquisition means for obtaining second image data from the second illumination light source,
  The pupil detection device is a means for detecting the pupil by performing arithmetic processing on the first image data and the second image data. Said calculating means pupil detection device according to claim 1 or 2, wherein being configured to detect a pupil by differential operation of the second image data and the first image data. 3. The camera unit according to claim 1 or 2 , wherein the first image data acquisition unit and the second image data acquisition unit are configured by dividing a pixel group of the same image sensor of a single camera. Pupil detection device. The pupil detection device according to claim 1 or 2 , wherein the camera means includes the first image data acquisition means and the second image data acquisition means configured by separate cameras. A predetermined distance from the subject, are spaced apart from each other a predetermined distance, each of which detects a pupil, a set of pupil detection device according to claim 1 to 5, wherein for measuring the three-dimensional position of the pupil.   By irradiating the subject's face with light from a light source, an image of the face including the pupil of the subject is formed on the camera, and then an image obtained by forming the image on the camera is calculated by the calculation means. A pupil detection method for detecting a pupil,
  The light source causes a first wavelength component in which the reflection in the pupil of the subject becomes a bright pupil, and a reflection in the pupil of the subject becomes a dark pupil. And the optical axes of the independent first wavelength component light and the second wavelength component light are combined so as to maintain a common optical axis, and further, Changing the optical path so that the common optical axis coincides with the optical axis of the camera means;
  Obtaining first image data of the first wavelength component by the camera;
  Obtaining second image data of the second wavelength component by the camera;
  A step of calculating the first image data and the second image data by the calculating means to detect a pupil;
A pupil detection method comprising: By irradiating the subject's face with light from a light source, an image of the face including the pupil of the subject is formed on the camera, and then an image obtained by forming the image on the camera is calculated by the calculation means. A pupil detection method for detecting a pupil,
The light source causes a first wavelength component in which the reflection in the pupil of the subject becomes a bright pupil, and a reflection in the pupil of the subject becomes a dark pupil, and the second has a lighting effect equivalent to that of the first wavelength component except for the pupil. The first wavelength component light and the second wavelength component light are condensed with their respective optical axes arranged close to each other in parallel, and further closer to each other. Changing the optical path so that the parallel optical axis coincides with the optical axis of the camera means ;
Obtaining first image data of the first wavelength component by the camera;
Obtaining second image data of the second wavelength component by the camera;
A step of calculating the first image data and the second image data by the calculating means to detect a pupil;
A pupil detection method comprising:

Images