JP6370168B2 - Illumination imaging apparatus and line-of-sight detection apparatus including the same - Google Patents

Description

  The present invention relates to an illumination imaging apparatus that drives an imaging element by a rolling shutter system, and a line-of-sight detection apparatus including the illumination imaging apparatus.

  The gaze point detection method described in Patent Literature 1 generates a face image of a subject as a bright pupil image and a dark pupil image using two or more cameras and a light source provided outside the opening of these cameras. Based on these images, a vector from the corneal reflection point of the subject to the pupil on a plane perpendicular to the reference line connecting the camera and the pupil is calculated, and the target for the reference line of each camera is calculated based on this vector. The direction of the person's line of sight is calculated using a predetermined function.

  Further, by correcting the function so that the direction of the line of sight calculated corresponding to each camera is close, and calculating the direction of the line of sight using the corrected function to obtain the intersection of the line of sight on a predetermined plane A gazing point on a predetermined plane of the subject can be detected.

  In the gazing point detection method described in Patent Document 1, a light emitting element that outputs light having different wavelengths is provided as the light source, and by alternately emitting light from these light emitting elements, one light emitting element is used. A bright pupil image is generated when illumination light is irradiated to the subject's eyes, and a dark pupil image is generated when illumination light is irradiated by the other light emitting element.

International Publication 2012/020760

  Generally, there are a global shutter system and a rolling shutter system for driving an image sensor of a camera. In the global shutter method, all pixels on one screen (frame) are exposed simultaneously, whereas in the rolling shutter method, exposure is performed in order from the upper side in the vertical direction of the screen. The pixel exposure timing varies depending on the position.

  Further, in the line-of-sight detection, in order to synchronize the emission of the detection light from the light source to the subject and the exposure in the image sensor, generally, using the VSYNC signal (vertical synchronization signal) of the image sensor as a trigger, Control is performed to switch the wavelength of light emitted from the light source. Such control does not affect the global shutter method that exposes all pixels on one screen at the same time, but when driven by the rolling shutter method, the exposure timing differs depending on the vertical position of the screen. The image before switching the light source and the image after switching are mixed in the screen for one frame acquired from the element. For example, when the wavelength of the detection light is changed by switching the light source, the images picked up before and after the switching are different, so that a pupil image capable of detecting the line of sight cannot be obtained, which hinders detection. There was a risk of it coming.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an illumination imaging device that includes an imaging device that is driven by a rolling shutter system, and that can obtain an image suitable for eye-gaze detection using the imaging device, and a gaze detection device including the illumination imaging device. And

In order to solve the above-described problem, an illumination imaging apparatus for detecting a pupil center and a center of corneal reflected light by applying detection light to the subject's eye of the present invention includes a light source that provides detection light to the subject's eye, A light source control unit that controls the light source so that the detection light is switched and emitted at a predetermined period, and an image sensor in which a plurality of pixels are arranged in a horizontal direction and a vertical direction, and the image sensor is driven by a rolling shutter system By doing so, the imaging element that acquires an image including the eyes of the subject and the imaging element is driven for at least two frames by the rolling shutter method during one detection light emission period, and the same detection light is obtained. And an exposure control unit that acquires only an image when it is irradiated.

  As a result, images before and after the periodic switching of the detection light are not acquired together, and an image for one frame irradiated with the same detection light can be acquired. A suitable image can be obtained. Furthermore, it is possible to provide an inexpensive illumination imaging apparatus by adopting a rolling shutter system as the imaging element driving system.

In the illumination imaging device of the present invention, the light source further switches and emits the detection light having the first wavelength and the detection light having the second wavelength longer than the first wavelength, and the light source control unit includes the predetermined period. each is characterized in that emit by switching detection light of detection light and the second wavelength of the first wavelength.

Alternatively , in the illumination imaging apparatus according to the present invention, the light source further includes a first light source located at a position close to the imaging element, and a second light source located at a position away from the first light source with respect to the imaging element. Prepared,
The light source control section, in the predetermined period, is characterized in that emit by switching a detection light and the first light source from the second light source.

  In the above means, it is possible to acquire a bright pupil image and a dark pupil image by selecting a light source located at a position distant from the light source located close to the image sensor, so that the first light source and the second light source are obtained. Can be composed of the same type of light source, thereby increasing the degree of freedom in selecting the light source and contributing to a reduction in component costs.

In the illumination imaging apparatus of the present invention, the imaging element is preferably a CMOS.
Thereby, an image pick-up element can be comprised at low cost.

  The line-of-sight detection device of the present invention calculates the pupil center from one of the illumination imaging devices described above, a pupil image extraction unit that extracts a pupil image from an image acquired by the imaging body, and a pupil image extracted by the pupil image extraction unit Pupil center calculation unit, corneal reflection light center detection unit for detecting the center of the corneal reflection light from the pupil image extracted by the pupil image extraction unit, and calculation of the gaze direction of the subject from the pupil center and the center of the corneal reflection light And a line-of-sight direction calculating unit.

  As a result, images with different detection lights are not mixed in one frame, so that accurate line-of-sight detection can be performed.

  According to the present invention, during a single detection light emission period, the image pickup device is driven by at least two frames by the rolling shutter method, and only images when the same detection light is given are acquired periodically. Since images before and after the detection light switching performed are not mixedly acquired, an image suitable for eye gaze detection can be obtained.

It is a front view which shows the structure of the gaze detection apparatus containing the illumination imaging device which concerns on embodiment of this invention. It is a block diagram which shows the structure of the gaze detection apparatus containing the illumination imaging device which concerns on embodiment of this invention. It is a top view which shows typically the relationship between the direction of the eyes | visual_axis of a subject's eye, and a camera. It is explanatory drawing for calculating the direction of eyes | visual_axis from the pupil center and the center of corneal reflected light. It is a figure which shows typically the image acquisition timing from an image pick-up element, and the light emission timing of a light source. It is a figure which shows typically the comparative example of the image acquisition timing from an image pick-up element, and the light emission timing of a light source.

  Hereinafter, an illumination imaging apparatus and a line-of-sight detection apparatus according to embodiments of the present invention will be described in detail with reference to the drawings.

<Structure of gaze detection device>
FIG. 1 is a front view illustrating a configuration of a line-of-sight detection device including the illumination imaging apparatus of the present embodiment, and FIG. 2 is a block diagram illustrating a configuration of the line-of-sight detection device including the illumination imaging apparatus of the present embodiment.

  The illumination imaging apparatus according to the present embodiment includes a first light source 11 and a second light source 21, two light source control units 31 and 32, a first camera 12 and a second camera 22 as an imaging body, as illustrated in FIG. An exposure control unit 33.

  Further, as shown in FIG. 2, the line-of-sight detection device of the present invention includes two image receiving devices 10 and 20 and a calculation control unit CC, such as an instrument panel or an upper part of a windshield in a vehicle interior. It is installed so as to face the driver's face as the subject.

  As shown in FIG. 1, the two image receiving devices 10 and 20 are arranged such that the optical axes 12C and 22C of the cameras 12 and 22 included in each of them are separated by a predetermined distance L10. The cameras 12 and 22 have, for example, CMOS (complementary metal oxide semiconductor) as an image sensor. This image sensor is driven by the exposure shutter 33 in a rolling shutter system, and acquires an image of a face including the driver's eyes. In the image sensor, light is detected by a plurality of pixels arranged in the horizontal direction and the vertical direction.

  As shown in FIGS. 1 and 2, the image receiving device 10 includes a first light source 11 and a first camera 12, and the first light source 11 includes twelve LED (light emitting diode) light sources 111. These LED light sources 111 are arranged outside the lens 12L of the camera 12 so that their optical axis 111C and the optical axis 12C of the camera 12 are separated by a distance L11. The image receiving device 20 includes a second light source 21 and a second camera 22, and the second light source 21 includes twelve LED light sources 211. These LED light sources 211 are arranged outside the lens 22L of the second camera 22 so that their optical axis 211C and the optical axis 22C of the second camera 22 are separated by a distance L21.

  In addition, in the cameras 12 and 22, it is preferable to arrange a band pass filter that matches the wavelength of the detection light emitted from the light sources 11 and 21. Thereby, the extraction of the pupil image in the pupil image extraction unit 40 and the calculation of the gaze direction in the gaze direction calculation unit 45 can be performed with high accuracy.

  The LED light source 111 of the first light source 11 and the LED light source 211 of the second light source 21 emit infrared light having the same wavelength as detection light, for example, infrared light (near infrared light) having a wavelength of 850 nm. It arrange | positions so that detection light can be given to a subject's eyes. Here, 850 nm is a wavelength with a low light absorption rate in the eyeball of the human eye, and this light is easily reflected by the retina at the back of the eyeball.

  The distance L11 between the optical axes of the first camera 12 and the LED light source 111 is the distance L10 between the optical axes of the first camera 12 and the second camera 22 in consideration of the distance between the visual line detection device and the driver as the subject. In contrast, it is sufficiently short. Therefore, the first light source 11 can be regarded as having substantially the same optical axis as the first camera 12. Similarly, since the distance L21 between the optical axes of the second camera 22 and the LED light source 211 is sufficiently shorter than the distance L10 between the optical axes of the first camera 12 and the second camera 22, the second light source 21 is It can be considered that the optical axes of the second camera 22 are substantially coaxial.

  On the other hand, since the distance L10 between the optical axes of the first camera 12 and the second camera 22 is sufficiently long, the optical axes of the first light source 11 and the first camera 12, the second light source 21 and the second light source 21 The optical axes of the two cameras 22 are not coaxial. In the following description, the above arrangement may be expressed as two members being substantially coaxial and the like, and the two members being non-coaxial.

  The arithmetic control unit CC is composed of a CPU and a memory of a computer, and the function of each block shown in FIG. 2 is calculated by executing software installed in advance.

  The arithmetic control unit CC includes a light source control unit 31, 32, an exposure control unit 33, an image acquisition unit 34, 35, a pupil image extraction unit 40, a pupil center calculation unit 43, and a corneal reflection light center detection unit 44. And a line-of-sight direction calculation unit 45 are provided.

  The light source control unit 31 and the light source control unit 32 perform control so that detection light is emitted alternately from the first light source 11 and the second light source 21 at predetermined intervals in accordance with an instruction signal from the exposure control unit 33.

  The exposure control unit 33 drives the image sensor by at least two frames by the rolling shutter method in the emission period of the detection light from either the first light source 11 or the second light source 21. At this time, the same detection light is transmitted. Get only the image being illuminated.

  For example, as shown in FIG. 5, when the same light source is turned on, the image data of the first frame is discarded, and the image data of the second frame is acquired for each line, so that the same detection light is always obtained. An illuminated image can be obtained.

  More specifically, focusing on the horizontal synchronization signal that determines the timing for acquiring an image for each line, when one light source is turned on, the horizontal synchronization signal is output twice from the camera. By acquiring image data obtained with pixels for one line exposed between immediately after the horizontal synchronization signal and immediately before the second horizontal synchronization signal, an image irradiated with the same detection light is always obtained. Can do.

  In this method, an image having a data amount of two frames is acquired when the same light source is turned on. However, since only one frame of image data is used, one unused frame is used. The image data is deleted after being acquired by the image acquisition units 34 and 35 or is overwritten with image data acquired thereafter by the image acquisition units 34 and 35.

  The images acquired by the image acquisition units 34 and 35 are read into the pupil image extraction unit 40 for each frame. The pupil image extraction unit 40 includes a bright pupil image detection unit 41 and a dark pupil image detection unit 42. The bright pupil image detection unit 41 detects an eye image when the combination of the light source and the camera satisfies one of the following bright pupil photographing conditions (a). The dark pupil image detection unit 42 detects the following dark pupils. An image of the eye when the combination of the light source and the camera satisfies any one of the shooting conditions (b) is detected.

(A) Bright pupil photographing conditions (a-1) During the lighting period of the first light source 11, an image is acquired by a first camera 12 substantially coaxial with the first light source 11,
(A-2) During the lighting period of the second light source 21, an image is acquired by a second camera 22 substantially coaxial with the second light source 21,
(B) Dark pupil photographing condition (b-1) During the lighting period of the first light source 11, an image is acquired by the second camera 22 that is non-coaxial with this.
(B-2) During the lighting period of the second light source 21, an image is acquired by the first camera 12 that is non-coaxial with the second light source 21,

<Light pupil image and dark pupil image>
FIG. 3 is a plan view schematically showing the positional relationship between the direction of the line of sight of the eye 50 of the subject and the image receiving devices 10 and 20. FIG. 4 is an explanatory diagram for calculating the direction of the line of sight from the center of the pupil and the center of the corneal reflected light. 3A and 4A, the line-of-sight direction VL of the subject is directed to the middle between the image receiving device 10 and the image receiving device 20, and in FIGS. 3B and 4B, the line of sight The direction VL is directed toward the optical axis 12C of the camera.

  The eye 50 has a cornea 51 in the front, and a pupil 52 and a crystalline lens 53 are located behind the cornea 51. And the retina 54 exists in the last part.

  Since the wavelength 850 nm of the first light source 11 has a low absorptance in the eyeball reaching the retina 54, light of this wavelength is easily reflected by the retina 54. When the first light source 11 provided in the image receiving device 10 is turned on, infrared light reflected by the retina 54 is detected through the pupil 52 in an image acquired by the first camera 12 substantially coaxial with the first light source 11. The pupil 52 appears bright. This image is extracted as a bright pupil image by the bright pupil image detection unit 41. The same applies to the image acquired by the second camera 22 that is substantially coaxial with the second light source 21 provided in the image receiving device 20 when it is turned on.

  On the other hand, when the first light source 11 provided in the image receiving device 10 is turned on, when an image is acquired by the second camera 22 that is non-coaxial with the first light source 11 provided in the image receiving device 20, Since the infrared light reflected by the retina 54 hardly enters the second camera 22, the pupil 52 appears dark. Therefore, this image is extracted by the dark pupil image detection unit 42 as a dark pupil image. The same applies to an image acquired by the non-coaxial first camera 12 provided in the image receiving device 10 when the second light source 21 of the image receiving device 20 is turned on.

  The pupil image extraction unit 40 subtracts the dark pupil image detected by the dark pupil image detection unit 42 from the bright pupil image detected by the bright pupil image detection unit 41, and the pupil image signal in which the shape of the pupil 52 is brightened. Is generated. This pupil image signal is given to the pupil center calculation unit 43. In the pupil center calculation unit 43, the pupil image signal is subjected to image processing and binarized, and an area image corresponding to the shape and area of the pupil 52 is calculated. Further, an ellipse including this area image is extracted, and the intersection of the major axis and the minor axis of the ellipse is calculated as the center position of the pupil 52. Alternatively, the center position of the pupil 52 may be calculated from the luminance distribution of the pupil image.

  The dark pupil image signal detected by the dark pupil image detection unit 42 is given to the corneal reflection light center detection unit 44. The dark pupil image signal includes a luminance signal by reflected light reflected from the reflection point 55 of the cornea 51 shown in FIGS. 3 and 4. The reflected light from the reflection point 55 of the cornea 51 forms a Purkinje image, and as shown in FIG. 4, a spot image with a very small area is acquired on the imaging device of the first camera 12. In the corneal reflection light center detection unit 44, the spot image is subjected to image processing, and the center of the reflection light from the reflection point 55 of the cornea 51 is obtained.

  The pupil center calculation value calculated by the pupil center calculation unit 43 and the corneal reflection light center calculation value calculated by the corneal reflection light center detection unit 44 are given to the gaze direction calculation unit 45. The line-of-sight direction calculation unit 45 detects the direction of the line of sight from the pupil center calculated value and the corneal reflection light center calculated value.

  In the case shown in FIG. 3A, the line-of-sight direction VL of the human eye 50 is directed to the middle between the imaging optical axis 12C of the first camera 12 and the imaging optical axis 13C of the second camera 22. At this time, the center of the reflection point 55 from the cornea 51 coincides with the center of the pupil 52 as shown in FIG. On the other hand, in the case shown in FIG. 3B, the line-of-sight direction VL of the human eye 50 is directed toward the outside of the imaging optical axis 13C. At this time, as shown in FIG. 4B, the center of the pupil 52 and the center of the reflection point 55 from the cornea 51 are displaced.

  The line-of-sight direction calculation unit 45 calculates a linear distance α between the center of the pupil 52 and the center of the reflection point 55 from the cornea 51 (FIG. 4B). Further, an XY coordinate having the center of the pupil 52 as the origin is set, and an inclination angle β between the line connecting the center of the pupil 52 and the center of the reflection point 55 and the X axis is calculated. Further, the line-of-sight direction VL is calculated from the linear distance α and the inclination angle β.

  In order to calculate the gaze direction VL with high accuracy, the gaze direction calculation unit 45 needs to detect the center coordinates of the pupil 52 and the center coordinates of the reflection point 55 with high accuracy.

<Imaging operation>
In the illumination imaging device and the line-of-sight detection device configured as described above, the first light source 11 and the second light source 21 are caused to emit light alternately at predetermined intervals. During the period when the first light source 11 is lit, the first camera 12 and the second camera 22 capture images simultaneously. At this time, a bright pupil image is acquired by the first camera 12 and a dark pupil image is acquired by the second camera 22. Even during the period when the second light source 21 is lit, the first camera 12 and the second camera 22 capture images simultaneously. At this time, a dark pupil image is acquired by the first camera 12 and a bright pupil image is acquired by the second camera 22.

  Focusing on the imaging operation at this time with respect to the first camera 12, when the light emission of the first light source 11 and the second light source 21 is alternately switched, the first camera 12 continues imaging. Focusing on the second camera 22, when the light emission of the first light source 11 and the second light source 21 is alternately switched, the second camera 22 continues to image.

  The imaging operation of the first camera 12 is performed by driving the imaging device for at least two frames by the rolling shutter method in one period of light emission of the first light source 11 and acquiring the line image of the second frame, thereby obtaining the first light source 11. It is possible to obtain only a line image when is emitted. Even when the second light source 21 emits light, the image sensor of the first camera 12 is driven for at least two frames by the rolling shutter method in one period of light emission, and the second light source 21 is obtained by acquiring the line image of the second frame. 21 can obtain only a line image when light is emitted.

  The same applies to the second camera 22, and even when the image sensor is driven for at least two frames by the rolling shutter method during one light emission period of the first light source 11 and the second light source 21 emits light, During the period, the image sensor is driven by at least two frames by a rolling shutter method.

  Hereinafter, with reference to FIG. 5, an example in which the first light source 11 and the second light source 21 emit light alternately and the image sensor of the second camera 22 is driven for two frames during one light emission period of each light source. explain. FIG. 5A shows image acquisition timing from the image sensor, and FIG. 5B shows light emission periods of the first light source 11 and the second light source 21. The imaging timing in FIG. 5 is the same for the imaging device of the second camera 22.

  In FIG. 5A, H000, H100, H200, H300, H400, H500, H600, H700, and H800 respectively indicate pixel lines arranged in order from the top to the bottom in the vertical direction. . The image sensor is driven for each of these lines by a rolling shutter system. Also, VSYNC1 to VSYNC10 in FIG. 5A are vertical synchronization signals given from the cameras 12 and 22 to the image acquisition units 34 and 35. The image acquisition units 34 and 35 are synchronized with these vertical synchronization signals. Thus, image data corresponding to the pixel line of the image sensor of the corresponding camera is captured.

  A11 to A19, A21 to A29, A31 to A39, A41 to A49, A51 to A59, A61 to A69, A71 to A79, A81 to A89, and A91 to A99 correspond to the pixel lines of the image sensor. The timing for capturing image data is shown, which means a horizontal synchronization signal.

  FIG. 5B shows detection light emission periods I11, I12, and I13 from the first light source 11, and detection light emission periods I21 and I22 from the second light source 21, respectively. In the example shown in FIG. 5B, the light emission times of the light sources 11 and 21 are the same, and light is emitted alternately at a constant cycle.

  VSYNC 1, 2, 3, 4,... Shown in FIG. 5 (A) is determined by the frame rate of the camera.

  Further, depending on the usage environment of the illumination imaging device and the line-of-sight detection device, for example, the exposure time in each frame of VSYNC1, VSYNC2, VSYNC3,... In accordance with this, the light emission times of the light sources 11 and 21 may be changed.

  In FIG. 5, in the period I11 during which the detection light from the first light source 11 is emitted, the imaging device acquires image data for two frames of VSYNC1 and VSYNC2. At the timing of VSYNC2, the image acquisition unit 34 acquires line image data for one line (or a plurality of lines) exposed immediately after the horizontal synchronization signal A11 to immediately before A21, and then immediately after the horizontal synchronization signal A12. Line image data for one line (or a plurality of lines) exposed immediately before A22 is acquired, and then one line (or a plurality of lines) exposed immediately after the horizontal synchronization signal A13 to A23. Min) of line image data is acquired, and this is repeated.

  In the period I11 during which the first light source 11 is lit, one frame of the vertical synchronization signal VSYNC1 and one frame of VSYNC2 are driven. By using the image data of VSYNC2, the first light source 11 is used. It is possible to acquire only one image for when one is on.

  This also applies to the period I21 when the second light source 21 is lit. Also at this time, driving for one frame of the vertical synchronization signal VSYNC3 and for one frame between VSYNC4 are performed. Also at this time, line image data for one line (or a plurality of lines) exposed immediately after the horizontal synchronization signal A31 to immediately before A41 at the timing of VSYNC4 is acquired, and then immediately after the horizontal synchronization signal A32 to immediately before A42. Line image data for one line (or a plurality of lines) exposed during the period is acquired, and then one line (or a plurality of lines) exposed between immediately after the horizontal synchronization signal A33 to immediately before A43. Line image data is acquired and this is repeated. By using the image data of VSYNC4, it is possible to acquire only one image when the second light source 21 is turned on for one frame.

  FIG. 6 shows a comparative example. In this comparative example, in the periods I11, I12, I13,... During which the first light source 11 is lit, the rolling shutter is driven for one frame and the periods I21, I22, I23 in which the second light source 21 is lit. ,..., The rolling shutter is driven only for one frame. In this case, in the horizontal synchronization signals B13 to B23, B24 to B24, etc., the light emission period I11 of the first light source 11 and the light emission period I21 of the second light source 21 are mixed within the exposure time for one line. Images cannot be acquired individually by the light emission of the first light source 11 and the light emission of the second light source.

With the configuration described above, the following effects are achieved according to the above embodiment.
(1) The detection light is emitted from the light sources 11 and 21 every predetermined period, and the image pickup device is driven at least two frames by a rolling shutter system in response to one detection light emission. Thus, images before and after switching of the detected light are not mixedly acquired, and an image for one frame irradiated with the same detection light is obtained. For this reason, it is possible to obtain an image suitable for eye-gaze detection in which images with different illumination conditions are not mixed.

(2) It is possible to provide an inexpensive illumination imaging apparatus by adopting a rolling shutter system as the imaging element driving system.

(3) The optical axis of the image sensor of the first camera 12 is substantially coaxial with the optical axis of the first light source 11, but is non-coaxial with the optical axis of the second light source 21, and Since the optical axis of the image sensor is substantially coaxial with the optical axis of the second light source 21 and is non-coaxial with the optical axis of the first light source 11, the optical axis of the image sensor and the light source is substantially coaxial or non-coaxial. Depending on whether or not a bright pupil image or a dark pupil image is to be detected, it is possible to select. For this reason, the 1st light source 11 and the 2nd light source 21 can be comprised with the same kind of light source, and, thereby, the freedom degree of selection of a light source increases and it can contribute to the fall of component cost.

(4) By configuring the imaging elements of the cameras 12 and 22 with CMOS, the imaging element can be configured at low cost.

(5) By acquiring a pupil image using the above-described illumination imaging apparatus, images with different detection lights do not coexist in one frame, so that accurate line-of-sight detection can be performed.

A modification will be described below.
(1) In the embodiment described above, detection light having the same wavelength is emitted alternately or simultaneously from the first light source 11 and the second light source 21, and an image for a bright pupil is detected from an image photographed by a camera substantially coaxial with the light source. A dark pupil image was detected from an image taken with a non-coaxial camera. On the other hand, the detection light from the first light source 11 and the detection light from the second light source 21 may have different wavelengths, and an image of the subject's eyes may be acquired with a camera substantially coaxial with each light source. .

  In this case, as the wavelength of the detection light of the first light source 11 and the second light source 21, for example, the wavelength of the detection light from one light source (first wavelength) is 850 nm, and the wavelength of the detection light from the other light source ( The second wavelength is 940 nm, which is longer than the first wavelength. Here, 850 nm has a low light absorption rate in the eyeball of the human eye and is easily reflected by the retina, whereas 940 nm is a wavelength having a high light absorption rate in the eyeball of the human eye and is reflected by the retina. It is hard to be done. Note that a combination of wavelengths other than 850 nm and 940 nm may be used as long as the relationship between the light absorptivity and the light reflectance in the human eyeball or on the retina is the same. In addition, the light source and the camera are not limited to the form in which two sets are provided as in the above-described embodiment. For example, the first wavelength is detected so that one camera is provided and is substantially coaxial with the camera. A configuration in which a light source that emits light and a light source that emits detection light having the second wavelength are also possible.

  When the wavelength of the detection light from the light source is set in this way, a bright pupil image can be detected from an image acquired with a camera that is substantially coaxial with the light source that provides the detection light with a wavelength of 850 nm, and the detection light with a wavelength of 940 nm is provided. A dark pupil image can be detected from an image acquired by a camera substantially coaxial with the light source.

(2) About a light source and a camera, it is not limited to the form provided in two sets like the above-mentioned embodiment, For example, the structure which provides a substantially coaxial light source and a non-coaxial light source with respect to this camera as one unit | set. Alternatively, a configuration in which a single light source is provided and a substantially coaxial camera and a non-coaxial camera are provided for the light source is also possible.

  Although the present invention has been described with reference to the above embodiment, the present invention is not limited to the above embodiment, and can be improved or changed within the scope of the purpose of the improvement or the idea of the present invention.

  As described above, the illumination imaging apparatus according to the present invention is useful for an apparatus in which an imaging element is driven by a rolling shutter system and illumination conditions change periodically.

DESCRIPTION OF SYMBOLS 10, 20 Image receiving device 11 1st light source 12 1st camera 12C optical axis 21 2nd light source 22 2nd camera 22C optical axis 31, 32 Light source control part 33 Exposure control part 34, 35 Image acquisition part 40 Pupil image extraction part 41 Bright Pupil image detection unit 42 Dark pupil image detection unit 43 Pupil center calculation unit 44 Cornea reflection light center detection unit 45 Gaze direction detection unit 50 eyes

Claims (4)

In an illumination imaging device that provides detection light to the subject's eye to detect the center of the pupil and the cornea reflected light,
A light source that provides detection light to the subject's eyes;
A light source controller that controls the light source to switch and emit the detection light at a predetermined period;
An imaging body that includes an imaging device in which a plurality of pixels are arranged in a horizontal direction and a vertical direction, and obtains an image including the eyes of the subject by driving the imaging device in a rolling shutter system;
An exposure control unit that drives the image pickup device for at least two frames by the rolling shutter method in one emission period of the detection light, and acquires only an image when the same detection light is irradiated;
With
The light source emits detection light having a first wavelength and detection light having a second wavelength longer than the first wavelength,
The light source control unit, lighting imaging device you characterized in that emit by switching detection light of detection light and the second wavelength of the first wavelength in the predetermined period. In an illumination imaging device that provides detection light to the subject's eye to detect the center of the pupil and the cornea reflected light,
A light source that provides detection light to the subject's eyes;
A light source controller that controls the light source to switch and emit the detection light at a predetermined period;
An imaging body that includes an imaging device in which a plurality of pixels are arranged in a horizontal direction and a vertical direction, and obtains an image including the eyes of the subject by driving the imaging device in a rolling shutter system;
An exposure control unit that drives the image pickup device for at least two frames by the rolling shutter method in one emission period of the detection light, and acquires only an image when the same detection light is irradiated;
With
The light source includes a first light source located near the image sensor and a second light source located farther from the first light source than the image sensor,
The light source control unit is configured in a predetermined cycle, the first light source and lighting imaging device you characterized in that emit by switching the detection light from the second light source. The illumination imaging apparatus according to claim 1 , wherein the imaging element is a CMOS. The illumination imaging device according to any one of claims 1 to 3 ,
A pupil image extraction unit that extracts a pupil image from an image acquired by the imaging body;
A pupil center calculation unit that calculates the pupil center from the pupil image extracted by the pupil image extraction unit;
A corneal reflection light center detection unit for detecting the center of the corneal reflection light from the pupil image extracted by the pupil image extraction unit;
A line-of-sight direction calculation unit that calculates the line-of-sight direction of the subject from the center of the pupil and the center of the corneal reflected light;
A line-of-sight detection device comprising:

Images