JP2001258845A - Visual line detector - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a visual line detector tolerant to an individual difference such as the eyelash form of an observer, the way of looking through an observ ing surface or the like and capable of performing a precise visual line detection. SOLUTION: In this detector, an imaging optical system for forming the eye image of the eye of the observer forms a plurality of eye images having a parallax. An arithmetic means for calculating the watching point of the observer from the eye images calculates the watching point by use of an eye image composed on the basis of the eye images (#301-#311, #316 and #317).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an improvement in a gaze detecting device used for a camera or the like.

[0002]

2. Description of the Related Art Conventionally, there has been known a technique in which the direction of a line of sight of an observer observing a finder is detected and the result is used for selecting a focus detection point of a camera or a distance measuring point. The gaze detection device used here is disclosed in, for example, Japanese Patent Laid-Open No. 6-1877.
No. 2, which irradiates infrared light to the observer's eye and forms an eye image from the returned reflected light;
Gaze detection is performed based on the eye image data.

[0003] Usually, in an eye image, in addition to the eyelids and the pupils, a first Purkinje image, which is a reflection image of a light source on the cornea, is recognized.
Since the positional relationship between the pupil and the first Purkinje image changes according to the rotation of the eyeball, it is possible to detect the gazing point by extracting information on the pupil and the first Purkinje image from the eye image.

Further, a technique using a plurality of eye images having parallax in order to further improve detection accuracy is known. JP-A-2-134130 and JP-A-8-1409
No. 37 is an example of this, in which the eye of the finder observer is imaged from two directions, and from the information of the pair of eye images, the real space coordinates of the first Purkinje image and the real space coordinates of the pupil center are obtained. Furthermore, after calculating the real space coordinate table of the corneal curvature center,
This is to guide the fixation point coordinates.

[0005]

However, in the conventional gaze detecting device, the pupil and the first Purkinje image cannot always be captured simultaneously and favorably as the eye image, and as a result, the detection error increases. For this reason, they often incorrectly output a position different from the actual point of gaze.

The cause is often that the pupil is partially hidden by eyelashes. Further, there is a case where the first Purkinje image cannot be detected because the observer looks obliquely into the finder. Further, the contour of the pupil may appear distorted due to a ghost generated in the finder optical system or the glasses, or the position of the first Purkinje image may appear shifted.

[0007] Such a problem is described in Japanese Patent Application Laid-Open No. H08-1409.
Even in the eye gaze detecting device shown in No. 37 which captures the eye from two directions, if the pupil and the first Purkinje image are not captured in each of the pair of images, an error will occur in the subsequent processing. Occurs.

(Aim of the Invention) A first object of the present invention is to tolerate individual differences such as eyelashes of an observer and how to look at an observation surface, and it is possible to detect a gaze with high accuracy. It is an object to provide a line-of-sight detection device.

A second object of the present invention is to provide an eye-gaze detecting device which makes it difficult for an eye-gaze to be detected even when an observer looks into the observation surface from an oblique direction. .

[0010]

In order to achieve the first object, the invention according to claim 1 provides an image forming optical system for forming an eye image of an observer's eye, and an image forming apparatus for observing the image from the eye image. A gaze detecting device having a calculating means for calculating a gaze point of a person, wherein the imaging optical system forms a plurality of eye images having parallax, and the calculating means also generates the plurality of eye images. And a gaze detection device that calculates a gazing point using the eye image synthesized with the above.

[0011] Similarly, in order to achieve the first object,
According to a fourth aspect of the present invention, in the eye-gaze detecting apparatus having an imaging optical system that forms an eye image of an observer's eye, and an arithmetic unit that calculates a gazing point of the observer from the eye image, The optical system is to form a plurality of eye images having parallax, and the calculating means obtains an eye image synthesized by arithmetic averaging of one of the plurality of eye images and the other image, A gaze detection device is to calculate a gazing point using the synthesized eye image.

[0012] Similarly, in order to achieve the first object,
According to a fifth aspect of the present invention, in the visual line detection device, the image forming optical system includes: an imaging optical system that forms an eye image of an observer's eye; and a calculation unit that calculates a gazing point of the observer from the eye image. The optical system is configured to form a plurality of eye images having parallax, and the calculating unit obtains an eye image synthesized by a geometric mean of one of the plurality of eye images and the other image. This is a gaze detection device that calculates a gazing point using the synthesized eye image.

Further, in order to achieve the second object,
According to a sixth aspect of the present invention, in the eye-gaze detecting apparatus having an imaging optical system that forms an eye image of an observer's eye, and a calculating unit that calculates a gazing point of the observer from the eye image, An optical system for forming a plurality of eye images having parallax, and the calculating unit calculates a gazing point using an eye image synthesized based on the plurality of eye images; and And a gaze detection device having a second calculation mode for calculating a gazing point using one eye image.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail based on illustrated embodiments.

FIG. 1 is a diagram showing a schematic configuration of a digital color camera equipped with a visual axis detection device according to a first embodiment of the present invention.

This camera is a single-panel digital color camera using a solid-state imaging device such as a CCD or CMOS sensor, and continuously or spontaneously drives the solid-state imaging device to generate an image signal representing a moving image or a still image. Get. Here, the solid-state imaging device is an imaging device that converts exposed light into an electric signal for each pixel, accumulates charges corresponding to the amount of light, and reads out the charges.

In FIG. 1, reference numeral 1 denotes a camera body, and 2 denotes an imaging lens having an image forming optical system 3 therein. The imaging lens 2 is electrically and mechanically connected to the camera body 1 via a known mount. By changing to an imaging lens having a different focal length, it is possible to obtain shooting screens with various angles of view. The focusing of the object is performed by moving the focusing lens, which is a part of the imaging optical system 3, in the direction of the optical axis L1 by a driving mechanism (not shown).

4 is an optical low-pass filter, 5 is a mechanical shutter, and 6 is a solid-state image sensor. The optical low-pass filter 4 limits the cut-off frequency of the imaging optical system 3 so that an object image with an unnecessarily high spatial frequency is not formed on the solid-state imaging device 6. An infrared cut filter is also formed on the optical low-pass filter 4. The object image captured by the solid-state imaging device 6 is displayed on a liquid crystal display 7. The solid-state imaging device 6 is a CMOS process-compatible sensor (hereinafter abbreviated as a CMOS sensor), which is one of amplification type solid-state imaging devices. One of the features of the CMOS sensor is that since the MOS transistor of the area sensor portion and the MOS transistor of the peripheral circuit can be formed in the same step, the number of masks and the number of process steps can be significantly reduced as compared with the CCD. By utilizing this feature, the charges of the two photoelectric conversion units can be transferred simultaneously or separately to the floating diffusion region (FD region), and only by the timing of the transfer MOS transistor connected to the FD region,
Addition and non-addition of the signal charges of the two photoelectric conversion units can be easily performed.

The solid-state image pickup device 6 uses the above-described structure to receive a light beam from the entire exit pupil of the image pickup lens and a light output from a part of the exit pupil of the image pickup lens. And the second output mode for receiving light. The output in the first output mode is used for imaging and finder output, and the output in the second output mode is used for focus detection and finder output. In the first output mode in which signals are added at a pixel level, a signal with less noise can be obtained as compared with a method in which signals are read and then added.

Reference numerals 8 and 9 denote a concave lens and a convex lens for observing the liquid crystal display 7, which have a positive overall power. Reference numeral 10 denotes an eyeball illumination prism which also serves to protect the finder optical system. The optical path portion is formed of a parallel plate and serves as a window member of a finder optical system. These constitute a finder optical system. By moving the convex lens 9 along the finder optical axis L2, the diopter is adjusted, and the liquid crystal display 7 suitable for the observer is adjusted.
Can be obtained. 12 and 13
A diffusion plate is a light diffusion means for diffusing the emitted light in a specific direction.

An LED 11 or the like, which is a light-emitting portion that emits red light having a light emission characteristic shown in FIG. 2, is attached to the side of the eyeball illumination prism 10, and the light emitted from the LED11 is an eyeball illumination prism. The light passes through the inside of the camera 10 and further passes through the diffusion plate 12 or 13 and is emitted to the outside of the camera. These are eyeball illumination systems that illuminate the observer's eye.

The central emission wavelength of the LED 11 is 720 nm.
(See FIG. 2), and the relative visibility at this wavelength is quite low. However, the emission intensity is sufficiently visible, partly because the skirt of the emission intensity extends to the shorter wavelength side. Therefore, if the observer is conscious, the operating state of the gaze detection device can be known through the emission of the LED 11.

The effective portion of the finder optical path of the concave lens 8, the convex lens 9, and the eyeball illumination prism 10 is provided with a transmission coating for a wavelength of about 760 nm from the visible region so as to include the emission wavelength of the LED 11. This is to prevent the illumination of the eye from being reflected by the finder optical system and returning to the eye again and appearing as a red ghost, and to prevent the ghost from being superimposed on the eye imaging system described below as much as possible. That's why.

A dichroic mirror 1 is provided between the liquid crystal display 7 and the concave lens 8 in the optical path of the finder optical system.
4 are arranged to split light coming from the finder in the opposite direction upward. FIG. 3 shows the spectral reflectance characteristics of the dichroic mirror 14, which shows a sharp cut characteristic with a half value at 670 nm. Therefore, the LED 11 shown in FIG.
Is reflected, and the reflected light from the observer's eye illuminated by the LED 11 is reflected by the eyeball illumination prism 10,
After passing through the convex lens 9 and the concave lens 8 in reverse, the light is reflected upward by the dichroic mirror 14.

A component having a wavelength shorter than 670 nm of the image displayed on the liquid crystal display 7 passes through the dichroic mirror 14 and is emitted through the concave lens 8, the convex lens 9, and the eyeball illumination prism 10 which is a window member of the finder optical system. Is done. At this time, strictly speaking, the component having a wavelength longer than 670 nm is cut off, but since the relative luminous efficiency is low in this wavelength region, no unnaturalness occurs in the finder image.

Reference numeral 15 above the dichroic mirror 14 is a mirror, 16 is an optical path changing prism, 17 is a compound eye lens, and 18 is a line-of-sight sensor. The reflected light from the observer's eye is bent at a right angle by a mirror 15 and then passes through an optical path changing prism 16 and a compound eye lens 17 to a line-of-sight sensor 18.
An image of the eye of the observer is formed thereon. As described below,
By obtaining an image of a pair of eyes having parallax, it is possible to tolerate individual differences in eyelash mode, how to look through a finder, and the like, and to realize highly accurate gaze detection.

Reference numeral 19 denotes a main switch, and reference numeral 20 denotes a release button.

FIG. 4 is a block diagram showing an electrical configuration of the digital color camera.

First, a description will be given of a portion relating to imaging and recording of a camera. The camera has an imaging system, an image processing system, a recording / reproducing system, and a control system.

The imaging system includes an imaging optical system 3, a mechanical shutter 5, and a solid-state imaging device 6, and the image processing system includes an A / D converter 30, an RGB image processing circuit 31, and a YC processing circuit 32. . Further, the recording / reproducing system includes:
The control system includes a recording processing circuit 33 and a reproduction processing circuit 34, and the control system includes a camera system control circuit 35, an operation detection circuit 36, and a solid-state imaging device driving circuit 37. Reference numeral 38 denotes a standardized connection terminal for transmitting and receiving data by connecting to an external computer or the like.

The image pickup system is an optical processing system for forming an image of light from an object on the image pickup surface of the solid-state image pickup device 6 via the image forming optical system 3. By further adjusting the mechanical shutter 5 in response, a solid-state image sensor 6 can be exposed to a subject image having an appropriate amount of light. The solid-state image sensor 6 has 3700 pixels in the long side direction and 2 pixels in the short side direction.
An imaging device having a total of about 10 million pixels of 800 pixels is applied, and optical filters of three primary colors of red (R), green (G), and blue (B) are arranged in a mosaic shape in front of the pixels. ing.

The image signals read from the solid-state image sensor 6 are supplied to an image processing system via the A / D converter 30. The A / D converter 30 is a signal conversion circuit that converts and outputs a 10-bit digital signal corresponding to the amplitude of the signal of each exposed pixel, for example, and the subsequent image signal processing is executed by digital processing. You.

The image processing system is a signal processing circuit for obtaining an image signal of a desired format from R, G, B digital signals, and converts the R, G, B color signals into a luminance signal Y and a color difference signal (R−G). Y) and (B-Y). The RGB image processing circuit 31 includes the A / D converter 3
“3700 × 28” received from the solid-state imaging device 6 through
A signal processing circuit that processes the image signal of the “00” pixel;
It has a white balance circuit, a gamma correction circuit, and an interpolation operation circuit for increasing the resolution by interpolation operation. The YC processing circuit 32 is a signal processing circuit that generates a luminance signal Y and color difference signals RY and BY, and generates a high-frequency luminance signal generating circuit that generates a high-frequency luminance signal YH and a low-frequency luminance signal YL. Low-frequency luminance signal generating circuit and color difference signals RY, B
-Y is generated by a color difference signal generation circuit. The luminance signal Y is formed by combining the high-frequency luminance signal YH and the low-frequency luminance signal YL.

The recording / reproducing system is a processing system for outputting an image signal to the memory and outputting an image signal to the liquid crystal display 7. The recording processing circuit 33 writes and reads the image signal to and from the memory. After performing the processing, the reproduction processing circuit 34 reproduces the image signal read from the memory and outputs the reproduced image signal to the liquid crystal display 7. Further, the recording processing circuit 33 has therein a compression / expansion circuit for compressing the YC signal representing a still image and a moving image in a predetermined compression format, and for expanding when reading the compressed data. The compression / expansion circuit includes a frame memory and the like for signal processing,
The YC signal from the image processing system is stored in this frame memory for each frame, and is read out for each of a plurality of blocks and compression-encoded. The compression encoding is performed, for example, by subjecting an image signal of each block to two-dimensional orthogonal transformation, normalization, and Huffman encoding. Reproduction processing circuit 34
Is a circuit for performing a matrix conversion of the luminance signal Y and the color difference signals RY and BY to convert them into, for example, RGB signals. The signal converted by the reproduction processing circuit 34 is output to the liquid crystal display 7, and a visible image is displayed and reproduced.

On the other hand, the control system includes a release button 20
An operation detection circuit 36 that detects operations such as an operation, a camera system control circuit 35 that controls each unit in response to the detection signal, and generates and outputs a timing signal and the like at the time of imaging.
Under the control of the camera system control circuit 35, a solid-state image sensor driving circuit 37 that generates a drive signal for driving the solid-state image sensor 6 is included.

This control system controls the imaging system, the image processing system, and the recording / reproducing system in response to an external operation. For example, when the release button 20 is pressed, the driving of the solid-state imaging device 6 and the RGB are performed. The operation of the image processing circuit 31 and the compression processing of the recording processing circuit 33 are controlled.

Next, a description will be given of the parts relating to the visual line detection and the focus adjustment.

The camera system control circuit 35 is further connected to a line-of-sight detection / AF control circuit 40 and a lens system control circuit 41. These mutually communicate data required for each processing with the camera system control circuit 35 at the center.

The line-of-sight detection / AF control circuit 40 detects the gazing point of the observer in the finder field of view from the finder observer's eye image projected on the line-of-sight sensor 18 through the finder optical system 42. As described above, the finder optical system 42 includes the concave lens 8, the convex lens 9, and the eyeball illumination prism 10 shown in FIG.

From the gazing point information in the finder visual field, a focus detection point is set on the gazing point in order to further focus the image forming optical system 3 on the object image at the gazing position, and the imaging state of this point is detected. . When the defocus is detected, it is converted into a driving amount of a focusing lens, which is a part of the imaging optical system 3, and transmitted to the lens system control circuit 41 via the camera system control circuit 35. Upon receiving the driving amount of the focusing lens, the lens system control circuit 41 moves the focusing lens in the direction of the optical axis L1 by a driving mechanism (not shown) to focus on the object on the gazing point.

When the line-of-sight detection / AF control circuit 40 detects that the object on the gazing point is in focus, this information is transmitted to the camera system control circuit 35, and the information is transmitted to the camera system control circuit 35. Imaging is permitted. At this time, if the release button 20 is pressed,
As described above, the imaging control is performed by the imaging system, the image processing system, and the recording / reproducing system.

Now, the configuration of the solid-state imaging device 6 will be described.

FIG. 5 is a circuit configuration diagram of the area sensor section in the solid-state imaging device 6. FIG. 1 shows a two-dimensional area sensor having 2 columns × 2 rows of pixels, but in actuality, it has 280 pixels.
By increasing the number of pixels to 0 columns × 3700 rows and the like, a practical resolution is obtained.

In FIG. 5, 301 and 351 are pn
A first and a second photoelectric conversion unit comprising a photodiode, 30
3 and 353 are transfer switch MOS transistors, 3
04 is a reset MOS transistor, 305 is a source follower amplifier MOS transistor, 306 is a vertical select switch MOS transistor, 307 is a source follower load MOS transistor, and 308 is a dark output transfer MOS.
Transistor, 309 is a light output transfer MOS transistor, 310 is a dark output storage capacitor C _TN , 311 is a light output storage capacitor C _TS , 312 and 354 are vertical transfer MOS transistors, 313 and 355 are vertical output line reset MOs.
The S transistor 314 is a differential output amplifier, 315 is a vertical scanning unit, and 316 is a horizontal scanning unit.

FIG. 6 is a sectional view of a light receiving section (for example, 330-11). The light receiving units 330-21 and 330-1
2, 330-22 and the like have the same structure.

In the figure, 317 is a P-type well, 31
8, 358 are gate oxide films and 320, 350 are poly S
i, 321 are n + floating diffusions (F
D) Region. Numerals 340 and 390 denote n layers, which are concentrations that can be completely depleted. The FD region 321 is connected to the first photoelectric conversion unit 30 via the transfer MOS transistors 303 and 353.
The first and second photoelectric conversion units 351 are connected. The charges generated by the control pulse φTX _e are completely transferred to the FD unit 321, and signals can be added or not added by the control pulse φTX ₀ . In FIG. 6, the first photoelectric conversion unit 3
Although the first photoelectric conversion unit 301 and the second photoelectric conversion unit 351 are drawn apart from each other, the boundary portion is actually extremely small, and the first photoelectric conversion unit 301 and the second photoelectric conversion unit 351 may be regarded as being in contact with each other in practical use. Hereinafter, the adjacent first and second photoelectric conversion units will be collectively referred to as a “light receiving unit”. The pixels including the light receiving section and the MOS transistor are laid out in a substantially square shape, and are arranged adjacent to each other in a lattice.

Reference numeral 322 denotes a color filter that transmits light of a specific wavelength, and 323 denotes a microlens for efficiently guiding a light beam from the imaging optical system 3 to the first and second photoelectric conversion units. The ratio of the light receiving section in each pixel is approximately
In order to effectively use the luminous flux emitted from the imaging optical system 3, which is about 0%, a microlens for condensing light is provided for each of the light receiving units, and light that is going to reach other than the light receiving units is provided. Must be deflected to the light receiving section.

FIG. 7 is a plan view showing the positional relationship between the microlens provided on the front surface of the solid-state image sensor 6 and the light receiving section.

In the same figure, the light receiving sections 330-21, 330-22, 330-11 described above with reference to 301 are shown.
In FIG. 7, reference numerals 330-12 denote 72-11 and 72-2.
1, 72-12, and 72-22. Microlenses 71-11 to 71-44 (corresponding to 323 in FIG. 6) are axially symmetrical spherical lenses or aspherical lenses in which the center of the light receiving section substantially coincides with the optical axis. They are densely arranged in a lattice with the light incident side being convex.

As described above, each pixel has two photoelectric conversion units. In the figure, R, G, and B denote photoelectric conversion units having red, green, and blue color filters, and 1 or 2 following R, G, and B denotes a first photoelectric conversion unit or a second photoelectric conversion unit. This indicates whether the photoelectric conversion unit is used. For example, R1 is a first photoelectric conversion unit having a red color filter, and G2 is a second photoelectric conversion unit having a green color filter.

In this area sensor section, R (red), G (green), and B (blue) color filters are alternately arranged for each pixel to form a so-called Bayer array in which four pixels constitute one set. In the Bayer array, the overall image performance is improved by arranging more G pixels that are easily felt by an observer when viewing an image than R and B pixels. Generally, in an image sensor of this type, a luminance signal is mainly generated from G, and a color signal is generated from R, G, and B.

Next, the operation of the above-described micro lens will be described.

FIG. 8 is a sectional view of the area sensor section.
An imaging optical system 3 (not shown) is located on the left side of the figure, and a light beam emitted from the imaging optical system 3 passes through the optical low-pass filter 4 and first passes through the micro lenses 71-11, 71-21,
It is incident on 71-31 and 71-41. A color filter is arranged behind each microlens, where only a desired wavelength band is selected and reaches each light receiving section from 72-11 to 72-41. The color filters constitute a Bayer array as described with reference to FIG.
There are seeds. Also, because of the Bayer arrangement, only two of them appear in the cross section, 22G being a green transmission color filter and 22R being a red transmission color filter.

The power of each microlens is set so that each light receiving portion of the solid-state imaging device 6 is projected onto the exit pupil of the imaging optical system 3. At this time, the projection magnification is set so that the projected image of each light receiving unit is larger than the exit pupil of the imaging optical system 3 when the aperture is open, and the amount of light incident on the light receiving unit and the stop ST of the imaging optical system 3 are adjusted. It is good to make the relationship with the opening degree approximately linear. Due to the action of the microlenses, in the entire solid-state imaging device, the light beam incident on the second photoelectric conversion unit passes through the upper half of the exit pupil of the imaging optical system 3 even if the light beam enters any position of the area sensor unit. It becomes a luminous flux. On the other hand, the luminous flux incident on the first photoelectric conversion unit of the entire solid-state imaging device is the imaging optical system 3.
It may be considered that the optical axis L1 is a symmetric axis and is inverted upside down.

In the above-described optical system, for example, when an object image is formed before the solid-state image sensor 6, the half light beam passing above the exit pupil is changed to the solid-state image sensor 6 in FIG.
The light beam shifts downward on the upper side, and the half beam passing below the exit pupil shifts on the upper side. That is, a pair of image signals formed by a light beam that has passed through each half of the pupil of the imaging optical system 3 has a phase shifted in the vertical direction in FIG. 8 according to the imaging state of the object image. By setting a focus detection point on the area sensor unit and examining the phase shift of a pair of image signals around this point, the imaging state of the imaging optical system 3 at this focus detection point can be known. According to this configuration, it is possible to place the focus detection point anywhere in the shooting screen.

Next, the charge storage operation of the solid-state imaging device 6 will be described.

First, during accumulation, the FD section 321 sets the control pulse φR ₀ to a high level and fixes it to the power supply V _DD in order to prevent blooming (prevent charges from leaking out when strong light is applied). . When photons hν are irradiated,
Electrons are accumulated in the pn photodiodes 301 and 351, and holes are discharged through the P-type well 317.

An energy barrier formed by the transfer MOS transistor 303 is formed between the photoelectric conversion unit 301 and the FD unit 321, and an energy barrier formed by the transfer MOS transistor 53 is formed between the photoelectric conversion unit 51 and the FD unit 21. Therefore, electrons are present in the pn photodiodes 301 and 351 during the accumulation of the photocharge. Thereafter, by scanning the horizontal scanning unit and performing the charge storage operation in the same manner, the charge is stored in all the photoelectric conversion units.

In the read state, the control pulses φTX ₀₀ and φTX _e0 are set so that the barrier below the transfer MOS transistors 303 and 353 is eliminated and the electrons of the pn photodiodes 301 and 351 are completely transferred to the FD section 321.

The readout of the focus detection image in the second output mode is as follows.

First, the control pulse φR _{0 is set} to a high level to reset the FD section 321 to the power supply V _DD ,
The dark output is stored in the storage capacitor 310 by setting S ₀ to the high level, and then the control pulse φTX _{00 is set} to the high level to set pn
The photoelectric charge accumulated in the photodiode 301 is converted into a source follower MOS transistor 305 and a selection switch MOS.
The signal is transferred to the storage capacitor 311 via the transistor 306, the noise component is canceled by the differential amplifier 314, and the image signal V _OUT of the first photoelectric conversion unit is output.

Further, the control pulse φR _{0 is set} to the high level to reset the FD section 321 to the power supply V _DD , and then the control pulse φTX _{e0 is set} to the high level so that the photocharge accumulated in the pn photodiode 351 is converted into the source follower MOS transistor. 305, selection switch MOS transistor 3
Then, the signal is transferred to the storage capacitor 311 via 06, the noise component is canceled by the differential amplifier 314, and the second photoelectric conversion unit image signal V _OUT is output.

At the same time, the transfer MOS transistor 30
If the control pulses φTX ₀₀ and φTX _e0 are set so that the barriers below 3 and 353 are eliminated and the charges of the two pn photodiodes 301 and 351 are completely transferred to the FD section 321, the charges of the first photoelectric conversion section The image signal can be read in the first output mode in which the charge of the second photoelectric conversion unit and the charge of the second photoelectric conversion unit are added.

Now, the gaze detection device will be described.

FIGS. 9 to 11 are detailed views of the eyeball illumination system. 9 is a side view, FIG. 10 is a front view of the eyeball illumination prism 10 viewed from the direction of arrow A in FIG. 9, and FIG. 11 is a developed view of an illumination optical path. Here, the expanded element is indicated by adding 'after the number attached to the corresponding element before expansion.

On the side of the eyeball illumination prism 10 (outside the visual field of the object image), the LED 11, LE
D50, LED51 and LED52 are attached. LED11, LED50, LED51, LED52
Emits red light having the emission characteristics shown in FIG. These L
Since the ED and its optical path are plane-symmetric with respect to a horizontal plane including the optical axis L2 and a vertical plane, an explanation will be given here focusing on the LED 11.

The eyeball illumination prism 10 includes surfaces 10e and 10m which are finder light beam passing surfaces, surfaces 10i, 10j, 10k and 10l which are LED light incidence surfaces, surfaces 10g and 10h which are reflection surfaces on which aluminum is deposited. It is composed of convex surfaces 10a, 10b, 10c, and 10d, which are emission surfaces of LED light. The window through which the light beam of the finder system passes is inside the wavy line 10f shown in FIG. 10 (this corresponds to the field of view of the object image), and the other surface can be formed in an optimal shape for the eyeball illumination system. .

The light emitted from the LED 11 enters the inside of the eyeball illumination prism 10 from the surface 10i of the eyeball illumination prism 10, is totally reflected by the surface 10e, is further reflected by the metal surface at the surface 10g, and is emitted from the convex surface 10a. Is done.

LED11, LED50, LED51, L
The emission wavelength of the ED 52 is located at the long wavelength side end of the visible region,
It is visible. The power of the convex surface 10a is set so that the LED 11 is located 1 m ahead in appearance, and the LED light is emitted from the convex surface 10a as light slightly diverging. That is, the convex surface 10a is an observation optical system of the LED 11. Since the observation optical system is formed integrally with the eyeball illumination prism 10 which is a window member constituting the finder optical system, the configuration becomes extremely simple, and furthermore, the interior of the eyeball illumination prism 10 is used as an illumination optical path, so that the size can be reduced. .

If it is attempted to illuminate the eye directly with the light emitted from the convex surface 10a, the emitted light diverges only slightly, so that it is possible to illuminate only the same area as the size of the convex surface 10a. LED light is applied to only a part of the LED. If the LED light shines only on a part of the eye, it is difficult to completely capture the pupil, and a decrease in the eye gaze detection accuracy is inevitable. The diffusion plate 12 is a member for solving such a problem, and has a role of illuminating the observer's eye widely and uniformly by diffusion of light.

FIGS. 12A, 12B and 12C are explanatory views of the diffusion plate 12. FIG. Specifically, FIG. 12A is a front view of the diffusion surface 12a, FIG. 12B is a side view of the diffusion surface 12a viewed from the direction of arrow B in FIG. 108A, and FIG. 11C is an arrow in FIG. It is the side view seen from C direction.

As shown in the figure, fine irregularities are formed on the diffusion surface 12a, and the light emitted from the eyeball illumination prism 10 is diffused here. At this time, since the fine irregularities are formed in a lattice shape, the light is diffused in the vertical direction and the horizontal direction in FIG. 12A, and the observer can see the LED spread in a cross shape 1 m ahead. become. If rain, snow, dust, etc. adhere to the diffuser 12a of the eyeball illumination system, L
Since the ED does not look like a normal cross shape, the observer can know from this viewpoint whether the gaze detection operation is normally performed.

Although the optical path from the LED 11 has been described above, the LED 50, the LED 51, and the LED 52 are described as follows: LED 50 → surface 10j → surface 10e → surface 10g → convex surface 1
0b → Diffusion plate 12 ・ LED51 → Surface 10k → Surface 10e → Surface 10h → Convex surface 1
0c → Diffusion plate 13 ・ LED52 → Surface 10l → Surface 10e → Surface 10h → Convex surface 1
0d → diffusion plate 13 Note that the convex surface 10b is an observation optical system of the LED 50,
The convex surface 10c is an observation optical system of the LED 51, and the convex surface 10d is L
This is an observation optical system of the ED 52.

As will be described later, the point of gaze is detected based on the position of the pupil and the position of the first Purkinje image which is a reflection image of the light source on the cornea. Since the eyelids of a person close from above, if the eyes are illuminated from below, the first Purkinje statue is less likely to be shaded by eyelashes. Therefore, when the camera is held at the correct position, L
When the ED 11 and the LED 50 are held in the vertical position, the LED 11 and the LED 51 or the L
The ED 50 and the LED 52 are used separately.

For example, when the LED 11 and the LED 50 are turned on, the eyes as viewed from the front are as shown in FIG.

In FIG. 13, 60 and 61 are LEDs 11
And the first Purkinje image formed by the LED 50. The two first Purkinje images have a cross shape due to the action of the diffusion plate 12. Japanese Patent Application Laid-Open No. Hei 6-230271 discloses a configuration in which a light source itself is formed in a cross shape or a plurality of LEDs are arranged to form a cross-shaped first Purkinje image. A cross-shaped first Purkinje image can be more easily formed by utilizing the action of the system and the diffusion plate.

LED11, LED50, LED51, L
The ED 52 has a convex surface 10a, a convex surface 10b, and a convex surface 1 respectively.
An observation optical system consisting of 0c and a convex surface 10d is apparently positioned 1 m ahead of the observer. As mentioned above, the light emitted by these LEDs is visible, but must not interfere with viewfinder viewing.

FIG. 14 is a diagram showing the relationship between the viewing angle and the number of photoreceptors when the viewing position having the best visual characteristics is taken as the origin. It can be considered that the distribution of photoreceptors well expresses visual characteristics.

The photoreceptor cells include a cone that can sense the difference between brightness and color, and a rod that has high sensitivity but senses only brightness. This rod has extremely low sensitivity to red. As shown in FIG. 14, the number of cones has a sharp peak at the center of the visual characteristic, and is considerably less outside the range of approximately ± 8 °. On the other hand, the number of rods is conversely small at the center of visual characteristics, and there is a peak at the periphery. A person detects details of an object with a cone, and supplements information on a peripheral visual field with a rod.

Considering the relationship with the eyeball illumination system, first, since the LED, which is the light source, has the light emission characteristics shown in FIG. 2, the rod cannot detect the illumination light. In addition, since the cone has a characteristic of low sensitivity in the peripheral visual field, it is hardly noticeable if the illumination light is projected to the eye at an angle of 8 ° or more when looking at the viewfinder visual field. become.

FIGS. 15A and 15B are explanatory diagrams showing the relationship between the finder visual field and the illumination system.

In the figure, reference numeral 60 denotes an observer's eyeball. It is assumed that the eyeball 60 is located at the eye point of the finder system. The eye point is a position where the entire finder field of view can be seen when the pupil is infinitely narrowed. 61 is a range in which an object image is displayed, 62 is an aperture display, 63 is a shutter speed display, 64 is a setting sensitivity of the solid-state imaging device 6, and 65 is an imaging optics when converted into a value using a camera using a 35 mm photographic film. This is the focal length of the system 3.

In a digital color camera of the interchangeable lens type, both the focal length of the imaging optical system and the image size captured by the solid-state imaging device change depending on the combination of the imaging lens 2 and the camera body 1. Therefore, in order to easily show the shooting angle of view for each combination, it is converted to the focal length value of a camera using a 35 mm photographic film and displayed.

Now, the relationship with an eyeball illumination system using the LED 11 as a light source will be considered.

When the user observes the lower end of the viewfinder display indicated by reference numerals 61 to 65, the light beam is L3 in FIG.
The light beam when the upper end is watched is L4. That is, the observer sees the lower end of the aperture display 62 and the lower end of the shutter speed display 63 in the direction of the light beam L3. Here, the area between the light beam L3 and the light beam L4 is the object image visual field. Further, among the light rays that are emitted from the convex surface 10a, which is the observation optical system formed in the eyeball illumination prism 10, and that reach the pupil, the light ray that forms the smallest angle with the optical path of the finder display is L5.

The angle formed between the light rays L3 and L5, that is,
If the angle θ formed with the object image visual field is 8 ° or more, the eyeball illumination light will enter the peripheral visual field with few cones among the visual cells even when the lower end of the finder display is viewed, and is hardly visually recognized. . On the other hand, if the line of sight is moved downward and directed in the direction of the ray L5 of the eyeball illumination light, it is captured by the cone,
The LED appears to glow red one meter forward. If water droplets, snow, dust, etc. adhere to the diffusion plate 12 and the eyeball illumination is not properly performed, the observer may send an LED.
Does not look like a normal cross, so it can be easily known that gaze detection is not operating normally. Therefore, it is possible to prevent the focus from being adjusted based on an incorrect gaze detection result.

Further, by illuminating the eyeball with visible light, there is an advantage even when the observer uses spectacles.

FIG. 16 shows an example of the spectral reflectance characteristics of spectacles having a lens surface coated with an anti-reflection coating. An object with a transparency coating is very preferable for everyday life because the object looks clear. This is similar to the spectral reflectance characteristic of FIG.
It can be realized by suppressing the reflectance to a low value in the visible region from about 400 nm to 700 nm.

However, in general, such glasses require 70
In the wavelength region longer than 0 nm, the reflectance becomes rather high,
This value is higher than the spectacles without the coating. Therefore, when the eyeball is illuminated with infrared light, the pupil and the first Purkinje image may not be properly captured due to the interference of the reflected light from the eyeglasses. In order to completely prevent such a situation, the eyeball illumination may be performed at a wavelength in the range of 400 nm to 700 nm. However, when the eyeball is illuminated in the wavelength range of high relative luminous efficiency, it enters the sensitivity range of the rod of photoreceptors this time,
It tends to hinder finder observation.

Therefore, as shown in FIG. 2, the reflected light from the spectacles is illuminated with light having a peak wavelength near 700 nm and including the bottom of the intensity distribution in the visible region from 400 nm to 700 nm. And an eyeball illumination system that does not hinder finder observation can be realized.

FIG. 17 is an optical path development diagram of the eyeball imaging system.

The dichroic mirror 14, shown in FIG.
The state where the reflection at the mirror 15 is developed and viewed from above is shown. Also, as a diagram viewed from the direction of arrow D in FIG.
FIGS. 18, 19, and 20 are plan views of the diaphragm 71, the compound eye lens 17, and the line-of-sight sensor 18, respectively.

In each figure, 70 is an eyeball, and 71 is a stop. The optical path changing prism 16 has two convex surfaces 16a and 16e which are light incident surfaces, and obtains images of the eyeball 70 captured from two directions through the finder optical system 42. The eyeball 70 is illuminated by the eyeball illumination system, and the reflected light incident from the convex surface 16a of the optical path changing prism 16 is then reflected by the surfaces 16b and 16c and exits from a position near the optical axis L16 on the surface 16d. . An aperture 71 is located behind the optical path changing prism 16. As shown in FIG.
1 has two openings 71a and 71b, and a light beam incident from the convex surface 16a of the optical path changing prism 16 is narrowed by the opening 71a. The luminous flux emitted from the aperture enters from the lens surface 17a of the compound eye lens 17, and
c, and forms an eye image of the imaging region Ea on the area sensor section 18a of the eye-gaze sensor 18 as shown in FIG.

The convex surface 16e of the optical path changing prism 16
Then, the reflected light incident on the surface 16f is reflected by the surfaces 16f and 16g, and is emitted from a position on the surface 16d near the optical axis L6.
At the next stop 71, the light beam emitted from the aperture 71b is stopped through the lens surface 17b of the compound eye lens 17, exits from the lens surface 17d, and exits from the lens surface 17d as shown in FIG. An eye image of the imaging region Eb is formed on the section 18b. At this time, by arranging convex lenses before and after the stop 71, an eye image with little distortion is obtained. This is effective for accurately synthesizing an eye image described later.

FIG. 21 is a diagram showing the relationship between the area sensor section of the visual axis sensor 18 and the eye image.

In the figure, reference numeral 80a denotes the area sensor unit 1.
The eye image obtained from 8a, 80b is the area sensor unit 18
b is an eye image obtained from b. Eye image 80a and eye image 8
0b is a pair of eye images having parallax.

Since the imaging region Ea and the imaging region Eb are shifted in the left-right direction of the eyes, the positions of the eyes appearing in the eye images 80a and 80b are different. Eye image 80a and eye image 8
0b becomes a common image because of the area sensor units 18a,
18b, which are indicated by hatching, and are the imaging areas Fa and Fb shown below the eye images 80a and 80b. Even when the observer looks into the finder extremely obliquely, the eyes can be captured in at least one of the imaging regions because the imaging region Ea and the imaging region Eb are shifted in the left-right direction. In the description of the arithmetic processing, the eye image 80a is referred to as an A image, and the eye image 80b is referred to as a B image.

In the eye image, in addition to a cross-shaped first Purkinje image formed by the action of the diffusion plate 12, a reflected luminescent spot 81 due to tear drops and a reflected luminescent spot 82 due to eyelashes can be seen. However, since the teardrops and eyelashes that cause the reflective luminescent spot do not have a large area like the cornea, the reflected image there is not a cross shape and can be easily distinguished from the first Purkinje image. Therefore, the gaze detection can be performed with high accuracy without mistaking the reflected luminescent spot 81 with the teardrop or the reflected luminescent spot 82 with the eyelash as a true first Purkinje image.

Although the parallax of the plurality of eye images is set in the horizontal direction with respect to the eye here, it may be in the vertical direction. Further, both the horizontal direction and the vertical direction may be used. In this case, since four eye images are obtained, it is possible to perform a more forgiving and highly accurate detection of individual differences such as eyelash modes and how to look through the viewfinder.

Next, the operation of the camera will be described with reference to the flowcharts shown in FIG. 22, FIG. 23, and FIG.

The flowchart in FIG. 22 is a control program written in the ROM in the camera system control circuit 35.

First, in FIG. 22, when the control program is started, first, in step # 201, it is checked whether or not the main switch 19 is ON via the operation detecting circuit. If it is OFF, step # 201 is repeated again, and if it is ON, the process proceeds to the next step # 202. In step # 202, the solid-state imaging device 6 is driven via the solid-state imaging device driving circuit 37, and continuous capture of display images is started. In the next step # 203, a finder display process for displaying an image captured by the solid-state imaging device 6 as a moving image on the liquid crystal display 7 via the reproduction processing circuit 34 is started. In the following step # 204, a flag F indicating the success or failure of the line-of-sight detection
Reset LGe. Then, in step # 205, it instructs the gaze detection / AF control circuit 40 to start the gaze detection subroutine.

At step # 206, the flag FL
Check whether Ge has been set. If it has not been set, the process returns to the previous step # 204. If it has been set, the process proceeds to the next step # 207. Step # 2
07, the eye-gaze detection / AF control circuit 4
0 is instructed to start the AF control subroutine. Then, in the next step # 208, it is checked via the operation detection circuit 36 whether or not the release button 20 is ON. If it is OFF, the process returns to the previous step # 204, and if it is ON, the process moves to the next step # 209.

In step # 209, since the release button 20 has been pressed, the solid-state image sensor driving circuit 37
Drives the solid-state imaging device 6 in the first output mode via
Capture high-definition images. Then, in the next step # 210, the data subjected to the RGB image processing and the YC processing is recorded in the internal memory via the recording processing circuit 33, and a series of controls is ended.

The flow chart of FIG.
This is a control program written in the ROM in the F control circuit 40, and corresponds to a calculating means for detecting a line of sight. A first calculation mode for calculating a gazing point using an eye image synthesized based on a plurality of eye images and a second calculation mode for calculating a gazing point using one eye image are provided.

In FIG. 23, when the line-of-sight detection subroutine starts, in step # 301, the LEDs are selectively turned on based on the attitude of the camera, and the A-line image, ie, the eye image 80a, and the B-image, ie, the eye image 80b, Is read. Next step # 302
In, the area having the lowest luminance is detected from the A image, and in the subsequent step # 303, the area having the lowest luminance is detected from the B image. The processing of steps # 302 and # 303 uses the characteristic that the pupil shows the lowest luminance in the eye image, and in a later step, preparations are made for detecting the first Purkinje image using the lowest luminance area as a clue. It is.

In the next step # 304, the barycentric coordinates of the first Purkinje image appearing in the A image and in the next step # 305 the barycentric coordinates of the first Purkinje image appearing in the B image are detected.

Although the first Purkinje image has a cross shape, the barycentric coordinates of the area where the vertical and horizontal lines overlap are stored as data. A technique such as pattern matching may be used for detecting the first Purkinje image. To be cross-shaped,
A highly accurate line of sight detection can be performed without confusing the reflected luminescent spot 81 with a teardrop or the reflected luminescent spot 82 with a eyelash with a true first Purkinje image.

In step # 306, it is determined whether or not the detection of the first Purkinje image of the A image in step # 304 is successful. In rare cases, detection of the first Purkinje image may fail due to ghosting from the glasses or direct entry of sunlight. In this case, the process branches to step # 312. If the first Purkinje image has been successfully detected, the process proceeds to step # 307, where it is determined in the previous step # 305 whether the first Purkinje image of the B image has been successfully detected. If unsuccessful, the process branches to step # 315. If successful, the process proceeds to the next step # 308.

Step # 308 is the same as step # 30
This is a process in the case where the detection of the first Purkinje image has succeeded in both the A image and the B image after the determinations of # 6 and # 307.
By combining the A image and the B image and using both information, unnecessary information is eliminated and missing information is complemented. At this time, the property that the first Purkinje image is generated at substantially the same depth as the pupil is used for image synthesis.

FIG. 24 is an explanatory diagram showing the relationship between the first Purkinje image and the pupil. In FIG. 24, reference numeral 90 denotes the eyeball of the observer, 91 denotes the iris forming the pupil, 92 denotes the cornea,
P1 is the center of curvature of the cornea 92. The eyeball 90 is located on the finder optical axis L2, and the eyeball illumination 93 is provided from a direction inclined by φ with respect to the finder optical axis L2.

At this time, the first Purkinje image P2, which is a reflection image from the cornea 92 of the light source, is formed by the light source and the curvature center P of the cornea.
1 is formed on a line connecting the two. The first Purkinje image P2 is a virtual image, and when the radius of the cornea 92 is R, the depth H of the first Purkinje image P2 viewed from the direction of the finder optical axis L2 is given by Expression (1).

H = (1−cosφ) × R + Gcosφ (1) Here, G is the depth of the first Purkinje image when viewed from the direction of eyeball illumination. Assuming that the light source is 1000 mm ahead, G is given by equation (2).

[0114] G = B / 2- (R / 2) 2/1000 ········· (2) Therefore, by substituting equation (2) into equation (1), H is a R It can be represented by φ.

Further, when H is obtained using φ = 25 ° R = 7.7 mm, H = 4.2 mm.

On the other hand, the depth K of the iris is K = 3.6 mm, which is extremely close to the depth of the first Purkinje image. As a result,
When a pair of eye images having parallax is captured by the compound-eye lens 17 described with reference to FIG. 17, the positional relationship between the pupil and the first Purkinje image hardly changes between the images. On the other hand, the position of the eyelash and the position of the teardrop differ between images. Also, when there is a ghost due to the reflected light from the glasses, the ghost also occurs at a depth considerably different from that of the iris or the first Purkinje image, and thus the position differs between the images.

Returning to the flowchart of FIG. 23, in step # 308, as shown in FIG. 25, the difference between the coordinates of the first Purkinje image of the A image (80a) and the B image (80b) is calculated. Then, in the next step # 309, as a preparation for combining the A image and the B image, the shift amount SB of the B image is set to the difference between the coordinates of the first Purkinje image of the A image and the B image. In the following step # 310, the A image and the B image are combined according to the following algorithm to generate a C image as shown in FIG.

The image synthesizing algorithm is as follows.

First, the integer part of the shift amount SB is e, the decimal part is f, and the area sensor 1
An address as shown in FIG. 27 is assigned to each pixel of the area sensor 8a and the area sensor section 18b. Since the direction of the parallax is the direction of the row, the processing of each row is the same, and the C image is generated by performing the following processing on all the rows.

It is assumed that the X column data of the A image in a certain row is A _X , the X column data of the B image is B _X , and the X column data of the C image is C _X.

1) When X <n or X <e, C _x = A _x 2) When n ≦ X <p + 1 and e ≦ X <e + m, C _x = MIN (A _X , (1-f) × B _Xe + F × B
_{X-e + 1} )) 3) When p + 1 ≦ X or e + m ≦ X, C _x = (1−f) × B _Xe + f × B _{X-e + 1 The} C image is only from the A image in the range of 1). In the range 2), the A image and the B image are formed, and in the range 3), only the B image is formed.

MIN (u, v) is an operator for selecting a smaller value between u and v. When the A image and the B image are superimposed and compared, it can be estimated that eyelashes and ghosts are superimposed on data indicating that the image is brighter even though the same element is captured. The luminance of the eyelashes is always higher than the luminance of the pupil, and the place where the ghost is superimposed is always brighter. Therefore, if the operator MIN (u, v) for selecting a small value is used, such disturbance is eliminated. Thus, a C image as shown in FIG. 26 can be generated.

Since the ghost 94 seen in the eye image 80a (A image) and the ghost 95 seen in the eye image 80b (B image) in FIG. 25 are not at the same position, the ghost which is unnecessary information is eliminated in the C image. The pupil which is necessary information is supplemented.

Such a ghost is caused by multiple reflections on the finder optical system, the spectacles, and the cornea, and is particularly increased when the spectacles are used. This MIN (u, v) operator is particularly useful when the observer is using glasses. Therefore, it is preferable to prepare a plurality of image synthesis algorithms, and select the MIN (u, v) operator when it is determined that the eyeglasses are used by a known eyeglass determination unit.

Returning to the flowchart of FIG. 23 again, in step # 311, the previously synthesized C image is designated as the target of the processing in the following step # 316.

Next, from step # 306 to step # 3
The case of branching to 12 will be described.

In step # 312, it is determined whether or not the detection of the first Purkinje image of the B image in step # 305 is successful. Step # 3 in case of failure
13 and, if successful, the next step # 314
Move to In step # 313, the determinations in step # 306 and step # 312 are both N (NO
This means that the detection of the first Purkinje image has failed in both the A image and the B image. Therefore, the line-of-sight detection processing is not performed any more, the FLGe is reset, and the subroutine is returned.

On the other hand, step # 314 is a process in the case where the detection of the first Purkinje image has been successful only with the B image after the determinations in the previous steps # 306 and # 312.

The first Purkinje image, which is originally optically present, cannot be captured as an image due to the influence of the dead zone between the light receiving portions of the line-of-sight sensor. In some cases, the light is superimposed, and only the first image detects the first Purkinje image. In addition, when the finder optical system is viewed from an oblique direction, the reflection surface of the light beam forming the first Purkinje image is removed from the cornea, the image disappears, and the first Purkinje image may appear only in one image. is there. In such a case, a B image in which the first Purkinje image can be captured is designated as a target of processing in the subsequent step # 316.

Next, the case where the process branches from step # 307 to step # 315 will be described.

Step # 315 is a process in the case where the detection of the first Purkinje image is successful only with the A image after the determinations of step # 306 and step # 307. Has occurred. in this case,
The A image in which the first Purkinje image can be captured is designated as a target to be processed in the subsequent step # 316.

The processing in the case where the first Purkinje image can be captured in both the A image and the B image is described in the first operation mode.
If the process in which the image can be captured by either the A image or the B image is set to the second operation mode, the first operation mode and the second operation mode are automatically switched. It is possible to obtain a gazing point output based on the processing after step # 316.

For example, when the finder is viewed from an oblique direction, the first Purkinje image cannot be captured in one of the images, but the sight line detection in the second calculation mode makes it difficult to detect the sight line. The device can be realized, and even if the pupil detection or the first Purkinje image cannot be detected for one of the eye images due to a ghost, the gaze detection is not disabled by the second calculation mode. The device can be realized.

In step # 316, the relative displacement between the center of the first Purkinje image and the center of the pupil image is calculated for the designated image. Then, in the next step # 317, the relative displacement between the center of gravity of the first Purkinje image and the center of the pupil image obtained in step # 316 is converted into gazing point information on the finder. Step # 3
16 and # 317 are described in JP-A-6-1587.
No. 72. Finally, in step # 318, since the line of sight has been successfully detected, FLGe is set and the subroutine is returned.

The flow chart of FIG.
This is a control program written in a ROM in the F control circuit 40.

In the figure, when the AF control subroutine starts, in step # 401, a focus detection point is designated on the gazing point based on the result of the line-of-sight detection, and a rectangular area surrounding the focus detection point on the solid-state imaging device 6 is designated. An area from which image data for focus detection should be obtained and a control area for focus detection calculation. In the next step # 402,
The solid-state imaging device 6 is driven in the second output mode to acquire image data for focus detection. In the following step # 403, a focus detection calculation is performed using the image data obtained in the above step # 402. The obtained defocus amount is converted into a driving amount of the focusing lens. Finally, in step # 404, the camera system control circuit 35
, The driving amount of the focusing lens is transmitted to the lens system control circuit 41. In response to this, the lens system control circuit 41 moves the focusing lens in the direction of the optical axis L1 of the imaging lens, and the focus is adjusted.

(Second Embodiment) In the first operation mode in which unnecessary information is eliminated and missing information is complemented, another method of synthesizing an A image and a B image according to the present invention will be described. The second embodiment will be described below. The other parts are the same as those of the first embodiment, and the description is omitted.

At step # 310 in the visual line detection subroutine of FIG. 23, the A image and the B image are combined according to the following algorithm to generate a C image.

First, the integer part of the shift amount SB is e, the decimal part is f, and the area sensor 1
An address as shown in FIG. 27 is assigned to each pixel of the area sensor 8a and the area sensor section 18b. The processing of each row is the same, and the following processing is performed for all rows to generate a C image.

It is assumed that the X column data of the A image in a certain row is A _X , the X column data of the B image is B _X , and the X column data of the C image is C _X.

1) When X <n or X <e, C _X = A _X 2) When n ≦ X <p + 1 and e ≦ X <e + m, C _X = (A _X + ((1-f) × B _Xe ) + F × B
_{X-e + 1} )) / 2 3) When p + 1 ≦ X or e + m ≦ X, C _X = (1−f) × B _Xe + f × B _{X-e + 1 The} C image is an A image in the range of 1). In the range of 2), only the A and B images are formed, and in the range of 3), only the B image is formed.

In the range 2), the C image is generated from the arithmetic mean of the A image and the shifted B image. This is equivalent to capturing the second eye image and averaging it, so that temporally random sensor noise and the like can be reduced by 1 /
効果 (2) has the effect of improving the accuracy of gaze detection. At coordinates where only one of the A image and the B image has an optical disturbance such as a ghost, the level of the optical disturbance in the composite image is reduced to half. As a result, the part where the contour of the pupil appears distorted due to the ghost generated by the finder optical system or the glasses or the position where the position of the first Purkinje image is shifted appears to be restored to the original shape in the C image.

(Third Embodiment) In the first operation mode in which unnecessary information is eliminated and missing information is complemented, another method of synthesizing the A image and the B image according to the present invention will be described. The third embodiment will be described below. The other parts are the same as those of the first embodiment, and the description is omitted.

At step # 310 in the visual line detection subroutine of FIG. 23, the A image and the B image are combined according to the following algorithm to generate a C image.

First, the integer part of the shift amount SB is e, the decimal part is f, and the area sensor 1
An address as shown in FIG. 206 is assigned to each pixel of the area sensor 8a and the area sensor section 18b. The processing of each row is the same, and the following processing is performed for all rows to generate a C image.

It is assumed that the X column data of the A image in a certain row is A _X , the X column data of the B image is B _X , and the X column data of the C image is C _X.

1) When X <n or X <e, C _X = A _X 2) When n ≦ X <p + 1 and e ≦ X <e + m, C _X = √ (A _X × ((1-f) × B _Xe + f × B
_{X-e + 1} )) 3) When p + 1 ≦ X or e + m ≦ X, C _X = (1−f) × B _Xe + f × B _Xe +1 The C image is from the A image only in the range of 1) and 2). Is made from both the A image and the B image, and in the range 3) is made only from the B image.

In the range 2), the C image is generated from the geometric mean of the A image and the shifted B image. Not equal 2
Because of the mathematical property that the geometric mean of numbers is always smaller than the arithmetic mean of those numbers, the combining method shown in the first embodiment and the combining method shown in the second embodiment Get intermediate results. That is, it has both the characteristic of eliminating optical disturbances such as ghosts and the characteristic of reducing random noise, and is highly accurate and very practical.

According to each of the above embodiments, the imaging optical system (the compound eye lens 17 and the like) for forming an eye image of the observer's eye through the finder optical system is replaced with a plurality of eye images having parallax. According to the form, an A image and a B image are formed, and a gaze point and an AF control corresponding to a calculating unit are determined by using an eye image synthesized based on the A image and the B image. Calculated by the circuit 40 (step # in FIG. 23)
301 to # 311, # 316, and # 317), it is possible to perform a forgiving gaze detection with respect to individual differences such as the shape of eyelashes by using a technique of capturing an eye image from two directions.

By performing the above calculations,
Ghosts that are harmful to the detection of the pupil and the detection of the first Purkinje image can be eliminated, and highly accurate gaze detection can be performed.

Furthermore, as described in the second and third embodiments, a combined eye image is obtained (a C image is obtained from an arithmetic mean or a geometric mean of an A image and a shifted B image). Therefore, it is possible to perform highly accurate gaze detection equivalent to performing two gaze detections.

The line-of-sight detection / AF control circuit 40
Is a first calculation mode for calculating a gazing point using an eye image synthesized based on a plurality of eye images (step # 3 in FIG. 23).
06 to # 311 → # 316) and a second calculation mode (step # 306 → # 309 → # 315 → # 316 in FIG. 23) for calculating the point of gaze using one eye image. Since the first Purkinje image cannot be captured in one of the images when the finder is viewed more diagonally, the gaze detection device that is less likely to be in a non-detectable state by performing the gaze detection in the second calculation mode. Can be realized. Further, even if the pupil detection or the first Purkinje image cannot be detected for one eye image due to a ghost, it is possible to prevent the detection from being disabled by the second calculation mode.

[0153]

As described above, according to the first aspect of the present invention, it is possible to tolerate individual differences such as the eyelash form of the observer and how to look into the observation surface, and also to perform highly accurate gaze detection. The present invention can provide a gaze detection device capable of performing the following.

According to the sixth aspect of the present invention, it is possible to provide a line-of-sight detecting device which can prevent a line of sight from being detected even when an observer looks into the observation surface from an oblique direction. It is.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a digital color camera equipped with a visual line detection device according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating light emission characteristics of the LED according to the first embodiment of the present invention.

FIG. 3 is a diagram illustrating a spectral reflectance characteristic of the dichroic mirror according to the first embodiment of the present invention.

FIG. 4 is a block diagram showing an electrical configuration of the digital color camera according to the first embodiment of the present invention.

FIG. 5 is a circuit configuration diagram of an area sensor unit in the solid-state imaging device according to the first embodiment of the present invention.

FIG. 6 is a sectional view of a light receiving unit according to the first embodiment of the present invention.

FIG. 7 is a plan view showing a positional relationship between a microlens and a light receiving unit according to the first embodiment of the present invention.

FIG. 8 is a sectional view of an area sensor unit according to the first embodiment of the present invention.

FIG. 9 is a side view of the eyeball illumination system according to the first embodiment of the present invention.

FIG. 10 is a front view of an eyeball illumination system according to the first embodiment of the present invention.

FIG. 11 is a developed view of an illumination light path according to the first embodiment of the present invention.

FIG. 12 is an explanatory diagram of a diffusion plate according to the first embodiment of the present invention.

FIG. 13 is a diagram illustrating an eye portion viewed from the front according to the first embodiment of the present invention.

FIG. 14 is a diagram showing the relationship between the viewing angle and the number of photoreceptors in the first embodiment of the present invention.

FIG. 15 is an explanatory diagram showing a relationship between a finder visual field and an illumination system according to the first embodiment of the present invention.

FIG. 16 is an example of the spectral reflectance characteristics of spectacles in which a lens surface is provided with an anti-reflection coating in the first embodiment of the present invention.

FIG. 17 is an optical path development diagram of the eyeball imaging system according to the first embodiment of the present invention.

FIG. 18 is a plan view of the stop according to the first embodiment of the present invention.

FIG. 19 is a plan view of a compound eye lens according to the first embodiment of the present invention.

FIG. 20 is a plan view of the eye-gaze sensor according to the first embodiment of the present invention.

FIG. 21 is a diagram illustrating a relationship between an area sensor unit of the eye-gaze sensor and an eye image according to the first embodiment of the present invention.

FIG. 22 is a flowchart showing an operation of the camera according to the first embodiment of the present invention.

FIG. 23 is a flowchart showing an operation of the camera according to the first embodiment of the present invention.

FIG. 24 is an explanatory diagram showing a relationship between a first Purkinje image and a pupil in the first embodiment of the present invention.

FIG. 25 is an explanatory diagram showing a difference between the coordinates of the first Purkinje image of the A image and the B image in the first embodiment of the present invention.

FIG. 26 is an explanatory diagram of a C image that is a composite image according to the first embodiment of the present invention.

FIG. 27 is an explanatory diagram of addresses assigned to respective pixels of the area sensor unit of the eye-gaze sensor according to the first embodiment of the present invention.

FIG. 28 is a flowchart showing an operation of the camera according to the first embodiment of the present invention.

[Explanation of symbols]

 Reference Signs List 6 solid-state image pickup device 11 LED 14 dichroic mirror 16 optical path changing prism 17 compound eye lens 18 gaze sensor 35 camera system control circuit 40 gaze detection / AF control circuit

Claims (7)

[Claims] 1. An eye-gaze detecting apparatus comprising: an imaging optical system that forms an eye image of an eye of an observer; and a calculating unit that calculates a gazing point of the observer from the eye image, wherein the imaging optical system includes: Forming a plurality of eye images having parallax, wherein the calculating means calculates a gazing point using an eye image synthesized based on the plurality of eye images; Detection device. 2. The arithmetic means, when obtaining the synthesized eye image, divides a synthesis range of the plurality of eye images into first, second, and third ranges, and includes a plurality of ranges in the first range. The synthesis is performed by using only the information of one of the plurality of eye images, the information of the plurality of eye images in the second range, and the information of the other one of the plurality of eye images in the third area. The eye-gaze detecting device according to claim 1, wherein an eye image is obtained. 3. The combined eye image in the second range is obtained by selecting a minimum value of each detection position corresponding to each of the plurality of eye images. Gaze detection device. 4. A line-of-sight detection apparatus comprising: an imaging optical system that forms an eye image of an eye of an observer; and a calculating unit that calculates a gazing point of the observer from the eye image. Forming a plurality of eye images having parallax, wherein the calculating means obtains an eye image synthesized by an arithmetic mean of one of the plurality of eye images and the other image, and A gaze detection apparatus for calculating a gazing point using an eye image. 5. A line-of-sight detection device comprising: an imaging optical system that forms an eye image of an eye of an observer; and a calculating unit that calculates a gazing point of the observer from the eye image. Forming a plurality of eye images having parallax, wherein the calculating means obtains an eye image synthesized by a geometric mean of one of the plurality of eye images and the other image; A gaze detection apparatus for calculating a point of gaze using an image. 6. A line-of-sight detection apparatus comprising: an imaging optical system that forms an eye image of an observer's eye; and a calculation unit that calculates a gazing point of the observer from the eye image. Forming a plurality of eye images having parallax, wherein the calculating means calculates a gazing point using an eye image synthesized based on the plurality of eye images; and A gaze detection device having a second calculation mode for calculating a point of regard using an eye image. 7. The arithmetic unit may select the first arithmetic mode by detecting a Purkinje image from any of the plurality of eye images, and detect a Purkinje image among the plurality of eye images. The eye gaze detecting apparatus according to claim 6, wherein the second calculation mode is selected when an unrecognizable eye image exists.