JP3211427B2 - Eye gaze detection device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a gaze detecting device for illuminating an eyeball, receiving an image of the illuminated eyeball and detecting a gaze.

[0002]

2. Description of the Related Art Various devices (for example, eye cameras) for detecting a so-called line of sight (a visual axis) for detecting which position on an observation surface the observer is observing have been proposed.

For example, in Japanese Patent Application Laid-Open No. 1-274736, a parallel light beam from a light source is projected to the anterior segment of an observer's eyeball, and a corneal reflection image formed by light reflected from the cornea and an image forming position of a pupil are used. Looking for the visual axis.

FIG. 7 is a diagram for explaining the principle of the visual line detection method, and FIG.
FIG. 8A is an explanatory diagram of a normal subject's eyeball image projected on the image sensor 14 surface of FIG. 7, and FIG. 8B is a diagram of FIG.
FIG. 4 is an explanatory diagram of an output signal of an eyeball image on a line La-Lb. FIG. 9 is an explanatory diagram of an eyeball image projected on the surface of the image sensor 14 when the subject wears glasses.

In the figure, reference numeral 50 denotes a so-called white eye portion of an eyeball, 51 denotes a pupil, and 52a and 52b denote Purkinje images (corneal reflection images) when the eyeball is illuminated with a light beam from a light source. 53
Reference characters a, 53b, 54a, and 54b denote ghost images based on the reflected light when the light beam from the light source unit is reflected by the subject's glasses.

Next, a method of detecting the eye gaze will be described with reference to FIGS. 7, 8 and 9. FIG.

The infrared light emitting diodes 13a and 13b are arranged substantially symmetrically in the X direction with respect to the optical axis A of the light receiving lens 12, and divergently illuminate, for example, the eyeball of a photographer who looks through a finder system as a subject. I have.

The infrared light emitted from the infrared light emitting diode 13b illuminates the cornea 16 of the eyeball 15. At this time, a corneal reflection image d due to a part of the infrared light reflected on the surface of the cornea 16
Is condensed by the light receiving lens 12 and the image sensor 1
The image is re-imaged at a position d 'on the upper side of FIG.

Similarly, infrared light emitted from the infrared light emitting diode 13a illuminates the cornea 16 of the eyeball. At this time, a corneal reflection image e due to a part of the infrared light reflected on the surface of the cornea 16 is condensed by the light receiving lens 12 and the image sensor 1
The image is re-imaged at a position e 'on the upper side of FIG.

An image of the pupil 19 of the eyeball 15 is also formed on the image sensor 14. At this time, the X coordinate of the center C when the pupil 19 is a circle is Xc, and the X coordinate of the image of the center C on the image sensor 14 is Xc '.

Also, since the X coordinate of the midpoint of the corneal reflection images d and e matches the X coordinate Xo of the center of curvature O of the cornea 16,
Occurrence position d of the cornea reflection image, the X coordinate of the e Xd, Xe, a standard distance from the center of curvature O of the cornea 16 to the center C of the pupil 19 and L _OC, consider the individual differences with respect to the distance L _OC coefficient Is A1, the rotation angle θ of the eyeball optical axis a substantially satisfies the relational expression of (A1 * L _OC ) * sin θ ≒ Xc− (Xd + Xe) / 2 (1)

For this reason, the eye-gaze arithmetic processing unit detects the position of each characteristic point (corneal reflection images d and e and the pupil center C) projected on a part of the image sensor, thereby obtaining the rotation angle of the optical axis a of the eyeball. θ can be obtained.

At this time, equation (1) can be rewritten as β (A1 * L _OC ) * sin θθXc ′ − (Xd ′ + Xe ′) / 2 (2) Here, β is a magnification determined by the position of the eyeball with respect to the light receiving lens 12, and is substantially obtained as a function of the interval | Xd′−Xe ′ | of the corneal reflection image.

The rotation angle θ of the eyeball 15 is rewritten as follows: θ {ARCSIN {(Xc′−Xf ′) / β / (A1 * L _OC )} (3) However, Zf ′ ≒ (Xd ′ + Xe ′) / 2.

Since the optical axis A of the photographer's eyeball and the visual axis do not coincide with each other, when the horizontal rotation angle θ of the optical axis A of the photographer's eyeball is calculated, the optical axis of the eyeball and the visual axis are calculated. By performing the angle correction δ, the horizontal line of sight θH of the photographer can be obtained.

Assuming that a coefficient considering the individual difference with respect to the correction angle δ between the optical axis A and the visual axis of the eyeball is B1, the horizontal line of sight θH of the photographer is θH = θ ± (B1 * δ) (4) ) Is required. Here, as for the sign ±, if the rotation angle to the right with respect to the photographer is positive, the sign + is selected when the photographer's eyes, except for the observation device, are the left eye, and the sign-is selected when the photographer's eyes are the right eye.

In FIG. 1, the eyeball of the photographer is ZX.
An example of rotation in a plane (for example, a horizontal plane) is shown,
Even when the photographer's eyeball rotates in a ZY plane (for example, a vertical plane), the same detection can be made. However, since the vertical component of the photographer's line of sight coincides with the vertical component θ 'of the optical axis of the eyeball, the vertical line of sight θV is θV = θ ′.

Further, the position (Xn, Xn,
Yn) is a Xn ≒ m * θH ≒ m * [ARCSIN {(Xc'-Xf') / β / (A1 * L OC)} ± (B1 * δ)] ··· (5) Yn ≒ m * θV Desired. Here, m is a constant determined by the finder optical system of the camera.

Here, the values of the coefficients A1 and B1 for correcting individual differences in the eyeball of the photographer are obtained by having the photographer fixate on an index provided at a predetermined position in the viewfinder of the camera, and setting the position of the index. And the position of the fixation point calculated according to the equation (5).

In this embodiment, the calculation for obtaining the line of sight and the gazing point of the photographer is executed by the software of the microcomputer of the line-of-sight arithmetic processing unit based on the above equations.

A coefficient for correcting the individual difference in the line of sight is determined, and the position of the line of sight of the observer looking into the viewfinder of the camera is calculated using equation (5), and the line of sight information is used to adjust the focus or exposure of the taking lens. Used for control.

As described above, the line-of-sight direction / coordinate of the photographer is based on the corneal reflection image (“Purkinje image”, hereinafter “P image”) of the infrared light emitting diode (hereinafter referred to as “IRED”) for illumination.
Called. ) And the positional relationship of the pupils in the eyeball image.

Since the P image is a corneal reflection image of the IRED,
The brightness is high in the whole eyeball image, and the image is IRED
Since this is a reflection image of the chip light emitting surface, the size thereof is also extremely small. Therefore, a P image can be extracted from the eyeball image signal by using the feature.

Specifically, when the luminance value of a certain pixel in the eyeball image signal is equal to or more than a predetermined value and the luminance difference between the pixel and the peripheral pixels is equal to or more than a predetermined value, the pixel (or the pixel including the pixel) (Peripheral pixels / areas) are set as P image candidates.

A line-of-sight detecting device provided with a distance detecting means for detecting a distance from a predetermined position on the device side to an eyeball has been proposed, for example, in Japanese Patent Laid-Open No. 2-209126. At this time, the distance detection means turns on the IREDs in pairs,
This is used for the P image pair, and the distance is detected from the interval between the P image pairs at that time. When there are a plurality of P image pair candidates at this time, a P image pair having an appropriate interval is selected.

[0026]

As shown in FIG. 8, when the high-brightness portion on the image sensor is one set of points (52a, 52b), the P image pair can be easily detected.

However, as shown in FIG. 9, when there are a plurality of sets of points in the high-brightness portion on the image sensor, the P image pair may be erroneously selected. In this case, a line-of-sight detection error occurs. .

In this case, a method of judging whether or not the selected P image pair is correct is determined based on the calculated coordinates of the line of sight or the rotation angle of the eyeball within a normal range.
No. 23431 is proposed.

However, this method has a problem that if the center coordinates of the erroneous P image pair are close to the coordinates of the true P image pair, the calculation result falls within the normal range and erroneous gaze detection is performed.

According to the present invention, the distance from the apparatus side (camera side) to the subject (photographer) is obtained from the interval between two P images of the selected P image pair, and the obtained distance information falls within the normal range. It is an object of the present invention to provide a gaze detection device that can always perform gaze detection correctly regardless of whether glasses are worn by determining whether gaze detection has been correctly performed based on whether or not the gaze is detected.

In order to solve the above-mentioned problem, the present invention according to claim 1 comprises illumination means for illuminating an eyeball, and light receiving means for receiving an image of the eyeball illuminated by the illumination means, A gaze detecting device that detects a line of sight using an output of the light receiving unit; an image detecting unit that detects a corneal reflection image candidate of the eyeball from an output of the light receiving unit; and detects a line of sight using the corneal reflection image candidate. Gaze detecting means for
A distance detecting means for detecting a distance from the eyeball to the visual line detecting device using the corneal reflected image candidate; and a corneal reflected image candidate different from the corneal reflected image candidate when the detected distance is not within a predetermined range. Is selected to detect the line of sight.

According to a second aspect of the present invention, there is provided illumination means for illuminating an eyeball, and light receiving means for receiving an image of the eyeball illuminated by the illumination means. In a gaze detection device that detects a gaze by using an image detection unit that detects a corneal reflection image candidate of an eyeball from an output of the light receiving unit, a gaze detection unit that detects a gaze using the corneal reflection image candidate, Distance detecting means for detecting a distance from the eyeball to the visual line detection device using a corneal reflection image candidate, and when the detected distance is not within a predetermined range, the visual line detection means performs a different corneal reflection image candidate. A line of sight is selected to detect the distance, and the distance detecting means detects a distance using the selected different candidate corneal reflection image. According to a third aspect of the present invention, in the first or second aspect, when the distance detected using the corneal reflection image candidate is within a predetermined range, the corneal reflection image candidate is determined. It is characterized in that the line of sight detected by using is a correct detection result. According to a fourth aspect of the present invention, in the invention according to any one of the first to third aspects, the illuminating means has a pair of light sources, and the distance detecting means uses a light receiving result of the light receiving means. The distance between the eyeball and the line-of-sight detection device is detected by calculating the interval between two corneal reflection images generated on the eyeball by the pair of light sources.

[0033]

FIG. 1 is a schematic view of a main part of a first embodiment when the present invention is applied to a single-lens reflex camera.

In FIG. 1, reference numeral 1 denotes a photographing lens for convenience.
Although shown with a single lens, it is actually composed of more lenses. Reference numeral 2 denotes a main mirror which is inclined or retreated to a photographing optical path according to the state of observation of the subject image by the finder system and the state of photographing of the subject image. Reference numeral 3 denotes a sub-mirror, which reflects a light beam transmitted through the main mirror 2 toward a later-described focus detection device 6 below the camera body.

Reference numeral 4 denotes a shutter and reference numeral 5 denotes a photosensitive member, which comprises a silver halide film, a solid-state image pickup device such as a CCD or a MOS type, or an image pickup tube such as a vidicon.

Reference numeral 6 denotes a focus detection device, which is a field lens 6a, reflection mirrors 6b and 6 arranged near the image plane.
c, a secondary imaging lens 6d, an aperture 6e, a line sensor 6f including a plurality of CCDs, and the like.

The focus detecting device 6 in this embodiment uses a well-known phase difference method. As shown in FIG. 2, a plurality of areas (five places) in an observation screen (in the finder field) are set as distance measuring points. The distance measuring point is configured to enable focus detection.

Reference numeral 7 denotes a focusing plate arranged on a predetermined image forming plane of the photographing lens 1, 8 denotes a pentaprism for changing a finder optical path, and 9 and 10 each denote an imaging lens for measuring the luminance of a subject in an observation screen. It is a photometric sensor. The imaging lens 9 conjugately connects the focusing plate 7 and the photometric sensor 10 via a reflection optical path in the penta roof prism 8.

Next, behind the exit surface of the penta roof prism 8, an eyepiece 11 provided with a light splitter 11a is arranged and used for observation of the focus plate 7 by a photographer's eye 15. The light splitter 11a is composed of, for example, a dichroic mirror that transmits visible light and reflects infrared light.

Reference numeral 12 denotes a light receiving lens, 14 denotes a light receiving unit,
It is composed of an image sensor in which photoelectric element arrays such as CDs are arranged two-dimensionally and arranged so as to be conjugate with the vicinity of the pupil of the photographer's eye 15 at a predetermined position with respect to the light receiving lens 12. Reference numerals 13a to 13f denote infrared light emitting diodes (light source units) each serving as an illumination light source. Light source units 13a to 13
f is arranged around the eyepiece 11.

Reference numeral 21 denotes a high-brightness superimposing LED that can be visually recognized even in a bright subject. The emitted light is reflected by the main mirror 2 via a light projecting prism 22 and provided on a display portion of the focus plate 7. The light is bent in the vertical direction by the micro prism array 7 a and reaches the photographer's eye 15 through the pentaprism 8 and the eyepiece 11.

The micro prism array 7a is located at a plurality of positions (ranging points) corresponding to the focus detection area of the focus plate 7.
Are formed in the shape of a frame, and each of them is provided with five superimposing LEDs 21 (each LED-L1, LED
-L2, LED-C, LED-R1, LED-R2).

Thus, as can be seen from the finder field of view shown in FIG.
Reference numerals 1, 202, 203, and 204 illuminate in the finder visual field to display a focus detection area (distance measurement point) (hereinafter, this is referred to as a superimposed display).

Reference numeral 23 denotes a field mask for forming a field of view in the viewfinder, and 24 denotes an LCD in the viewfinder for displaying photographing information outside the field of view of the viewfinder.
-LED) 25.

The light transmitted through the LCD 24 is transmitted to the triangular prism 2
The light is guided into the field of view of the finder by the reference numeral 6, and 207 in FIG.
Are displayed outside the finder field of view, as shown by, so that the photographer can know the photographing information.

Reference numeral 31 denotes an aperture provided in the taking lens 1;
A diaphragm driving device including a diaphragm driving circuit 111 described later;
Reference numeral 3 denotes a lens driving motor, reference numeral 34 denotes a lens driving member including a driving gear and the like, and reference numeral 35 denotes a photocoupler which detects rotation of a pulse plate 36 interlocked with the lens driving member 34 and transmits the rotation to a lens focus adjustment circuit 110. Focus adjustment circuit 110
Drives the lens driving motor 33 by a predetermined amount based on this information and the information on the lens driving amount from the camera side, and moves the photographing lens 1 to the focusing position.
Reference numeral 37 denotes a mount contact which serves as an interface between a known camera and a lens.

FIG. 3 is an explanatory diagram of an electric circuit built in the camera of the present invention. 1 are given the same numbers.

A central processing unit (hereinafter referred to as a CPU) 100 of a microcomputer built in the camera body includes a line-of-sight detection circuit 101, a photometry circuit 102, and an automatic focus detection circuit 10.
3. Signal input circuit 104, LCD drive circuit 105, LE
A D drive circuit 106, an IRED drive circuit 107, a shutter control circuit 108, and a motor control circuit 109 are connected. Also, a focus adjustment circuit 1 disposed in the taking lens
10. Signals are transmitted to the aperture drive circuit 111 via the mount contacts 37 shown in FIG.

EEPROM 10 attached to CPU 100
Reference numeral 0a has a storage function of a line of sight correction data for correcting an individual difference in line of sight as storage means. Eye-gaze detection circuit 101
Converts the output of the eyeball image from the image sensor 14 (CCD-EYE) from analog to digital and converts this image information to the CPU 10.
Send to 0. As will be described later, the CPU 100 extracts each feature point of the eyeball image necessary for eye gaze detection according to a predetermined algorithm, and further calculates the gaze of the photographer from the position of each feature point.

The photometric circuit 102 amplifies the output from the photometric sensor 10, performs logarithmic compression and A / D conversion, and sends the result to the CPU 100 as luminance information of each sensor. The photometric sensor 10 is located at the left distance measuring point 2 in the viewfinder visual field shown in FIG.
SPC-L for metering the left area 210 including 00 and 201
SPC-C for metering a central area 211 including a center ranging point 202, an SPC-R for metering a right area 212 including right ranging points 203 and 204, and their peripheral areas 21.
It is composed of SPC-A for photometry 3 and photodiodes for photometry in four regions.

As shown in FIG. 2, the line sensor 6f shown in FIG. 3 includes five sets of line sensors CCD-L2 and CCD-L corresponding to the five ranging points 200 to 204 in the screen.
It is a known CCD line sensor composed of 1, CCD-C, CCD-R1, and CCD-R2.

The automatic focus detection circuit 103 A / D converts the voltage obtained from the line sensor 6f,
Send to 0. SW-1 is O with the first stroke of the release button.
N, a switch for starting photometry, AF, line-of-sight detection operation, etc., SW-2 is a release switch that is turned on by the second stroke of the release button, SW-AEL is an AE lock switch that is turned on by pressing the AE lock button, SW −DI
AL1 and SW-DIAL2 are electronic dials (not shown)
The signal is input to an up / down counter of the signal input circuit 104 by a dial switch provided therein, and the rotation click amount of the electronic dial is counted.

Reference numeral 105 denotes a known LCD drive circuit for driving and driving the liquid crystal display element LCD, and displays the aperture value, the shutter time, the set photographing mode, and the like on the monitor LCD 42 and the LCD 24 in the finder according to a signal from the CPU 100. Both can be displayed at the same time. L
The ED drive circuit 106 is an illumination LED (F-LED) 25
And superimpose LED 21 are turned on and off. The IRED drive circuit 107 includes an infrared light emitting diode (I
RED1 to 6) 13a to 13f are selectively turned on according to the situation.

The shutter control circuit 108 controls the magnet MG-1 for running the front curtain and the magnet MG-2 for running the rear curtain when energized, and exposes the photosensitive member to a predetermined amount of light. The motor control circuit 109 controls a motor M1 for winding and rewinding the film and a motor M2 for charging the main mirror 2 and the shutter 4. These shutter control circuits 108,
A series of camera release sequences is operated by the motor control circuit 109.

Next, FIG. 4 shows a flowchart of the operation of the camera having the visual axis detection device, and the following description is based on these flowcharts.

When the mode dial (not shown) is rotated to set the camera from a non-operation state to a predetermined shooting mode (this embodiment will be described based on the case where the shutter priority AE is set), the power supply of the camera is turned off. ON (# 100), CP
Variables used for U100 gaze detection are reset (#
101).

Then, the camera stands by until the release button is pressed and the switch SW1 is turned on (# 102). When the signal input circuit 104 detects that the release button has been pressed and the switch SW1 has been turned on, the CPU 100 checks with the visual line detection circuit 101 (# 103).

At this time, if the line-of-sight prohibition mode is set, a specific distance-measuring point is selected by the distance-measuring-point automatic selection subroutine (# 116) without executing line-of-sight detection, that is, without using line-of-sight information. At this distance measuring point, the automatic focus detection circuit 1
03 performs a focus detection operation (# 107).

As described above, the photographing mode in which the focus detection point is selected without using the line-of-sight information (the line-of-sight prohibition automatic focus photographing mode) and the photographing mode in which the distance measurement point is selected using the line-of-sight information (the line of sight automatic focus photographing mode) Both are provided, and the photographer can arbitrarily select whether or not to set the gaze prohibition mode.

There are several possible algorithms for automatic selection of ranging points. For example, a near-point priority algorithm in which the center ranging point is weighted is effective.
Since this is not directly related to the present invention, the description is omitted.

If the visual axis detection mode is set, visual axis detection is executed (# 104). At this time, the LED driving circuit 10
6 turns on the lighting LED (F-LED) 25, and LC
The D drive circuit 105 turns on the line-of-sight input mark 78 of the LCD 24 in the finder, and the photographer can confirm that the camera is performing line-of-sight detection from outside the finder screen 207.

Here, the line of sight detected by the line of sight detection circuit 101 is converted into coordinates of the gazing point on the focus plate 7. C
PU 100 selects a ranging point close to the gazing point coordinates,
A signal is transmitted to the LED drive circuit 106 to blink the distance measuring point mark using the superimposing LED 21 (# 105).

When the photographer sees that the distance measuring point selected by the photographer's line of sight is displayed, recognizes that the distance measuring point is incorrect, releases the release button and turns off the switch SW1 ( (# 106), the camera waits until the switch SW1 is turned on (# 102).

As described above, the fact that the focus detection point has been selected based on the line-of-sight information is indicated by blinking the focus detection mark in the viewfinder field, so that the photographer can be selected at will. You can check if it is.

Further, after the photographer has displayed the distance measuring point selected by his / her line of sight, the switch SW1 is subsequently turned on.
If the detection is continued (# 106), the automatic focus detection circuit 103 executes the focus detection of one or more ranging points using the detected line-of-sight information (# 107).

Here, it is determined whether or not the selected ranging point cannot be measured (# 108).
A signal is sent to the drive circuit 105, and the LCD 2 in the finder is
The focus mark 4 is blinked to warn the photographer that the distance measurement is NG (impossible) (# 118), and the operation is continued until SW1 is released (# 119).

Distance measurement is possible, and if the focus adjustment state of the distance measurement point selected by a predetermined algorithm is not in focus (# 10
9), the CPU 100 sends a signal to the lens focus adjustment circuit 110 to drive the photographing lens 1 by a predetermined amount (# 117). After the lens is driven, the automatic focus detection circuit 103 performs focus detection again.
(# 107), it is determined whether or not the taking lens 1 is in focus (# 109).

If the photographing lens 1 is in focus at a predetermined distance measuring point, the CPU 100
To turn on the focus mark on the LCD 24 in the finder, and also sends a signal to the LED drive circuit 106 to display the focus on the focusing point 201 in focus (#
110).

At this time, the blinking display of the distance measuring point selected by the line of sight is turned off, but the distance measuring point displayed by focusing often coincides with the distance measuring point selected by the line of sight. The focusing distance measuring point is set to a lighting state so that the photographer can recognize that the subject is in focus. When the photographer sees that the focus point is displayed in the viewfinder, recognizes that the focus point is incorrect, releases the release button and turns off the switch SW1 (# 111), and the camera continues. Waits until the switch SW1 is turned on (# 102). If the photographer continues to turn on the switch SW1 while watching the focusing point displayed in focus (# 111), the CPU 100 sets the photometry circuit 1 to ON.
A signal is transmitted to 02 to perform photometry (# 112). At this time, a weighted exposure value is calculated for the photometric regions 210 to 213 including the focused distance measuring point.

When the release button is further depressed and the switch S
It is determined whether or not W2 is ON (# 113),
If switch SW2 is OFF, switch SW2 is turned off again.
Check the status of 1 (# 111). If the switch SW2 is turned on, the CPU 100 sets the shutter control circuit 1
08, the motor control circuit 109, and the aperture drive circuit 111.

First, power is supplied to M2 to raise the main mirror 2 and the aperture 31 is stopped down. Then, power is supplied to MG1 to open the front curtain of the shutter 4. The aperture value of the aperture 31 and the shutter speed of the shutter 4 are determined from the exposure value detected by the photometric circuit 102 and the sensitivity of the film 5.
M after a lapse of a predetermined shutter time (for example, 1/250 second)
G2 is energized to close the rear curtain of shutter 4. When the exposure of the film 5 is completed, power is supplied to M2 again, mirror down and shutter charging are performed, and power is supplied to M1 to feed the film, thereby completing a series of shutter release sequence operations. (# 114) Then
The camera waits until switch SW1 is turned on again
(# 102).

FIG. 5 and FIG. 6 are flowcharts of gaze detection according to the first embodiment of the present invention. As described above, upon receiving a signal from the CPU 100, the gaze detection circuit 101 executes gaze detection (# 104).

First, the CPU 100 selects an appropriate combination of infrared light emitting diodes (IREDs) Ba to Bf for illuminating the photographer's eyes and turns them on (# 20).
1). The IRED is selected by a posture switch (not shown) depending on whether the camera is in the horizontal position or the vertical position, or whether the photographer wears glasses.

Next, charge is stored in the image sensor 14 for a predetermined storage time (# 202). When the accumulation is completed, the IRED is also turned off (# 203).

The CPU 100 reads the eyeball image of the photographer from the image sensor 14 for which the accumulation has been completed, and at the same time, sequentially performs the process of extracting the P image and the feature of the pupil (# 204). A specific method at this time is described in, for example, Japanese Patent Application No. 3-121098 filed by the present applicant.

When the reading of the entire eyeball image is completed, P
After the image and pupil feature extraction is completed, a set of P image positions is detected based on the information (# 205). As described above, since the P image is a corneal reflection image of the IRED for eyeball illumination, it appears in the image signal as a bright spot having a high light intensity. Candidate), and its position (xd ', yd'), (xe ', y
e '). As described above, the CPU 100 has a function of an image detecting unit that detects a corneal reflection image candidate.

In the case of an eyeball image as shown in FIG. 7, since there is only one set of luminescent spots which seem to be a P image, its detection is easy,
As shown in FIG. 9, when a ghost caused by the glasses worn by the photographer has occurred, it becomes difficult to detect a correct P image pair. In order to recognize that the image is a P image, a ghost caused by eyeglasses generally has a large bright spot.
Alternatively, it is desirable to add the size of the bright spot on the image sensor to the condition.

If one P image pair can be determined in (# 205), the pupil center (xc ', yc') and pupil diameter (rc) are detected (# 206). The method of detecting the pupil center and the pupil diameter at this time is described in Japanese Patent Application No. 3-121098.

The position of the P image and the pupil are detected from the eyeball image of the photographer, and the coordinates in the line of sight of the photographer or in the viewfinder field are calculated from equation (5) (# 207).

In this embodiment, whether or not the line of sight has been detected correctly at this time is determined based on distance information from distance detecting means for detecting distance information from a predetermined position of the apparatus to the eyeball. A specific method of obtaining the distance is performed as follows.

Using the distance ΔP = | xd′−xe ′ | between the positions xd ′ and xe ′ of the two P images on the image sensor 14, for example,

[0082]

(Equation 1) The distance L is obtained by the following equation.

As an example, when the distance L between the camera and the observer's eyeball is 25 mm, assuming an optical system in which the distance ΔP on the image sensor 14 is about 8 pixels, a
_By setting ₁ = −165, a ₂ = about 3.8, Δ
Once P is determined, the distance L can be calculated. And
The obtained distance L is, for example, 5 mm or less, or 40 mm
When the above result is obtained, such a situation is impossible, so that it is determined that the gaze detection at this time is unusable.

That is, the line-of-sight information is invalidated. CPU10
0 has a function as invalidating means for invalidating the line-of-sight information as described above. Then, if the detection result is OK, it is regarded that "the gaze detection is successful"(# 209), and the routine of the gaze detection is returned (# 210).

On the other hand, when the detection result is not normal (that is, when the distance information is out of the predetermined range), it is determined that there is a possibility that the P image pair has been erroneously selected, and another P image pair (corneal reflection image candidate) is determined. Check if there is (# 211).

In the case of an eyeball image as shown in FIG. 9, a pair of P images includes reflected images (52a, 52b), (53a, 53).
b) and (54a, 54b). Of course, the correct P image pair is (52a, 52b), and the reflected image (5
The pair of 3a, 53b) and (54a, 54b) is a ghost with glasses.

If the P image pair (52a, 52b) is selected from the beginning, a correct detection result is given. However, if the pair of reflection images (53a, 53b) is selected first and the line of sight is detected, the detection result is It is not normal.

Therefore, if there are a plurality of P image pair candidates as in the example of FIG. 9, another P image pair is selected (# 212), and eye gaze detection is attempted again (T1).

If there is only one P image pair in the first place as shown in FIG. 8, even if it is an incorrect P image pair, it is impossible to select another P image pair again. In that case, it is regarded as "line-of-sight detection failure"(# 213),
The routine returns to the line of sight detection (# 214). In the present embodiment, the correct P image pair is selected as described above, and thereby the line-of-sight information of the photographer is obtained with high accuracy.

[0090]

As described above, according to the present invention,
An illuminating unit that illuminates an eyeball, and a light receiving unit that receives an image of the eyeball illuminated by the illuminating unit, and a line-of-sight detecting device that detects a line of sight by using an output of the light receiving unit. Image detection means for detecting a corneal reflection image candidate of the eyeball from the output, gaze detection means for detecting a gaze using the corneal reflection image candidate, and the gaze detection device from the eyeball using the corneal reflection image candidate Distance detection means for detecting the distance to, and when the detected distance is not within a predetermined range, the gaze detection means selects a different corneal reflection image candidate and performs gaze detection, thereby obtaining an eyeball image. Even when there is an image that is erroneously selected as the P image, it is possible to determine whether or not the correct P image has been selected and the gaze detection has been performed, so that highly accurate gaze detection can be performed.

[Brief description of the drawings]

FIG. 1 is a schematic diagram of a main part of a first embodiment in which the present invention is applied to a single-lens reflex camera.

FIG. 2 is an explanatory view of the viewfinder in FIG. 1;

FIG. 3 is an electric circuit diagram of a camera having a visual line detection device according to the present invention.

FIG. 4 is a flowchart of the operation of FIG. 3;

FIG. 5 is a flowchart of gaze detection according to the present invention.

FIG. 6 is a flowchart of gaze detection according to the present invention.

FIG. 7 is a diagram illustrating the principle of a conventional gaze detection method.

FIG. 8 is an explanatory diagram of an eyeball image.

FIG. 9 is an explanatory diagram of an eyeball image when wearing glasses.

[Explanation of symbols]

Reference Signs List 1 shooting lens 2 main mirror 6 focus detection device 6f image sensor 7 focus plate 10 photometric sensor 11 eyepiece 13 infrared light emitting diode (IRED) 14 image sensor (CCD-EYE) 15 eyeball 16 cornea 17 iris 21 LED for superimpose 24 LCD in viewfinder 25 LED for illumination 31 Aperture 50 White part of eyeball 51 Pupil part of eyeball 100 CPU 101 Eye gaze detection circuit 103 Focus detection circuit 104 Signal input circuit 105 LCD drive circuit 106 LED drive circuit 107 IRED drive circuit 110 Focus adjustment circuit 200 to 204 AF point mark (calibration target)

Claims (4)

(57) [Claims] [Claim 1 further comprising illuminating means for illuminating the eye, and a light receiving means for receiving an image of the eye illuminated by said illuminating means, the line of sight by using an output of said light receiving means
An eye-gaze detecting device that detects a candidate corneal reflection image of the eyeball from an output of the light receiving unit.
Gaze detection means for detecting a gaze using the corneal reflection image candidate
Means, a distance detecting means for detecting a distance from the eyeball to the visual line detection device using the corneal reflection image candidate, and the distance detecting means for detecting the distance when the detected distance is not within a predetermined range.
The line detecting means selects a different corneal reflection image candidate and detects the line of sight.
A gaze detection device for performing a gaze. 2. Illumination means for illuminating an eyeball; Receiving the image of the eyeball illuminated by the illumination means;
Light receiving means, and using the output of the light receiving means
In a gaze detection device that detects Detecting a cornea reflection image candidate of the eyeball from the output of the light receiving means
Image detecting means; Gaze detection for detecting a gaze using the corneal reflection image candidate
Means, Sight line detection from the eyeball using the corneal reflection image candidate
Distance detecting means for detecting the distance to the device; If the detected distance is not within the predetermined range, the
The line detecting means selects a different corneal reflection image candidate and detects the line of sight.
Out, and the distance detecting means outputs the selected different angle.
The feature is to detect the distance using the film reflection image candidate
Eye gaze detection device. 3. A corneal reflection image candidate detected using the corneal reflection image candidate
When the distance is within a predetermined range, the corneal reflection image candidate is used.
That the detected gaze is the correct detection result
The gaze detection device according to claim 1 or 2, wherein: Wherein said illuminating means includes a pair of light sources, the distance detecting means, seeking distance between the two cornea reflection images generated in the eye by the pair of light sources from the light receiving result of the light receiving means, wherein from claim 1 characterized by detecting the distance from the eyeball to the visual axis detecting apparatus in any of the 3
Line-of-sight detection device as claimed.