JP3210089B2 - Eye gaze detection device and camera - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye-gaze detecting device most suitable for a single-lens reflex camera, a still video camera, a video camera and the like.

[0002]

2. Description of the Related Art Conventionally, as disclosed in Japanese Patent Application Laid-Open No. 3-109029, there is known an eye-gaze detecting device built in a camera. By the way, there are various ways to look through the viewfinder of the camera. Some people look closely at the eyepiece and some people look far from it. In particular, when wearing glasses, the distance from the eyepiece tends to be long. At this time, in order to prevent deterioration of an image on a gaze detection sensor, a camera having a mechanism for changing the configuration of a gaze detection optical system according to the presence or absence of glasses is disclosed in Japanese Patent Application Laid-Open No. 4-138431. I have.

[0003] However, a gaze detection device that can handle various ways of looking through a viewfinder has not been realized yet.

[0004] Further, when separating a light beam used for line-of-sight detection from a finder optical system using a beam splitter,
A ghost caused by the beam splitter may cause erroneous detection of the line of sight. Japanese Patent Laid-Open No. 2-5 discloses a technique for slightly tilting each transmission surface of the beam splitter as a countermeasure for this. However, as the number of parallel surfaces or surfaces perpendicular to each other among the constituent surfaces of a beam splitter generally made of glass decreases, the difficulty in manufacturing increases, and as a result, the manufacturing cost increases.

[0005]

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned drawbacks, and it is an object of the present invention to provide an eye-gaze detecting device that does not reduce the eye-gaze detection accuracy even when an observer looks into an observation system from any distance. It is another object of the present invention to realize a simple visual axis detection optical system free from ghost.

[0006]

[Means for Solving the Problems] In order to solve the above-mentioned problems
The invention according to claim 1 of the present application provides a light source for illuminating an observer's eye observed by an observation optical system, a sensor, and an image forming method for forming an image of the observer's eye on the sensor. A lens unit, and a line-of-sight detection device that detects the line of sight of the observer based on a reflection image generated on the sensor from the eye of the observer, and among the sensor outputs, the cornea by the light source
A / D converting an analog signal with a lower value than the reflected image
Performs an operation for detecting lines, the imaging lens unit which has features that the focus at a position closer to the observation optical system than the eye point of the optical system for the observation
You. Further, in order to solve the above-mentioned problem, the present invention relates to claim 3 of the present application.
The invention according to the present invention has a light source for illuminating an observer's eye observed by an observation optical system, a sensor, and an imaging lens unit for forming an image of the observer's eye on the sensor. A line-of-sight detection device that detects a line of sight of an observer based on a reflected image from the observer's eye generated on the sensor, wherein the imaging lens unit includes a corneal reflection image recognized from the sensor output . magnitude, and are configured to not substantially vary depending on the distance of the observer's eyes from the observation optical system that
It is characterized by. In order to solve the above problems,
The invention according to claim 4 of the present application provides a light source for illuminating an observer's eye observed by a finder optical system, an area sensor, and an imaging lens for forming an image of the observer's eye on the area sensor. Unit, the output of the area sensor is input to a non-linear amplifier, the output of the non-linear amplifier, the corneal reflection image of the light source generated on the area sensor using the relative relationship between the pupil image and the A line-of-sight detection device that detects a line of sight of an observer, wherein the imaging lens unit is configured to focus on a position closer to the finder optical system than an eye point of the finder optical system.
It is characterized by having. With these configurations, a gaze detection device and a camera are realized in which the gaze detection accuracy does not decrease even if the observer looks into the finder from any distance.

Further, in order to solve the above-mentioned problems, the present invention
The invention according to claim 5 , wherein a light source for illuminating an observer's eye to be observed by the observation optical system, a sensor, and an optical path of the observation optical system for guiding a light beam from the observer's eye to the sensor. A beam splitter disposed therein, and a light flux from the observer's eye disposed adjacent to the beam splitter for forming an image of the observer's eye on the sensor. An image lens unit, and a line-of-sight detection device that detects the line of sight of the observer based on a reflection image generated on the sensor from the observer's eyes, wherein the imaging lens unit is a lens. It is characterized by having a stop provided with an eccentric aperture having a different optical axis . In addition, the above problems
In order to solve the above problem, the invention according to claim 6 of the present application is directed to a light source for illuminating an observer's eye observed by an observation optical system, a sensor, and a light flux from the observer's eye to the sensor. A beam splitter disposed in an optical path of the observation optical system, and an imaging lens for forming an image of the observer's eye on the sensor with a light beam from the observer's eye from the beam splitter. A line-of-sight detection device that detects a line of sight of an observer based on a reflected image from the observer's eye generated on the sensor, wherein the imaging lens unit includes a corneal reflection unit of the light source. It is characterized in that it has a diaphragm for blocking reflected light generated from the sensor by an image from entering the beam splitter . In addition, to solve the above-mentioned problem,
The invention according to claim 7 of the present application is directed to a light source for illuminating an observer's eye observed by an observation optical system, a sensor, and the observation device for guiding a light beam from the observer's eye to the sensor. A beam splitter disposed in the optical path of the optical system, and an imaging lens unit for forming an image of the observer's eye on the sensor with a light beam from the observer's eye from the beam splitter. A gaze detection device that detects a gaze of the observer based on a reflection image generated from the eye of the observer on the sensor, wherein the imaging lens unit includes a cornea of the light source on the sensor. It is characterized by having a stop provided with an optical axis passing through a position deviating from the position where the reflection image is formed . these
With this configuration, the light reflected from the sensor
May cause ghosting
And has prevented.

[0008]

FIG. 8 is a diagram schematically showing a main part of a camera incorporating a visual line detection device according to the present invention. In FIG. These are interchangeable lenses that are freely mounted, and are connected with the camera mount 5a and the lens mount 351a.

In the camera body 5, reference numeral 10 denotes a movable half mirror, reference numeral 12 denotes a pentaprism, and reference numeral 14 denotes an eyepiece, which constitutes a finder optical system. Further, the finder may look at the image display tube. Reference numeral 306 denotes a photometric light receiving element. A photometric operation circuit 307 is connected to the film sensitivity information input circuit 308, the shutter control circuit 309, and the microcomputer 310 (hereinafter, MCU).

The microcomputer 310 has a built-in A / D converter, and this A / D converter converts an analog signal from the light-receiving element for photometry described above or an analog signal from a distance measuring sensor to be described later into a digital signal.

Reference numeral 334 denotes a focal plane shutter, behind which a silver halide film or a solid-state image sensor is disposed. Reference numeral 311 denotes a mirror driving cam, and 312 denotes a mirror driving motor, which is connected to the motor drive circuit 313. A film winding / rewinding motor 314 is connected to the motor drive circuit 313 like the mirror driving motor 312. 55 is focus detection (ranging)
The sensor is connected to the distance calculation circuit 317.

Reference numeral 43 denotes an area sensor such as a CCD, which is connected to the visual line detection circuit 330. The aperture ratio of a CCD is generally 20 to 75% and has a dead zone of 80 to 25%. Further, reference numeral 25 denotes a display element formed of a liquid crystal display panel, which is connected to a display drive circuit 331 together with an LED for display illumination.

Reference numeral 318 denotes a battery for operating the entire camera system; 319, a main power switch; and 320, a DC / DC switch.
A DC converter is connected from the battery 318 to the microcomputer 310. Reference numeral 321 denotes a photometric distance measuring switch, and 322 denotes a release switch. Generally, the switches 321 and 322 are two-stage stroke switches, and are configured such that the switch 321 is turned on by a first stroke of a release button (not shown) and the switch 322 is turned on by a second stroke. . Also, S
Reference numeral WS denotes a switch connected to a setting dial (not shown), a shooting mode selection member, and a distance measurement field fixing button.

Reference numerals 323a to 323e denote contact pin groups on the camera body 5 side, which are installed near the mount 5a. On the other hand, reference numerals 352a to 352e denote contact pin groups on the interchangeable lens body 6, which face the contact pin groups 323a to 323e on the camera side.

On the interchangeable lens 6 side, reference numeral 7 denotes a photographing optical lens such as a zoom lens or a single focus lens. 3
Reference numeral 53 denotes a lens driving motor used for focus adjustment, which is connected to the lens driving circuit 354. The number of pulses is input to the counter 362 via the ratchet 360 by the rotation of the lens driving motor 353. An aperture driving circuit 361 is connected to the microcomputer 355,
The motor 35 is connected to the pulse motor 356.
The diaphragm 9 is driven by 6. Reference numeral 357 denotes a zoom drive motor used for zooming, and is connected to the zoom drive circuit 358. 359a to 359e are encoders for transmitting the lens focal length to the microcomputer 355, and 363 is a power zoom switch.

FIGS. 1 and 2 are diagrams for explaining in detail the optical configuration of a visual axis detection device to which the present invention is applied.

First, FIGS. 2A and 2B show a camera, in particular, an example of a single-lens reflex camera.
3 is a cross-sectional view of the camera as viewed from the side, and FIG. 3B is a cross-sectional view of the camera as viewed from above. In the figure, 5 is a single-lens reflex camera body, 6
Reference numeral denotes a lens barrel that moves and holds the photographing lens 7 in the direction of the optical axis 8 based on a command from the single-lens reflex camera body 5. Reference numeral 9 denotes an aperture for restricting a luminous flux, which defines a so-called F number.

Reference numeral 10 denotes a half mirror, 11 denotes a focusing screen having a Fresnel lens 11a on a light incident surface and a matte surface 11b on a light exit surface, and provided adjacent to the display element 25 described above. 12 is a pentaprism, 13
Is a beam splitter for separating infrared light for line-of-sight detection, and 14 is an eyepiece, which constitutes a finder system.

FIG. 2B shows an eccentric diaphragm 47,
Is an imaging lens that forms an image of the eyeball 48b on the area sensor 43.

In FIG. 2A, reference numeral 16 denotes a known multi-point focus detecting device including the above-described distance measuring sensor 55 and the distance measuring operation circuit 317. The object light transmitted through the half mirror 10 is disposed behind the same. The light is guided through the provided sub-mirror 15.
One of the distance measurement areas of the focus detection device 16 is selected by the output of the eye gaze detection device.

Reference numeral 17 denotes a front plate that holds the above-described finder components and the focus detection device and constitutes a mirror box.

Next, the gaze detecting device will be described. FIG.
FIG. 2 is a perspective view showing components of a visual line detection device. In the figure, reference numerals 13 and 14 denote a beam splitter and an eyepiece shown in FIG. 2, and a dichroic mirror 13a of the beam splitter 13 reflects infrared light used for line-of-sight detection to form an optical path for line-of-sight detection. ing. 40a, 40
An infrared light emitting diode (IRED) disposed below the eyepiece 14 irradiates infrared light toward the eyeball of the finder observer. The light beam reflected by the cornea of the eyeball passes through the eyepiece lens 14 and enters the incident surface 13b of the beam splitter 13. Inside the beam splitter 13, the light is first reflected by the dichroic mirror 13a, then totally reflected by the entrance surface 13, and then exits from the exit surface 13c. An eccentric diaphragm 47 and an image forming lens 42 are provided so as to face the exit surface 13c, and an eyeball 48b of a finder observer and an IRED
An image (P image described later) of the light emitting units 40a and 40b is formed.

In general, the reflectivity of the light receiving portion of an area sensor such as a CCD is considerably high, and a part of incident light is irregularly reflected without being photoelectrically converted. Further, since the imaging magnification of the imaging lens 42 is considerably small, the light receiving portion of the area sensor 43 is almost at the focal position of the imaging lens 42. As a result, the reflected light from the area sensor 43 is reflected by the exit surface of the beam splitter 13. When the light is reflected again at 13c, a spot-like ghost is generated on the area sensor.

In order to prevent such a ghost from occurring, the eccentric diaphragm 47 is, as shown in FIG.
The center M of the opening 47a is set below the second optical axis L.

That is, since the IREDs 40a and 40b are located below the eyepiece 14, an IRED image generated by reflection from the cornea of the eyeball is formed above the optical axis L on the light receiving portion of the area sensor 43. However, the aperture 47a is arranged on the side opposite to the IRED image so that the reflection component close to the regular reflection of the reflected light does not pass through the aperture 47a.

With this configuration, it is possible to perform highly accurate line-of-sight detection without being affected by ghost reflections on the sensor surface.

Next, the principle of gaze detection will be described with reference to FIGS. FIG. 4 shows a light receiving section 43 of the area sensor 43.
a and an image 48 of the anterior segment of the finder observer projected thereon. The two bright spots visible inside the pupil 49 shown as black circles in the center of the figure are IREDs.
The corneal reflection images 40a and 40b are so-called Purkinje first images (P images). Since the curvature of the cornea is as small as about 7.7 mm, the position in the depth direction of the virtual image generated by the reflection is substantially the same as the position of the iris. Therefore, the P image is also projected on the area sensor with the same sharpness as the eye image.

In order to know the line of sight of the finder observer, the relative positional relationship between the center of the two P images and the center of the pupil 49 may be obtained, and the rotation angle of the eyeball may be calculated.

FIG. 5 shows the horizontal line output of the area sensor 43a at the position of the arrow A shown in FIG. 4, and more specifically, the output of the linear amplifier in the visual line detection circuit 330. The two peaks at the center correspond to the P image, the area B corresponds to the pupil, the areas C1 and C2 correspond to the iris, and the areas D1 and D2 correspond to the white eye. The intensity of the P image is extremely high as compared with the contrast of the edge of the pupil, and in order to detect the pupil edge with high accuracy when determining the center of the pupil, only the range indicated by the arrow E is subjected to A / D conversion. Will be used for the subsequent arithmetic processing.

As a result, even if the P image is defocused and projected on the area sensor, a signal having a sharp rise and fall is obtained in order to enlarge the bottom of the blurred image.

When the user of the eye-gaze detecting device wears glasses, the IRED light reflected by the glasses is reflected by the eyepiece or the beam splitter, reflected again by the glasses, and projected onto the area sensor. A ghost-forming optical path exists. The ghost generated in this way is I
Because the virtual image of RED can be far away, it is quite spot-like,
In addition, when the infrared reflectance of the glasses is high, the intensity becomes several times the pupil level. Generally, this ghost is mistakenly
Although detection as an image is conceivable, the optical configuration described later, which is a feature of the present embodiment, can prevent such a problem beforehand.

As an area sensor, for example, a CCD
As described above, there is a dead zone for each pixel when is used, but this configuration also has the disadvantage that the image of the IRED is buried in this dead zone and is not output when the eye moves away from the eyepiece. There is also the advantage of preventing it.

Next, a method of detecting a P image and a method of detecting a line of sight will be described in detail with reference to FIGS.

FIG. 9 is a main flow chart of gaze detection. When the MCU 310 starts the line-of-sight detection operation, the data is initialized in step (# 001) through step (# 000).

The variable EYEMIN is a variable for storing the lowest luminance value in the photoelectric conversion signal of the eyeball reflection image, and is an A / D built in the microcomputer (MCU 310).
Assuming that the resolution of the converter is 8 bits, the lowest value is sequentially compared and updated as the image signal is read. The initial value stores 255 representing the maximum value of 8 bits.

The variable EDGCNT is a variable for counting the number of boundaries extracted between the iris and the pupil as edges.

Variables IP1, IP2, JP1, JP2 are variables representing the position of the P image of the corneal reflection of the light emitting diodes 40a, 40b, and are in the lateral (X-axis) range IP1 to IP2.
2. Two P images exist in the region of the eyeball reflection image surrounded by the ranges JP1 and JP2 in the vertical direction (Y axis).

Now, the number of pixels of the area sensor 43a is assumed to be 150 pixels in the horizontal direction and 100 pixels in the vertical direction, and IP1, IP2, JP1 and JP2 indicate the position (75, 50) in the center of the whole. It is stored as an initial value.

Following the data initialization, step (# 00)
Go to 2).

In step (# 002), the light emitting diodes 40a and 40b for the P image are turned on. Next step (#
At 003), the accumulation operation of the area sensor 43a is started. Since the control of the sensor has no direct relation to the present invention, a detailed description thereof will be omitted. However, in the embodiment of the present invention, the drive is controlled by a sensor interface circuit (not shown).

In step (# 004), the process waits until the accumulation of the area sensor is completed.

When the predetermined charge accumulation is completed, the light emitting diode is turned off in the next step (# 005).

From the next step (# 006) onward, reading of the photoelectric conversion signal of the area sensor is started.

In the step (# 006), the loop variable J is set to 0.
This represents a so-called "loop process" in which the process in the frame is executed while counting up from "99" to "99".

In the loop processing in step (# 006), first, in step (# 007), a photoelectric conversion signal of one line in the horizontal direction (X-axis) of the area sensor is read. The reading of one line is in a subroutine format. FIG. 10 shows a flowchart of the subroutine "1 line reading".

When the subroutine "1 line read" is called, the next step (# 101) is executed via the step (# 100) in FIG. Step (# 10
1) and the step (# 102) in the frame represent the same loop processing as the above-described step (# 006).
In step (# 101), the process in each frame is executed while incrementing the variable K from 0 to 3 and in step (# 102), incrementing the variable I from 0 to 149. Therefore, the step (# 101) and the step (# 102) include the variable K and the variable I
Represents a so-called "nested" loop process. In step (# 103) in the loop processing of step (# 102), the array variable IM (i, k) is restored.

In this embodiment, the microcomputer MC
Although the U310 performs signal processing, the storage capacity of the built-in RAM (random access memory) of the microcomputer is generally not large enough to store all pixel information of the area sensor at one time. Therefore, in the present embodiment, while reading out the image signals output from the area sensor one by one, only the latest image signals corresponding to five lines in the horizontal direction (X axis) are stored in the built-in RAM of the microcomputer. The processing for gaze detection is executed every time a line is read.

From step (# 101) to step (# 1)
The content executed in the double loop processing of 03) is an operation of updating the stored image signal data of the past five lines in order to read a new image signal of one line. That is, among the array variables IM (i, k), IM (i, 0)
[I = 0 to 149] is the oldest and IM (i, 4)
[I = 0 to 149] represents the most recent one-line image data. The data is updated as follows to obtain a new one-line image signal as IM (i, 4) [i = 0 to 149].

IM (i, 0) ← IM (i, 1) IM (i, 1) ← IM (i, 2) IM (i, 2) ← IM (i, 3) IM (i, 3) ← IM (I, 4) [i = 0 to 149]

When the loop process for updating data in steps (# 101) to (# 103) is completed, the loop process in the next step (# 104) is executed.

In the loop processing of step (# 104),
One line in the horizontal direction (X axis) of the area sensor (150
A / D conversion of the image signal of the pixel
And the minimum value of the image signal is detected.

In the first step (# 105) in the loop of step (# 104), the microcomputer MC
The digital value ADC obtained by A / D converting the image signal is taken out from the built-in A / D converter of U310, and the value is temporarily stored in the conversion EYEDT. Then, in the next step (# 106), the value of EYEDT is stored in the array variable IM.
(I, 4). The variable I is counted up from 0 to 149 in the outer loop processing step (# 104).

Steps (# 107) and (# 108) are processing for detecting the minimum value of the image signal. The variable EYEMIN is a variable that holds the minimum value of the image signal, and is set in step (# 10).
In 7), if EYEDT is smaller than EYEMIN, the process branches to step (# 108), and EYEMIN
Is updated with this small EYEDT value.

When the loop processing of steps (# 104) to (# 108) is completed and storage of a new image signal for one line and detection of the minimum value are completed, the next step (# 10)
In 9), the subroutine "read one line" is returned.

Returning to the flow chart of FIG. 9, when the subroutine "read one line" of step (# 007) is completed, the process proceeds to the next step (# 008) and the loop of the outer loop processing step (# 006) Variable J is 5
Check to see if this is the case.

The loop variable J represents a pixel line in the vertical direction (Y axis) of the area sensor. In this embodiment, since the number of pixels of the area sensor is (150 × 100), J is from 0 to 99. Counted up.

If the loop variable J is 5 or more in step (# 008), the flow branches to step (# 009). This is because when the number of lines of the read image signal becomes 5 or more,
This is because processing in the vertical direction (Y axis) of the area sensor can be performed.

In the step (# 009) ahead of the branch, a subroutine "detection of P image" is executed.

The subroutine "detection of P image" is a process for detecting the position of the P image of corneal reflection described above, and is executed every time one line in the horizontal direction (X axis) of the area sensor is read. The flowchart is shown in FIG.

When the subroutine "P image detection" is called, the program proceeds to step (# 20) through step (# 200).
The loop processing of 1) is executed. In the loop processing, the position of the P image in the image data (stored in the array variable IM (i, k)) is searched, and if found, the position on the area sensor is stored. In this embodiment, since two P images are generated, two pieces of position information are stored.

In the first step (# 202) in the loop, it is determined whether or not the image data at a predetermined position satisfies the condition as a P image. The conditions are as follows.

"P image condition" in step (# 202) IM (I, 2)> C1 and IM (I, 1)> C2 and IM (I, 3)> C2 and IM (I-1,2)> C2 and IM (I + 1,2)> C2 where C1 and C2 are threshold constants and have a relationship of C1 ≧ C2, for example, C1 = 230 and C2 = 200. The variable I is a loop variable of the loop processing, and represents the position of the area sensor in the horizontal direction (X axis).

The above conditions are based on the fact that the P image is like a spot image as described with reference to FIG.
It is defined in both directions of the vertical direction (X / Y axis). When this condition is satisfied, it is considered that a P image exists at the position (I, 2).

As described above, the array variable IM (i, k) is updated every time one line is read in the horizontal direction (X axis) of the area sensor, and the vertical (Y axis) position J line is set to IM.
(I, 4) [i = 0 to 149]. Therefore, the address (1, 2) for the variable IM is the position (I, J-2) on the area sensor.

In step (# 202), if there is image data that satisfies the condition of the P image, step (# 203)
Branching to the following, if not, the outer loop variable I is counted up.

Step (# 203) and subsequent steps are processing for determining the range of existence of the two P images (the range [IP1 to IP2] in the X-axis direction and the range [JP1 to JP2] in the Y-axis direction).

First, in step (# 203), a variable I and a variable I representing the position of the area sensor in the horizontal direction (X axis) are set.
P1 is compared. If I <IP1, step (# 20)
Branch to 4). In other words, the variable I is smaller than the position of the P image position IP1 on the left side in the horizontal direction in the P image existence range.
If the position is on the left, IP1 is to be rewritten.

In step (# 204), the value of the variable I is stored in the variable IP1, and the vertical position (J-
2) is stored in the variable JP1.

In steps (# 205) and (# 206), the P image position IP2 on the right side in the horizontal direction in the P image existence range and the update of JP2 representing the vertical position are determined.

As described above, step (# 201)
In the loop processing, the position I in the horizontal direction (X axis) is changed from 0 to 14
When the processing of one line up to 9 is completed, the process proceeds to the next step (# 207).

When the above process is completed, the subroutine "P image detection" is returned in the next step (# 208).

Returning to the flowchart of FIG.

The subroutine "P" of step (# 009)
When the "image detection" is completed, a subroutine "pupil edge detection" is executed in the next step (# 010).

The "pupil edge detection" is a subroutine for detecting the position of the pupil edge (the boundary between the iris and the pupil) in the eyeball reflection image.

When the subroutine "pupil edge detection" in step (# 010) is completed, the loop variable J (representing the vertical direction of the area sensor and the coordinates of the Y axis) in the outer loop processing step (# 006) is counted up. Then, the processing after step (# 007) is executed again until J becomes 99.

When the loop variable J becomes 99 and reading and processing of all the pixels of the area sensor are completed, the process proceeds from step (# 006) to step (# 011).

In steps (# 011) to (# 013), the center coordinates of the pupil and the line of sight are detected from the P image position and the pupil edge information detected in the loop processing in step (# 006).

First, in step (# 011), a subroutine "setting of estimated pupil range" is called.

The pupil edge points detected in the subroutine “pupil edge detection” in step (# 010) include those other than the edge points that actually represent the pupil circle (the circle formed by the boundary between the iris and the pupil). Also, false edge points generated by various noises are included.

"Setting of estimated pupil range" is a subroutine for limiting the coordinate range of a probable edge point based on P image position information in order to eliminate the false edge point.

Next, the subroutine "detection of pupil center" in step (# 012) is called.

“Detection of the pupil center” is based on the coordinates of the pupil edge points (center coordinates and size) based on the coordinates of the pupil edge points that are likely to be present.
Is a subroutine for estimating.

To estimate the shape of the pupil circle, the "least squares method"
Is used.

When the "detection of pupil center" in step (# 012) is completed, the process proceeds to step (# 013) and calls a subroutine "detection of line of sight". “Detection of line of sight” is a subroutine for detecting a line of sight (point of gaze) from the P image position and the center position of the pupil circle detected in the processing up to this point.

Basically, it is sufficient to calculate the rotation angle θ of the optical axis of the eyeball. In the embodiment of the present invention, the center of the pupil is moved in the horizontal direction (X
Axis) and the vertical direction (Y axis) are detected in two dimensions, so that not only the horizontal direction but also the direction of the visual line in the vertical direction can be detected.

When the detection of the line of sight is completed, step (# 0)
The process proceeds to 14), and a series of processing ends.

By the way, the condition of the P image shown in the description of the step (# 202) in FIG. 11 is based on the premise that the size of the P image after A / D conversion is constant about 3 pixels in each of the vertical and horizontal directions. As a result, the P image position can be detected with high accuracy under such a simple condition, and if this condition is broken, the detection accuracy is greatly reduced.

However, there are various ways of looking through the viewfinder of the camera. Some people look closely at the eyepiece lens 14 while others look from a far distance. In particular, when wearing glasses, the distance from the eyepiece 14 tends to be long, and the magnification of the eye image projected on the area sensor is not constant.

The feature of the present embodiment is that the P image can be detected with high accuracy even if there is such a change in the image magnification. Next, referring to FIGS. 6 and 7, an optical configuration for this purpose will be described. Will be described.

FIG. 6 is a development explanatory view of the visual axis detection optical system shown in FIG. 1 along the optical path.
A virtual lens representing a combined optical system of the eyepiece 14 and the eyepiece 14;
Is the optical axis, 61b is a virtual image of the cornea of the IRED 40a or 40b, and its position on the optical axis is matched with the eye point of the finder optical system. The imaging magnification by the virtual lens 62 is about -0.1. Also, the direction orthogonal to the optical axis is shown enlarged about five times. In addition,
For simplicity, the IRED will be described on the optical axis.

Consider the optical behavior when the virtual image position of the IRED changes in such an optical system. FIG.
(A) IR for various viewfinder viewing methods
It is a figure showing the state where the virtual image position by the cornea of ED changed. In the figure, reference numerals 61a to 61e denote virtual images formed by the IRED cornea, and sequentially show the state in which the eye shown in 61a is close to the eyepiece and the state in which the eye is moved away from the eyepiece like 61e. When the distance between the eyepiece 14 and the eye changes, the distance between the IRED and the cornea also changes, so that the sizes of the virtual images 61a to 61e decrease in order from a to e.

FIG. 7B is an enlarged view showing only the vicinity of the image plane of the optical system shown in FIG.
Are images of the virtual images 61a to 61e by the virtual lens 62, and are P images by IRED. Reference numerals 64 to 67 denote marginal rays in this image formation. In order to avoid complicating the drawing, only the light rays forming the P image 63a and the P image 63e at both ends are shown. Also, the subscripts for each image and light beam are associated.

As is clear from the consideration of geometrical optics, the farther the virtual images 61a to 61e are from the virtual lens 62, the closer the images 63a to 63e to the virtual lens 62 and the smaller the size.

By the way, as described above, the output of the area sensor 43a used for gaze detection is the area E shown in FIG.
The A / D conversion is performed in the range of, and the foot portion of the P image is enlarged and used. Therefore, it can be considered that the magnitude of the P image information after the A / D conversion is equal to the magnitude of the spread of the light beam forming the P image. That is, P on the area sensor
Even if the image is blurred, the outermost portion of the light beam forming the blurred image is captured as a P image after A / D conversion.

Therefore, the size of the P image 63a after the A / D conversion is defined by the solid line portions of the light beams 64a, 65a, 66a, and 67a, and the size of the P image 63e after the A / D conversion is the light beam 64e. , 65e, 66e, and 67e. If the light receiving surface of the area sensor 43a is P
Assuming that the image 63e is on the same plane as the image 63e, the defocused image of the P image 63a is output as a point image with the points 68 and 69 on the light receiving surface where the light rays 64a and 65a are incident.

As a result, the size of the output P image differs depending on the position of the eye of the finder observer, and is not suitable for the P image condition described in the description of step (# 202) in FIG.
It is difficult to accurately detect the position of the P image.

However, the closer the eye is to the eyepiece 14, the larger the P image becomes.
The spread of the luminous flux of the image is almost the same on the area sensor, and the spread of the luminous flux is output as the P image regardless of the illuminance distribution of the P image as a result of the A / D conversion. Then, the P image size can be kept constant.

That is, the dependence of the size of the P image on the eyeball position changes depending on the position of the area sensor 43a. If the light receiving surface of the area sensor 43a is set further away from the virtual lens 62, the position of the eyeball of the P image size is Dependence can be reduced. For example, in FIG. 7B, a P image 6 obtained when an eye is placed on an eye point determined by design.
If the light receiving portion of the area sensor 43a is placed slightly to the right in the drawing of 3b, the P images 63b, 63c, 63 in the front focus state
The spread of the light beams on the light receiving portions d and 63e and the spread of the light beam of the P image 63a in the back focus state are substantially the same. In such a case, the P image can always be accurately detected under simple conditions.

From the above properties, if the P image is captured in the front focus state, its size does not change. However, as described above, the positions of the virtual images 61a to 61e in the depth direction are almost the same as the iris. If the front focus state becomes too large, the sharpness of the image of the eye is reduced this time, which hinders pupil detection.
From this, it can be said that it is appropriate to set the position of the area sensor so as to focus on any position between the eye point and the eyepiece 14.

Although the eye-gaze detecting optical system has been described by paying attention to the position at which the area sensor is arranged, optically equivalent is obtained even if the position of the imaging lens 42 is adjusted to obtain a similar image-forming state. It is.

When the user of the eye-gaze detecting device wears glasses, the IRED light reflected by the glasses is reflected by the eyepiece or the beam splitter, reflected again by the glasses, and projected onto the area sensor. However, there is an advantage that the ghost is considerably defocused by focusing on the vicinity of the eyepiece in this manner, so that the ghost is not erroneously detected as a P image.

Further, as an area sensor, for example, C
When a CD is used, a dead zone exists for each pixel. However, according to the above configuration, the P image does not become too small regardless of where the eye is placed. As described above, the problem that the image is not output because it is buried in the dead zone is eliminated.

(Other Embodiments) FIGS. 12 and 13 show a second embodiment of the present invention. FIG. 12 is a characteristic diagram when the characteristic of the amplifier in the visual line detection circuit shown in FIG. 8 is made nonlinear. As shown in the figure, when the input becomes large, the output does not increase so much. FIG. 13 shows the signal of the area sensor 43 after passing through the non-linear amplifier, which is extracted from the horizontal line on which the P image is projected, as in FIG.

Pupil B, irises C1 and C2, white eyes D1 and D2
Are extended by the characteristic 80 in FIG. 12, and higher levels are compressed by the characteristic 81. Accordingly, the two P images shown in FIG.

As a result, as in the first embodiment, the size of the P image can be considered to be the range of the spread of the light beam forming the P image, and as described with reference to FIG. By setting the position of the area sensor so as to focus on any position from the eye point to the eyepiece 14, a P image of a fixed size can be obtained regardless of the eyeball distance of the finder observer.

[0106]

The present invention described above has the effect of uniformizing the line-of-sight detection accuracy even if there is an individual difference in the distance of clear vision of the observer of the observation system.

As an area sensor, for example, a CCD
Is used, a dead zone exists for each pixel. However, according to the above configuration, the P image does not become too small regardless of where the eye is placed, so that the IRED image is not buried in the dead zone and output. Such a defect has disappeared.

Further, since the ghost of the IRED generated by the glasses or the like is further blurred, the ghost is not erroneously detected as the P image.

Alternatively, it was possible to suppress the occurrence of spot-like ghosts generated by the light reflected by the area sensor being reflected again by the exit surface of the beam splitter.

As a result, it is not necessary to slightly incline the surface of the beam splitter, and the line-of-sight detection optical system has an extremely simple structure.

[Brief description of the drawings]

FIG. 1 is a perspective view of an eye-gaze detecting optical system according to an embodiment of the present invention.

FIG. 2A is a longitudinal sectional view of a single-lens reflex camera, and FIG.
1 is a top perspective view of a single-lens reflex camera.

FIG. 3 is a plan view of a stop of the visual line detection optical system shown in FIG. 2;

FIG. 4 is an explanatory diagram showing an area sensor and an image of an eye projected on the area sensor.

FIG. 5 is an explanatory diagram showing an output of a horizontal line on which a first Purkinje image is projected.

FIG. 6 is a development explanatory view of the visual line detection optical system shown in FIG. 1;

7A is a development explanatory view of the visual line detection optical system shown in FIG. 1, and is a diagram for explaining a behavior when a virtual image position of the IRED changes, and FIG. 7B is a diagram of the visual line detection optical system. The figure which expanded the vicinity of an image plane.

FIG. 8 is a block diagram of a camera system including a visual line detection device.

FIG. 9 is a flowchart for explaining a gaze detection operation.

FIG. 10 is a flowchart illustrating reading of image data from a CCD area sensor.

FIG. 11 is a flowchart for explaining a P image detection operation.

FIG. 12 is a characteristic diagram of a nonlinear amplifier in the visual line detection circuit.

FIG. 13 is an explanatory diagram showing an output of a horizontal line on which a first Purkinje image is projected after passing through the amplifier shown in FIG. 12;

[Explanation of symbols]

 Reference Signs List 13 beam splitter 14 eyepiece 40a / 40b illumination light source (infrared light emitting diode) 42 imaging lens 43 area sensor 47 eccentric diaphragm

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-59-146028 (JP, A) JP-A-61-172552 (JP, A) JP-A-63-194237 (JP, A) JP-A-1- 241511 (JP, A) JP-A-1-274736 (JP, A) JP-A-2-5 (JP, A) JP-A-3-87818 (JP, A) JP-A-3-177826 (JP, A) JP-A-3-109029 (JP, A) JP-A-4-138431 (JP, A) JP-A-4-240438 (JP, A) JP-A-2-206425 (JP, A) JP-A-3-109031 (JP, A) JP-A-2-271825 (JP, A) JP-A-3-107909 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G02B 7/28 G03B 13 / 02

Claims (9)

(57) [Claims] 1. A light source for illuminating an observer's eye observed by an observation optical system, a sensor, and an imaging lens unit for forming an image of the observer's eye on the sensor, A line-of-sight detection device that detects a line of sight of an observer based on a reflected image generated from the observer's eyes on the sensor, wherein a corneal reflection image by the light source is included in the sensor output
A / D conversion of analog signal of lower value to detect gaze
A visual axis detection device , wherein the imaging lens unit focuses on a position closer to the observation optical system than an eye point of the observation optical system. 2. The reflection image from the observer's eye is a corneal reflection image and a pupil image of the light source.
Gaze detection device. 3. A light source for illuminating an observer's eye observed by an observation optical system, a sensor, and an imaging lens unit for forming an image of the observer's eye on the sensor. A line-of-sight detection device that detects a line of sight of an observer based on a reflected image generated on the sensor from the observer's eyes, wherein the imaging lens unit is recognized from an output of the sensor. A line-of-sight detection device, wherein a size of a corneal reflection image is configured not to substantially change depending on a distance of the observer's eye from the observation optical system. 4. A light source for illuminating an observer's eye observed by a finder optical system, an area sensor, and an imaging lens unit for forming an image of the observer's eye on the area sensor. Inputting the output of the area sensor to a non-linear amplifier, and using the output of the non-linear amplifier to change the line of sight of the observer based on the relative relationship between the corneal reflection image and the pupil image of the light source generated on the area sensor. A gaze detection device for detecting, wherein the imaging lens unit focuses on a position closer to the finder optical system than an eye point of the finder optical system. 5. A light source for illuminating an observer's eye observed by an observation optical system; a sensor; and a light source arranged in an optical path of the observation optical system for guiding a light beam from the observer's eye to the sensor. A beam splitter, and an imaging lens unit that is disposed adjacent to the beam splitter and forms an image of the observer's eye on the sensor with a light beam from the observer's eye from the beam splitter. A gaze detection device that detects a gaze of the observer based on a reflection image generated on the sensor from the eye of the observer, wherein the imaging lens unit has an optical axis with the lens. A line-of-sight detection device comprising a stop provided with a different eccentric aperture. 6. A light source for illuminating an observer's eye observed by an observation optical system, a sensor, and a light source arranged in an optical path of the observation optical system for guiding a light beam from the observer's eye to the sensor. A beam splitter, and an image forming lens unit for forming an image of the observer's eye on the sensor with a light beam from the observer's eye from the beam splitter. A line-of-sight detection device that detects a line of sight of an observer based on a reflected image from the eyes of the observer, wherein the imaging lens unit is configured to generate reflected light generated from the sensor by a corneal reflection image of the light source; A line-of-sight detection device comprising a stop for blocking incidence on a splitter. 7. A light source for illuminating an observer's eye observed by an observation optical system, a sensor, and a sensor disposed in an optical path of the observation optical system for guiding a light beam from the observer's eye to the sensor. A beam splitter, and an image forming lens unit for forming an image of the observer's eye on the sensor with a light beam from the observer's eye from the beam splitter. A line-of-sight detection device that detects a line of sight of the observer based on a reflection image from the eyes of the observer, wherein the imaging lens unit is displaced from a formation position of a corneal reflection image of the light source on the sensor. A line-of-sight detection device comprising a stop having an optical axis passing through a position. 8. The light source is configured to illuminate the observer's eye from below the optical axis of the observation optical system, and the optical axis of the stop is deviated downward from the optical axis of the imaging lens unit. visual axis detecting device according to claim 5 or 6 or 7, characterized in that it is the core. 9. A camera having a control circuit for controlling the apparatus for photographing using the output of the visual line detecting device and the visual axis detecting device according to any one of claims 1 to 8.