JP2002186587A - Visual line detector - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately correct the position of a visual line on the basis of proper calibration data. SOLUTION: This visual line detector, which has a plurality of image forming optical systems for forming a plurality of eyeball images to detect the position of the visual line of an observer through a finder optical system, has first computing means (#305-#312 or #314-#320, #322) for detecting the position of the visual line of the observer from the eyeball image synthesized on the basis of a plurality of the eyeball images, second computing means (#323-#329) for detecting the position of the visual line of the observer from one eyeball image among a plurality of the eyeball images, calibration means (#301-#304, #312, #329, #322) performing operation for obtaining calibration data for correcting the position of the visual line of the observer using the first or second arithmetic means, and a memory means for storing the calibration data obtained by the calibration means.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an improvement in an eye-gaze detecting device mounted on a camera or the like.

[0002]

2. Description of the Related Art Conventionally, the gaze direction of a photographer (observer) is detected, and which area (position) in the viewfinder field the photographer is observing, that is, the so-called gaze direction of the photographer is partly determined by the camera. Various cameras have been proposed in which various types of photographing functions, such as automatic focus adjustment and automatic exposure, are controlled based on a signal detected by a visual axis detection device provided in the above.

[0003] For example, the present applicant has disclosed in
Patent Document 1 discloses a gaze detection device for detecting a gaze direction of a photographer, a focus detection device, and an automatic exposure control device having a plurality of photometric sensitivity distributions, based on an output signal from the gaze detection device. A camera has been proposed which controls the driving of a focus detection device and an automatic exposure control device.

The eye-gaze detecting device used here irradiates infrared light to the eye of the observer, forms an eyeball image from the returned reflected light, and performs eye-gaze detection based on the eyeball image data. is there.

Usually, in these eyeball images, in addition to the eyelids and the pupils, Purkinje images, which are reflection images of the light source on the cornea, occur, and the positional relationship between the pupils and the Purkinje images changes according to the rotation of the eyeballs. Therefore, by extracting information on the pupil and the Purkinje image from the eyeball image, it is possible to detect the gazing point.

[0006] Further, there is known a technique of using a plurality of eyeball images having parallax in order to further improve the line-of-sight detection accuracy.

[0007]

When the observer performs the gaze detection in the above-described gaze detection device, the observer performs a so-called calibration to correct a deviation between the optical axis of the observer's eyeball and the visual axis. The line-of-sight correction data calculated along with the data collection is stored in the camera, and is used in the calculation for determining the line-of-sight position of the observer when the observer detects the line of sight. The calibration in the line of sight detection is disclosed in Japanese Patent Application Laid-Open No. 4-236935 and the like.

A gaze detecting apparatus for detecting a gaze of a photographer based on a plurality of eyeball images having parallax obtained by a plurality of imaging optical systems according to the present invention will be described below. In the same manner, it is necessary to execute calibration, and detect the line of sight of the observer using the calibration data obtained therefrom.

Here, if the plurality of imaging optical systems are arranged symmetrically with respect to the optical axis of the finder system of the line-of-sight detection device, and if the observer's eyeball is located on the optical axis of the finder, the plurality of imaging optical systems are connected to the plurality of imaging optical systems. A plurality of eyeball images having parallax obtained by the above can be obtained as symmetric eyeball images (for example, a left-right symmetric image if two optical systems are disposed symmetrically with respect to the viewfinder optical axis), It is possible to perform exactly the same calibration as in the past and perform gaze detection with one piece of calibration data obtained therefrom. However, in practice, the observer's eyeball is rarely always on the correct viewfinder optical axis, and when the eyeball is located at a position off the viewfinder optical axis, light is received by the plurality of imaging optical systems. The plurality of eyeball images formed on the element are formed at different positions on the light receiving element. On the other hand, an optical system for obtaining an eyeball image has optical distortion such as distortion.

FIG. 12 shows two image-forming light receiving elements (CCD1 and CCD2) in which a grid-like chart provided at a specific distance from the finder optical system is symmetrically arranged at an angle from the finder optical axis. It is a diagram showing a state where the image is formed on the upper side, and it can be seen that the images of the light receiving elements CCD1 and CCD2 are non-uniform inverted right and left images. In other words, a plurality of eyeball images that exist on the optical axis of the finder and are formed at different positions on the light receiving element are affected by the distortion according to the image formation position, and thus are information necessary for gaze detection. The accuracy of the Purkinje image (corneal reflection image) of a certain eyeball and the center position of the pupil and the relative relationship between the two differ in each of the plurality of imaging optical systems, and it is necessary to perform eye gaze detection accurately with only one calibration data. There was a problem that it was difficult.

(Object of the Invention) A first object of the present invention is to provide an eye-gaze detecting device capable of accurately correcting an eye-gaze position with appropriate calibration data.

A second object of the present invention is to provide a visual axis detection device capable of performing an accurate visual axis detection.

[0013]

In order to achieve the first object, according to the first aspect of the present invention, a plurality of objects having a parallax with respect to the optical axis of a finder optical system for observing an object are provided. In a line-of-sight detection device that has a plurality of imaging optical systems that form an eyeball image and detects a line-of-sight position of the observer through the finder optical system, the viewer's eyeball image is synthesized from the eyeball images synthesized based on the plurality of eyeball images. First calculating means for detecting a gaze position, second calculating means for detecting a gaze position of an observer from one of the plurality of eyeball images, and the first calculating means or the second calculating means. Calibration means for performing an operation of obtaining calibration data for correcting the observer's line of sight position using the arithmetic means, and storage means for storing calibration data obtained by the calibration means. It is an eye-gaze detecting device having.

Further, in order to achieve the second object,
The invention according to claim 2 is configured to detect the line of sight of the observer,
The gaze according to claim 1, wherein the gaze position detected by the first calculation unit or the second calculation unit is corrected based on the calibration data stored in the storage unit, and the gaze position of the observer is detected. It is a detection device.

[0015]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail based on illustrated embodiments.

FIG. 1 is a structural view of a principal part of a single-lens reflex camera according to an embodiment of the present invention, FIG. 2 is a diagram schematically showing a line-of-sight detecting device of the single-lens reflex camera of FIG. 1, and FIG. FIG. 2 is a diagram showing the inside of a finder of one single-lens reflex camera.

In these figures, reference numeral 1 denotes a photographing lens, which is shown by two lenses 1a and 1b for convenience, but is actually constituted by a larger number of lenses. Reference numeral 2 denotes a main mirror which is inclined to the photographing optical path or retreated depending on the state of observation of the subject image by the finder system and the state of photography of the subject image. Reference numeral 3 denotes a sub-mirror which converts a light beam transmitted through the main mirror 2 into a focus detection device 6 described below below the camera body.
Reflects toward. Reference numeral 4 denotes a shutter. Reference numeral 5 denotes a photosensitive member, which is formed of a silver halide film, a solid-state image pickup device such as a CCD or MOS type, or an image pickup tube such as a vidicon.

Reference numeral 6 denotes a focus detection device, which includes a field lens 6a, reflection mirrors 6b and 6 arranged near the image plane.
c, a secondary imaging lens 6e, an aperture 6d, a line sensor 6f including a plurality of CCDs, and the like. The focus detection device 6 according to the present embodiment performs focus detection by a well-known phase difference method, and as illustrated in FIG. 3, a plurality of regions in the object scene (focus detection region marks 70 to 74). (Five locations shown) are used as focus detection areas so that focus detection can be performed in each of the focus detection areas. Reference numeral 7 denotes a focusing plate arranged on a predetermined image forming plane of the photographing lens 1, and 8 denotes a pentaprism for changing a finder optical path. Reference numerals 9 and 10 denote an imaging lens and a photometric sensor for measuring the luminance of the subject in the observation screen, respectively. The imaging lens 9 conjugates the focusing plate 7 and the photometric sensor 10 via a reflection optical path in the pentaprism 8. Be related. Reference numeral 11 denotes an eyepiece disposed behind the exit of the pentaprism 8, and is used for observing the focus plate 7 with the eyeball 15 of the photographer.

A finder optical system is constituted by the main mirror 2, the focusing plate 7, the pentaprism 8, and the eyepiece 11.

Reference numeral 23 denotes a pentaprism 8 and an eyepiece 11
A dichroic mirror, which is a light splitter that transmits visible light and reflects infrared light, for example, and is an infrared reflected light of a photographer's eye 15 illuminated by an infrared light emitting diode described later. Is guided to an image forming optical system for line of sight detection.

Reference numerals 12a and 12b denote imaging lenses, respectively.
14a and 14b are vertical columns of photoelectric conversion elements such as CCDs, respectively.
Image sensors (hereinafter referred to as CCD-EYE14a, CCD-
EYE14b), and FIG. 2 shows a perspective view of their configuration.

As can be seen from FIG. 2, the imaging lens 12
a and 12b are arranged at a predetermined angle with respect to the finder optical axis on the horizontal of the finder optical axis, respectively.
Further, the CCD-EYEs 14a and 14 are arranged so as to have a conjugate imaging relationship with the vicinity of the pupil of the eyeball 15 of the photographer located immediately before the eyepiece corresponding to the imaging lenses 12a and 12b, respectively.
The light receiving means for detecting the line of sight is constituted by these.

As described above, the imaging lens 12a and the CCD-
In the EYE 14a, the first imaging optical system
b and the CCD-EYE 14b constitute a second imaging optical system, respectively, and it is possible to obtain a pair of eyeball images having parallax of a photographer obtained by the two imaging optical systems. is there.

Reference numerals 13a to 13h denote light-emitting elements which constitute illumination means for illuminating the vicinity of the pupil of the eyeball 15 of the photographer. Infrared light-emitting diodes (hereinafter referred to as IREDs) are used for these light-emitting elements. And eight IREDs 13a to 13h are arranged around the eyepiece 11.
A set of two of them emits light upon one gaze detection depending on the situation.

As described above, the line of sight detecting device is constituted by the light receiving means, the illuminating means, and the above-mentioned dichroic mirror 23.

Returning to FIG. 1, reference numeral 21 denotes a high-brightness superimposing LED which can be visually recognized even in a bright subject. Light emitted from the superimposing LED is reflected by the main mirror 2 via a light projecting prism 22. It is bent in the vertical direction by the micro prism array 7a provided on the display unit of the focus plate 7,
The light reaches the photographer's eyeball 15 through the pentaprism 8 and the eyepiece 11. That is, as can be seen from the finder field of view shown in FIG.
Reference numeral 4 denotes a light which shines in the finder visual field to display a focus detection area (this is referred to as a superimposed display).

In FIG. 3, dot marks 70 'and 74' are engraved inside the left and right focus detection area marks 70 and 74, respectively, for correcting a line-of-sight detection error due to individual differences in eyes. FIG. 9 shows a target at the time of calibration for collecting line-of-sight correction data.

Here, 51 is a shutter speed display, 52 is an aperture value display segment, 50 is a line-of-sight input mark indicating that the line of sight is being input, and 53 is a focus mark that indicates the state of focus of the photographic lens 1. . Reference numeral 24 denotes a finder LCD (hereinafter, referred to as an LCD) for displaying photographing information outside the finder field of view.
F-LCD), and is illuminated by the illumination LED 25.

The light transmitted through the F-LCD 24 is guided by the triangular prism 26 to the outside of the finder visual field as shown at 24 in FIG. 3, so that the photographer can know various photographing information.

Referring back to FIG. 1, reference numeral 31 denotes an aperture provided in the taking lens 1, reference numeral 32 denotes an aperture driving device including an aperture driving circuit 111 described later, reference numeral 33 denotes a lens driving motor, and reference numeral 34 denotes a driving gear and the like. This is a lens driving member. Reference numeral 35 denotes a photocoupler which detects rotation of a pulse plate 36 interlocked with the lens driving member 34 and
The focus adjustment circuit 110 drives the lens driving motor 33 by a predetermined amount based on this information and information on the amount of lens driving from the camera side, and moves the photographing lens 1 to a focusing position. It has become. Reference numeral 37 denotes a mount contact which serves as an interface between a known camera and a lens. Reference numeral 27 denotes a posture detection switch such as a mercury switch, which detects whether the camera is held in a horizontal position or a vertical position.

FIG. 4 is a block diagram showing an electrical configuration incorporated in the single-lens reflex camera having the above-described configuration. The same components as those in FIG. 1 are denoted by the same reference numerals.

A central processing unit (hereinafter, referred to as a CPU) 100 of a microcomputer built in the camera body includes a line-of-sight detection circuit 101, a photometry circuit 102, an automatic focus detection circuit 103, a signal input circuit 104, an LCD drive circuit 1
05, LED drive circuit 106, IRED drive circuit 10
7, a shutter control circuit 108 and a motor control circuit 109 are connected. Further, signals are transmitted to the focus adjustment circuit 110 and the aperture driving circuit 111 disposed in the taking lens 1 via the mount contact 37 shown in FIG.

EEPROM 10 attached to CPU 100
Reference numeral 0a has a storage function of a line of sight correction data for correcting an individual difference of the line of sight as storage means.

The line-of-sight detection circuit 101 A / D-converts the eyeball image signals from the CCD-EYEs 14a and 14b, which are image sensors, and converts the two image information into the CPU 100.
Send to Then, the CPU 100 extracts each feature point of the eyeball image necessary for gaze detection according to a predetermined algorithm, and further calculates the gaze of the photographer from the position of each feature point. The photometric circuit 102 amplifies the signal from the photometric sensor 10, performs logarithmic compression and A / D conversion, and transmits the result to the CPU 100 as luminance information of each sensor. The photometric sensor 10
Is an SPC that measures light in six areas in the viewfinder screen.
0, SPC-1, SPC-2, SPC-3, SPC-
It is composed of six photodiodes 4 and 4, SPC-5, and is capable of so-called split photometry.

The line sensor 6f includes five sets of line sensors CCD-0, CCD-1, CCD-C corresponding to the five focus detection areas 70 to 74 in the screen shown in FIG.
2. A well-known CC composed of CCD-3 and CCD-4
It is a D line sensor. The automatic focus detection circuit 103
A / D converts the voltage obtained from the line sensor 6f and sends it to the CPU 100.

SW1 is a switch for turning on the first stroke of the release button of the camera to start photometry, AF, line of sight detection, and the like, and SW2 is a release switch for turning on the second stroke of the release button. SW-DIAL1
And SW-DIAL2 are dial switches provided in an electronic dial (not shown) provided in the camera overview section, and rotate to generate a click pulse. The pulse signal generated here is input to an up / down counter of the signal input circuit 104, and the rotation click amount of the electronic dial is counted. SW-HV1 and SW-HV2 are posture detection switches corresponding to the posture detection switch 27,
The posture state of the camera is detected based on this signal.

The state signals of these switches are input to the signal input circuit 104 and transmitted to the CPU 100 via the data bus.

The LCD drive circuit 105 has a well-known structure for driving and driving an LCD, which is a liquid crystal display element.
At the time of shutter release, the display of the set photographing mode and the like can be simultaneously displayed on both the monitor LCD 42 (not shown) and the finder LCD (F-LCD) 24 in the camera exterior part.

The LED driving circuit 106 includes an illumination LE.
D (F-LED) 25 and Superimpose LED2
1 is turned on and off. The IRED drive circuit 107
Correspond to the IREDs 13a to 13
h is selectively turned on or the IREDs 13a to 13h
The illumination power is controlled by changing the output current value (or the number of pulses). The shutter control circuit 108 controls the magnet MG-1 for running the front curtain and the magnet MG-2 for running the rear curtain when energized, and exposes the photosensitive member with a predetermined amount of light. The motor control circuit 109 includes:
This is for controlling a motor M1 for winding and rewinding the film and a motor M2 for charging the main mirror 2 and the shutter 4.

The shutter control circuit 108 and the motor control circuit 109 execute a series of camera release sequence operations.

Next, the operation of the main part of the single-lens reflex camera having the above-mentioned eye-gaze detecting device will be described with reference to the flowchart of FIG.

A mode dial (not shown) provided on the exterior of the camera is used to select a photographing mode and the like, and includes a program AE, a shutter priority AE, and an aperture priority A.
Each shooting mode of E and manual exposure can be set.
A "CAL" position for line-of-sight input is also in the mode dial, and by turning the electronic dial to the "CAL" position, the line-of-sight input is turned on and off.
f, and execution and selection of calibration can be performed. When the mode dial (not shown) is rotated to set the camera from a non-operation state to a predetermined shooting mode (this embodiment will be described based on a case where the shutter priority AE is set), the power of the camera is turned off. ON (#
100), variables used for gaze detection other than the gaze calibration data stored in the EEPROM 100a of the CPU 100 are reset (# 101). Then, when the release button is depressed, the switch SW1 is pressed.
Is turned on (# 102).

When the release button is depressed, the switch SW1 is pressed.
Is turned on, the signal input circuit 104 detects
The CPU 100 checks with the eye-gaze detection circuit 101 which calibration data to use when performing eye-gaze detection (# 103). At this time, if the calibration data of the confirmed calibration data number is unchanged from the initial value or is set to the line-of-sight prohibition mode (yes in # 103), the line-of-sight detection is not executed. That is, a specific focus detection area is selected by the automatic focus detection area selection subroutine without using the visual line information (# 115). Then, in this focus detection area, the automatic focus detection circuit 103 performs a focus detection operation (# 107).

There are several possible algorithms for automatic selection of the focus detection area. For a multi-point AF camera, a near-point priority algorithm that is known and weighted to the central focus detection area is effective.

When it is recognized that the line-of-sight calibration data corresponding to the calibration data number has been set to a predetermined value and that the data has been input by the photographer (n in # 103).
o), the line-of-sight detection circuit 101 performs line-of-sight detection according to the calibration data, and
The coordinates are converted to the coordinates of the upper gazing point (# 104). The details of this gaze detection will be described later with reference to FIG. Then, it is determined whether or not the detection of the line-of-sight information is successful (# 105). The determination conditions here include the Purkinje image, which is a corneal reflection image, the reliability of the pupil center position, the rotation angle of the eyeball, and the like. As a result, if unsuccessful, the process proceeds to a "focus detection area automatic selection subroutine" of step # 115. If the gaze detection is successful, the CPU 100 selects a focus detection area close to the gazing point coordinates (# 106). As a result, the automatic focus detection circuit 103 performs focus detection in the selected focus detection area using the detected line-of-sight information (# 107).

Next, it is determined whether the focus detection result in the selected focus detection area indicates that the focus cannot be detected (NG) (# 108), and if not, the CPU 100 sends a signal to the LCD drive circuit 105. The focus mark 53 on the LCD 24 in the finder is blinked to warn the photographer that the focus detection is NG (# 117), and this warning is continued until the switch SW1 is released (# 118 → # 117 → # 11).
8 ...).

On the other hand, if focus detection is possible (No in # 118) and the focus adjustment state of the focus detection area selected by a predetermined algorithm is not in focus (No in # 109),
The CPU 100 sends a signal to the lens focus adjustment circuit 110 to drive the photographing lens 1 by a predetermined amount (# 116). After driving the lens, the automatic focus detection circuit 103 performs focus detection again, and determines whether or not the photographing lens 1 is in focus (# 101 → # 108 → # 109).

In a predetermined focus detection area, the photographing lens 1
Is focused (no in # 109), the CPU 10
0 sends a signal to the LCD drive circuit 105 to turn on the focus mark 53 of the LCD 24 in the finder,
A signal is also sent to the ED drive circuit 106 to turn on the superimpose LED 21 corresponding to the focused focus detection area, and the focus detection area is illuminated to display focus (# 110).

The photographer observes that the focused focus detection area is displayed in the viewfinder, determines that the focus detection area is incorrect, or decides to stop photographing, releases his hand from the release button, and switches SW1. When it is turned off (# 11
1) Subsequently, the camera returns to step # 102 to wait until the switch SW1 is turned on.

If the photographer looks at the focus detection area where the focus is displayed and continues to turn on the switch SW1, the CPU 100 transmits a signal to the photometric circuit 102 to perform photometry (# 112). . At this time, an exposure value is calculated by weighting the photometry area including the focused focus detection area. In the case of the present embodiment, an aperture value (F5.6 shown at 52 in FIG. 3) is calculated using a segment and a decimal point as a photometric calculation result.
) Is displayed.

Further, it is determined whether the release button is depressed and the switch SW2 is turned on (# 11).
3) If the switch SW2 is off, the state of the switch SW1 is confirmed again (# 111).

When the switch SW2 is turned on, the CPU 100 transmits a signal to each of the shutter control circuit 108, the motor control circuit 109, and the aperture driving circuit 111 to perform a known shutter release operation (# 11).
4).

Specifically, first, the motor control circuit 109
Then, the motor M2 is energized to raise the main mirror 2 and the aperture 31 is stopped down. Then, the magnet MG-1 is energized to open the front curtain of the shutter 4. The aperture value of the aperture 31 and the shutter speed of the shutter 4 are determined by the photometric circuit 1
02 is determined from the exposure value detected and the sensitivity of the film 5. After the lapse of a predetermined shutter time (1/125 seconds),
The magnet MG-2 is energized, and the rear curtain of the shutter 4 is closed. When the exposure of the film 5 is completed, the motor M2 is energized again to perform mirror down and shutter charging, and is also energized to the film feeding motor M1 to feed the film frame, thereby performing a series of shutter release sequence operations. To end.

Thereafter, the camera again sets the switch SW1 to o
n (# 102).

FIG. 6 is a flowchart showing the line-of-sight detection operation executed in step # 104.

In FIG. 6, as described above, the line-of-sight detection circuit 101 executes line-of-sight detection upon receiving a signal from the CPU 100 (# 104). First, the gaze detection circuit 101 determines whether the gaze is detected in the photographing mode or the gaze is detected in the gaze calibration mode (# 200). Actually, when the mode dial is set to the calibration mode for the line-of-sight detection operation, a later-described calibration (CAL) operation (# 300) is executed.

The mode dial has a line-of-sight detection mode setting. At this set position, calibration data numbers 1, 2, and 3 for registering and executing calibration data for three persons, and performing line-of-sight detection Not of
f, a total of four positions can be set by operating the electronic dial.

When the camera is not set to the calibration mode, the line-of-sight detection circuit 101 recognizes which calibration data number the camera is currently set to.

Subsequently, in the case of gaze detection in the photographing mode, the gaze detection circuit 101 first detects the attitude of the camera via the signal input circuit 104 (# 201). The signal input circuit 104 processes the output signal of the attitude detection switch 27 (SW-ANG) in FIG. 1 to determine whether the camera is in the horizontal position or the vertical position. If the camera is in the vertical position, for example, the release button 41 is pressed. It is determined whether it is in the top direction or the ground (plane) direction.

Next, the IR information is obtained from the camera attitude information and the photographer's eyeglass information included in the calibration data.
The EDs 13a to 13h are selected (# 202). That is, if the camera is held in the horizontal position and the photographer does not wear glasses, the IREDs 13a and 13b shown in FIG.
Select Also, if the camera is in the horizontal position and the photographer wears glasses, the IREDs 13c and 13d having a wider interval than the intervals between the IREDs 13a and 13b are selected so as to reduce the influence of the spectacle reflected light of the photographer. I do. Also, if the camera is held, for example, with the release button (right side of the photographer) in the top direction or the vertical position in the ground direction, a combination of IREDs that illuminates the photographer's eyeball from below, that is, photographing I did not wear glasses
The REDs 13a and 13e are selected, and the combination of the IREDs 13c and 13g is selected if the photographer wears glasses.

Next, CCD-EYEs 14a and 14b which are two image sensors for obtaining an eyeball image of a photographer.
The storage time of the CCD (illustrated as CCD1 and CCD2 in the flow of FIG. 6) and the illumination power of the IRED are determined by the CCD-EYE14.
The setting is made based on the image signal output of the preliminary accumulation performed prior to the actual accumulation of a and 14b, information on whether or not glasses are worn (# 203). The CCD-EYE pre-accumulation means that an image signal is always taken in after a fixed time, for example, 1 ms, is determined immediately before the main accumulation, and the accumulation time of the actual eyeball image taking is determined according to the level of the signal level. By controlling, a stable eyeball image signal can be obtained.

As the actual preliminary storage control, a CCD-EY
Since two image signals E14a and E14b are obtained, the main accumulation is controlled using the average signal value of the two.

The storage time and IRE of the CCD-EYE 14
When the lighting power of D is set, the CPU 100
The IRED 13 is turned on with a predetermined power via the drive circuit 107, and the line-of-sight detection circuit 101 starts accumulation of the two image sensors CCD-EYEs 14a and 14b (# 2).
04, # 207). Also, the CCD-EY set earlier
The accumulation here ends according to the accumulation time of E14,
At the same time, the IRED 13 is turned off.

Next, the image signals accumulated in the CCD-EYEs 14a and 14b are sequentially read out,
A / D-converted and stored in the CPU 100 (# 2
05, # 208).

Here, the principle of gaze detection will be briefly described with reference to FIGS.

FIG. 10 shows an image of an eyeball image signal of the CCD-EYE 14. The IREDs 13a and 13b emit light at the cornea 16 of the eyeball 15 in FIG. 11, and the Purkinje images 19a and 19b shown in FIG. Occurs. In addition, 17 is an iris and 18 is a pupil.

A known line-of-sight detection process is performed on these image signals. That is, in the CPU 100, one set of the IREDs 13
a, a Purkinje image (P image) 19a, which is a virtual image of 13b,
The position of 19b is detected. As described above, since the Purkinje images 19a and 19b appear as bright spots with high light intensity,
A predetermined threshold value for the light intensity is provided, and a light intensity exceeding the threshold value can be detected as a Purkinje image. The center position of the pupil is the pupil 18 and the iris 1
7 are detected by performing a least-squares approximation of a circle based on each of the boundary points. The rotation angle θ of the eyeball can be obtained from the Purkinje image position and the pupil center position, and the distance between the eyepiece 11 of the camera and the photographer's eyeball 17 is calculated from the interval between the two Purkinje images 19. The imaging magnification β of the eyeball image projected on the EYE 14 can be obtained.

From the above, the position coordinates on the focus plate 7 in the line of sight of the photographer are obtained using the rotation angle θ of the eyeball, the imaging magnification β, and the individual difference correction information obtained by the calibration. Can be.

Returning to FIG. 6, the CPU 100 calculates the above-described eyeball rotation angle based on the stored eyeball image signals of the CCD-EYEs 14a and 14b (# 206, #).
209). Next, it is determined whether the Purkinje image has been calculated from the respective image signals obtained by the CCD-EYEs 14a and 14b (# 210). Here, CCD-
If the Purkinje image is detected only from the image signal of the EYE 14a, the Purkinje image position and the pupil center position are first obtained according to the above-described eye gaze detection process, and the rotation angle of the eyeball is calculated. The direction is converted into coordinates on the focus plate (# 212). In addition, CCD-E
Similarly, when the Purkinje image is detected only from the YE14b image signal, the above-described line-of-sight detection processing is performed.

In the above two examples, in one of the eyeball images of the two image sensors, the reflected light of the photographer's spectacles or the reflected light from the sun, or the Purkinje image or This is an example in which the detection of the pupil center position has failed. At this time, it is possible to perform the gaze detection by using only the other image sensor image that can detect the gaze.

On the other hand, if the Purkinje image can be detected from both the CCD-EYEs 14a and 14b, the two image signals of the CCD-EYEs 14a and 14b are combined to form one eyeball image ( # 211). The algorithm for image synthesis is described in Japanese Patent Application
No. 071981, which is described in detail, basically, the above-mentioned CCD-E of two Purkinje images each obtained from two eyeball images on the CCD-EYEs 14a and 14b.
Pixel coordinates on YE are detected, and when they are matched with each other, the signal level in the pixels other than the Purkinje image is calculated as follows:
Uses the lower signal value of the two images (because the higher signal value can be estimated as the part where the eyelashes and ghosts are superimposed)
By doing so, one eyeball image can be created.
Further, in the case of a configuration in which an eyeball image is formed by an optical system having parallax, since the image becomes non-uniform in a horizontally inverted image due to optical distortion as shown in FIG. As for (2), only the image closer to the finder optical axis is used (in the example of FIG. 12, the image on the left side of the CCD2 image is used for the non-uniform area on the left side of the image, , The image on the right side of the CCD1 image is adopted), so that it is possible to combine images in consideration of the optical distortion.

Here, the synthesis of the eyeball image will be described in more detail with reference to the example of FIG.

The image signals obtained from the CCD-EYEs 14a and 14b are the CCD1 image and the CCD2 image in FIG. 8, respectively. These eyeball images are obtained by the eyelashes of the photographer looking through the viewfinder of the camera being illuminated by sunlight. And pupil edge detection is inaccurate. On the other hand, in the image synthesized by the above method, it can be seen that the signal level of the shining eyelashes is suppressed and the pupil edge is easily detected. This is because the pupil and the Purkinje image exist at approximately the same distance in the coordinates on the viewfinder optical axis, whereas the eyelashes exist at the level of several millimeters near the eyepiece lens.
Parallax occurs between the image and the image, and the reflected light of the eyelashes is suppressed by performing the synthesis. Similarly, the effect of reducing the influence can be expected for eyeglass ghosts in which the IRED 13 that illuminates the eyeball of the photographer reflects on the surface of the glasses worn by the photographer.

Returning to FIG. 6, the Purkinje image and the center of the pupil are detected from the synthesized eyeball image, the rotation angle of the eyeball is calculated, and the direction of the line of sight of the photographer is converted into coordinates on the focusing screen. (# 212).

When the line of sight of the photographer has been detected as described above, step # 10 of the camera operation flowchart of FIG.
The process returns to # 6 (# 213).

Next, the calibration (CAL) operation will be described with reference to the flowchart of FIG.

As described above, calibration is
The photographer performs fixation on the right end 74 'and the left end 70' of the dot mark in the focus detection area in the viewfinder visual field, respectively, for a certain period of time, and collects eye-gaze correction data from eyeball image data obtained therefrom. By setting the mode dial (not shown) to the “CAL” position,
The calibration operation starts (# 300).

First, the output signal of the attitude detection switch 27 (SW-ANG) is detected via the signal input circuit 104 to determine the attitude of the camera (# 30).
1). This corresponds to step # 201 in the gaze detection flow in FIG.
This is the same detection processing as. Next, the right end 74 'of the dot mark in the focus detection area in the finder visual field is blinked to display a target to be fixed to the photographer (# 302). At the same time, the calibration data stored in the CPU 100 is checked from the currently set calibration number. If the calibration data has already been registered, the “CAL” display on the monitor LCD 42 of the camera external part is lit as it is, If not registered, the “CAL” display flashes.

Next, an IRED selecting operation for illuminating the photographer's eyeball at the time of performing the calibration is performed (# 30).
3). The operation of selecting the IRED in this case is slightly different from the operation described in FIG. 6, and is similar to using the posture information of the camera, but the photographer performs calibration in the past, and the calibration data at that time is already If stored in the camera, the stored set of IREDs
In other words, one of the pair for wearing glasses or for not wearing (with naked eyes) is selected from the beginning according to the stored information. On the other hand, when the calibration is performed for the first time, since there is no information for selecting the pair of the IREDs for wearing or not wearing the glasses, the first eye image illumination of the calibration is performed by the IRED for the glasses not wearing. The set is selectively illuminated, and according to the determination of the presence or absence of ghost due to the reflection of the glasses in the eyeball image signal of the CCD described later, if the occurrence of the ghost of the glasses is detected, the illumination after the second time is changed to the IRED set for the glasses. Things.

Then, the set of IREDs to emit light is determined (# 203), and the camera waits for the photographer's SW1 ON signal, and when the camera detects the SW1-ON signal, the CCD-C as described in the flowchart of FIG. Steps # 204 to # 206 in the EYE 14a, C
The same eyeball image capturing operation as in steps # 207 to # 209 in the CD-EYE 14b is executed.

That is, the storage time of the CCD-EYE 14a and the illumination power of the IRED 13 are set, and
The accumulation of the EYE 14a and the turning on of the IRED 13 (# 30
5), the image signals accumulated in the CCD-EYE 14a are sequentially read out, and are stored in the CPU 100 after A / D conversion (# 306).

Next, the CPU 100 detects a pair of Purkinje images from the A / D-converted image signals on the memory, and determines whether or not a Purkinje image has been detected (#).
307).

The same process is performed in the CCD-EYE 14b in steps # 314 to # 316.

Next, when the Purkinje image is detected by the CCD-EYE 14a, the Purkinje image and the center of the pupil are calculated, and the rotation angle θ of the eyeball is calculated (# 30).
8). When the rotation angle θ is calculated, it is determined whether or not the value is appropriate (# 309). Since the deviation between the optical axis of the eyeball and the visual axis hardly deviates by several tens of degrees biologically, the threshold value for determination is set to ± 10 degrees here. In step # 309, the detected rotation angles OK and NG of the eyeball are detected.
Is satisfied, the process proceeds to the next step # 310 whether the result is OK or NG. If the total number of rotation angle detection of the eyeball is less than 10, the process returns to step # 304.
The eyeball image capturing operation is performed again, and when the total number of rotation angle detections of the eyeball reaches ten times, this time, out of the ten times, O
Calibration (CA
L) Success or failure is determined (# 311).

In this embodiment, the CAL succeeds at the right end 74 'of the focus detection area mark (no in # 311) when the rotation angle has been detected more than six times.
'Starts the calibration operation. If the calibration at the left end 70 is also successful, the monitor L
The “CAL” display on the CD 42 is displayed, and the calibration data is stored in the EEPROM 100a of the CPU 100 as CAL-A1, which is calibration data at the lateral position of the camera of the CCD-EYE 14a. If the calibration data has already been registered, the newly collected calibration data is integrated with the past data stored in the memory.

If the number of successful rotation angle detections is less than 6, the CAL fails (yes in # 312), and the "CAL" display on the monitor LCD 42 flashes.
The photographer is notified that the calibration has failed (# 313).

In this flowchart, the distinction between the calibration operation at the right end 74 'of the focus detection area mark and the calibration operation at the left end 70' is omitted.

On the other hand, the same applies to the CCD-EYE 14b. When the Purkinje image is detected in step # 316, the CAL-EYE 14b is calibrated data at the camera horizontal position of the CCD-EYE 14b through steps # 317 to # 320. CPU 10 as A2
Store to 0.

Here, the description returns to step # 307. If the Purkinje image can be detected in the image signal processing of the CCD-EYE 14a, the eyeball rotation angle is calculated in step # 308 as described above. CCD-
At step # 316 in the image signal processing of the EYE 14b, it is determined whether or not the Purkinje image detection is possible at the same timing (# 323). Here, CCD-EY
When the Purkinje image can be detected from both the image signal of E14a and the image signal of CCD-EYE14b, FIG.
As described with reference to the flowchart of (1), each image signal is combined to create one eyeball image (# 324).

Hereinafter, the CCD-E is obtained from the obtained synthesized eyeball image.
In the same process as in the case of the YEs 14a and 14b, the Purkinje image and the pupil center are calculated, and the rotation angle θ of the eyeball is calculated (# 325). Further, when the rotation angle θ is calculated, it is determined whether the value is appropriate (# 326), and the number of detections (# 32)
7), the number of successful detections (# 328) is checked, and finally the eyeball image synthesized from the CCD-EYEs 14a and 14b is stored in the CPU 100 as CAL-A0, which is calibration data at the camera lateral position ( #
329).

In the flowchart of FIG. 7, the description has been given of the case where the CAL-A2 is calculated from the calibration data CAL-A0 and stored, taking the case where the camera is held in the horizontal position as an example. The posture discrimination in, whether the posture of the camera held when the photographer is performing calibration is the horizontal position, whether the release button (right side of the photographer) is the vertical position in the top direction, or the vertical position in the ground direction Is determined, and according to the calibration in the posture state, as shown in FIG.
Store different calibration data in memory.

In the gaze detection operation described with reference to FIG. 6, based on the camera posture detection information in step # 201, the position of the camera at the time of gaze detection execution from FIG. Call the calibration data, and use these calibration data for individual difference correction during the line-of-sight detection process of step # 212,
A gaze detection process is performed.

According to the above embodiment, a plurality of imaging lenses 12 for forming a plurality of eyeball images having parallax.
a and 12b for detecting the line-of-sight position (line of sight) of the photographer from the eyeball image synthesized based on the eyeball image formed thereby (based on the outputs of the CCD-EYEs 14a and 14b). 1 calculation means (# 305 to # 307, # 314 to #
316, # 323 to # 329) and second arithmetic means (# 305 to # 312 or # 314 to # 314 in FIG. 7) for detecting the line of sight of the photographer from one of the plurality of eyeball images.
330, # 322) and an EEPROM 1 for storing calibration data obtained by using the first or second calculating means during the operation of the calibration data.
Since the eye-gaze detecting device has the eye gaze position 00a, it is possible to correct the eye-gaze position obtained by actual eye-gaze detection using appropriate calibration data.

At the time of actual line-of-sight detection used for selecting a focus detection area and the like, when the line-of-sight position is detected by the first arithmetic unit, the line-of-sight position is also obtained by using the first arithmetic unit. The gaze position is corrected by the calibration data, and when the gaze position is detected by the second calculating means, the gaze position is corrected by the calibration data similarly obtained by using the second calculating means. Is performed (# 21 in FIG. 6).
0 to # 212), it is possible to perform accurate gaze detection. It should be noted that the line of sight detected by any of the arithmetic means at the time of line of sight detection is determined by whether or not a Purkinje image has been detected (extracted) from any of the eyeball images. Specifically, when the Purkinje image can be detected from any of the eyeball images, the gaze detection is performed by the first calculation unit, and when the Purkinje image can be detected by only one of the eyeball images, the gaze detection is performed by the second calculation unit. The decision is made.

Furthermore, as shown in FIG. 9, by previously collecting and storing the calibration data corresponding to each posture, it is possible to select the calibration data suitable for the actual gaze detection posture. It is possible to perform more accurate gaze detection.

(Modification) Although the present invention has been described as applied to a single-lens reflex camera, it is also applicable to cameras such as a lens shutter camera and a video camera. Further, the present invention can be applied to other optical devices, other devices, and constituent units.

[0097]

As described above, according to the first aspect of the present invention, it is possible to provide an eye-gaze detecting device capable of accurately correcting the eye-gaze position using appropriate calibration data. .

Further, according to the second aspect of the present invention, it is possible to provide a visual axis detection device capable of performing an accurate visual axis detection.

[Brief description of the drawings]

FIG. 1 is a configuration diagram schematically showing a single-lens reflex camera according to an embodiment of the present invention.

FIG. 2 is a perspective view of a gaze detecting device incorporated in the single-lens reflex camera of FIG. 1;

FIG. 3 is a diagram showing a display of photographing information in a finder visual field of the single-lens reflex camera of FIG. 1;

FIG. 4 is a block diagram showing an electrical configuration of the single-lens reflex camera of FIG. 1;

FIG. 5 is a flowchart showing an operation of a main part of the single-lens reflex camera of FIG. 1;

FIG. 6 is a flowchart showing a gaze detection operation in the single-lens reflex camera of FIG. 1;

FIG. 7 is a flowchart illustrating gaze detection calibration in the single-lens reflex camera of FIG. 1;

FIG. 8 is a diagram for explaining a parallax image synthesizing effect in the present embodiment.

FIG. 9 is a diagram showing a list of calibration data according to the present embodiment.

FIG. 10 is a diagram for explaining the general principle of gaze detection.

FIG. 11 is a diagram for explaining the principle of gaze detection.

FIG. 12 is a diagram for explaining general optical system aberrations.

[Explanation of symbols]

Reference Signs List 1 shooting lens 6 focus detection device 6f image sensor 11 eyepiece lens 12a, 12b imaging lens 1, imaging lens 2 13 infrared light emitting diode (IRED) 14a, 14b image sensor (CCD-EYE1,
CCD-EYE2) 15 eyeball 16 cornea 17 iris 18 pupil 27 posture detection switch 100 CPU 101 gaze detection circuit 103 focus detection circuit 107 IRED drive circuit 110 focus adjustment circuit

Claims (6)

[Claims] 1. A finder optical system for observing an object, comprising: a plurality of imaging optical systems for forming a plurality of eyeball images having parallax with respect to an optical axis of the finder optical system; A gaze detection device that detects a position, a first calculation unit that detects a gaze position of an observer from an eyeball image synthesized based on the plurality of eyeball images, and one eyeball image of the plurality of eyeball images And an operation of obtaining calibration data for correcting the observer's line-of-sight position using the first operation unit or the second operation unit. An eye gaze detecting apparatus comprising: a calibration unit; and a storage unit for storing calibration data obtained by the calibration unit. 2. The method according to claim 1, wherein the gaze position detected by the first calculation means or the second calculation means is corrected based on the calibration data stored in the storage means when the gaze of the observer is detected. The gaze detection device according to claim 1, wherein the gaze position of a person is detected. 3. When the gaze position of the observer is detected using the first calculation means, the gaze position is corrected based on calibration data obtained using the first calculation means, and When the gaze position of the observer is detected using the second calculation means, the gaze position is corrected using calibration data obtained using the second calculation means. Item 3. A gaze detection device according to item 2. 4. When it is possible to obtain a corneal reflection image from all of the plurality of eyeball images, it is determined that the gaze position of the observer has been detected using the first calculation means,
When a corneal reflection image can be obtained from one of the plurality of eyeball images, it is determined that the gaze position of the observer has been detected using the second arithmetic unit. The eye gaze detecting device according to claim 3. 5. An attitude detecting means for detecting an attitude state of the visual line detection device at the time of observation, wherein calibration data obtained by the calibrating means is provided for each attitude state detected by the attitude detecting means. The gaze detecting device according to claim 1, wherein the gaze detecting device is stored in a storage unit. 6. The first calculating means extracts a corneal reflection image from each of the eyeball images, and thereafter shifts the positions of the corneal reflection images by calculation to make them coincide with each other. The eye-gaze detecting device according to claim 1, wherein a synthesized eye-ball image signal is generated by adopting a low signal value of each eye-ball image signal other than the reflection image signal.