JP2002328294A - Line-of-sight detector, apparatus and camera with function of line-of-sight detection - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make the information detection region exactly corresponding to the use's will rapidly selectable with good accuracy. SOLUTION: This line-of-sight (LOS) detector has selecting means (#102 to #120) which have plural first decision ranges including the information detection regions dispersed within an observation plane and second decision ranges existing at the boundary segments of the plural first decision ranges, select at least one information detection region among the plural information detection regions in accordance with the first LOS detection positions when the first LOS detection position detected by the first LOS detection operation by the LOS detecting means is not included in the second decision range and is included in any of the first decision ranges, detect the second LOS detection position by performing the LOS detection again when the first LOS position is included in the second decision range, determine the average of the first and second LOS detection positions and select the at least one information detection region among the plural information detection regions in accordance with this average.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an improvement in a gaze detection device, a device having a gaze detection function having a gaze detection means, and a camera.

[0002]

2. Description of the Related Art Conventionally, there have been provided various devices (for example, eye cameras) for detecting what position on a viewing surface a photographer is observing, that is, for detecting a so-called line of sight (a visual axis). For example, in Japanese Patent Application Laid-Open No. 1-274736, a parallel light flux from a light source is projected to the anterior segment of an eyeball of a photographer, and a gaze position is determined by using a corneal reflection image by reflected light from the cornea and an image formation position of a pupil. Seeking. In addition, the publication discloses an example in which a line-of-sight detection device is provided in a single-lens reflex camera and automatic focusing of a photographing lens is performed using line-of-sight position information of a photographer.

FIG. 9 is a schematic diagram of a line-of-sight detecting optical system provided in a single-lens reflex camera.

In FIG. 1, reference numeral 1 denotes a photographing lens, 2 denotes a main mirror, 7 denotes a focusing plate, 8 denotes a pentaprism, and 11 denotes an eyepiece. Reference numeral 15 denotes a photographer's eyeball. The photographer brings his or her eyes close to the eyepiece 11 and observes a subject in the viewfinder. Reference numerals 13a and 13b denote light sources such as light-emitting diodes that emit infrared light insensitive to the photographer. 14
Denotes an image sensor, and a part of the illumination light reflected by the eyeball 15 of the photographer is condensed on the image sensor 14 by the light receiving lens 12.

FIG. 10A is a schematic diagram of an eyeball image projected on the image sensor 14, in which 52a and 52b are shown.
Is the corneal reflection images of the red light sources 13a and 13b, 50 is the white part of the eyeball, 51 is the pupil, 53 is the skin part around the eyes, 5
4 is an iris.

FIG. 10B shows the image sensor 1 of FIG.
4 is the intensity of the output signal from a certain line (cross section EE ′ in FIG. 10A). The feature of the eyeball image is that the luminance of the corneal reflection image of the infrared light emitting diode is the highest, but the area is not so large. In addition, the pupil portion has a very low reflectance, and therefore has the lowest luminance level and occupies a certain area. The white-eye part is intermediate between the corneal reflection and the luminance level of the pupil part, and the skin part has higher or lower luminance depending on external light or illumination conditions.

FIG. 11 is a diagram for explaining the principle of gaze detection. In FIG. 11, reference numeral 15 denotes an eyeball of a photographer, 16 denotes a cornea, and 17 denotes an iris.

Hereinafter, a method of detecting a line of sight will be described with reference to the drawings.

[0009] The infrared light emitted from the light source 13b irradiates the cornea 16 of the eyeball 15 of the observer. At this time, the cornea 1
The corneal reflection image d (virtual image) formed by a part of the infrared light reflected by the surface of No. 6 is condensed by the light receiving lens 12 and forms an image at a position d 'on the image sensor 14. Similarly, the infrared light emitted by the light source 13 a
Illuminate 6. At this time, a corneal reflection image e formed by a part of the infrared light reflected on the surface of the cornea 16 is condensed by the light receiving lens 12 and is located at a position e ′ on the image sensor 14.
Image.

Light beams from the ends a and b of the iris 17 form images of the ends a and b at positions a 'and b' on the image sensor 14 via the light receiving lens 12. When the rotation angle θ of the optical axis a of the eyeball 15 with respect to the optical axis a of the light receiving lens 12 is small, the x coordinates of the ends a and b of the iris 17 are xa and x
Assuming that b, the coordinate xc of the center position c of the pupil 19 is expressed as xc ≒ (xa + xb) / 2.

The x coordinate of the midpoint between the corneal reflection images d and e substantially matches the x coordinate xo of the center of curvature o of the cornea 16.
Therefore, the x coordinate of the corneal reflection image generation position d, e is x
d, xe, assuming that a standard distance between the center of curvature o of the cornea 16 and the center c of the pupil 19 is oc, the rotation angle θx of the optical axis a of the eyeball 15 is oc × SINθx ≒ (xd + xe) / 2−xc ... (1) The relational expression (1) is substantially satisfied. Therefore, as shown in FIG. 10A, the eyeball 15 projected on the image sensor 14
By detecting the position of each characteristic point (the corneal reflection image and the center of the pupil), the rotation angle θ of the optical axis a of the eyeball 15 can be obtained.

From the equation (1), the rotation angle of the optical axis a of the eyeball 15 is given by β × oc × SINθx ≒ ｛(xpo-δx) -xic｝ × pitch (2) β × oc × SINθy ≒ ｛(ypo-δy ) −yic｝ × pitch (3). Here, θx is the rotation angle of the eyeball optical axis in the zx plane, and θy is the rotation angle of the eyeball optical axis in the yz plane. (Xpo, ypo) is the image sensor 1
4, the coordinates of the midpoint of the two corneal reflection images on (xic, yi
c) is the coordinates of the center of the pupil on the image sensor 14.
pitch is the pixel pitch of the image sensor 14. Β is an imaging magnification determined by the position of the eyeball 15 with respect to the light receiving lens 12, and is substantially obtained as a function of the interval between two corneal reflection images.

Δx and δy are correction terms for correcting the coordinates of the midpoint of the corneal reflection image, and are correction terms for correcting an error caused by illuminating the photographer's eyeball with divergent light instead of parallel light. Also, regarding δy, a correction term for correcting an offset component caused by illuminating the eyeball of the photographer with divergent light from the lower eyelid is also included.

Rotation angle of the optical axis of the photographer's eye (θx, θy)
Is calculated, the line-of-sight position (x, y) on the observation plane (focusing plate) of the photographer is as follows: x = m * (θx + Δ) when the camera is in the horizontal position (4a) y = M * θy (5a). Here, the x direction indicates the horizontal direction with respect to the photographer when the camera is in the horizontal position, and the y direction indicates the vertical direction with respect to the photographer when the camera is in the horizontal position. m is a conversion coefficient for converting the rotation angle of the eyeball into coordinates on the focus plate, and Δ is an angle between the eyeball optical axis 15a and the visual axis. In general, it is known that the rotation angle of the eyeball and the visual axis which the observer actually sees are shifted from the observer by about 5 ° in the horizontal direction and hardly shifted in the vertical direction.

By the way, the human eye involuntarily finely moves the eye even though he intends to gaze at one point, so that the line of sight calculated by the fixed amount of fine movement is displaced, and the human eye is unable to detect the line of sight. There was a problem that the accuracy was reduced.

In order to solve this problem, various measures have been taken such as increasing the number of times of gaze detection and improving the accuracy of gaze. Japanese Patent Laying-Open No. 5-20040 proposes a camera that determines whether or not a photographer is gazing based on a plurality of gaze detection results.

Japanese Patent Application Laid-Open No. Hei 6-289285 proposes a focus area setting device for selecting an optimum one from a plurality of focus detection areas based on the amount of movement of the photographer's line of sight.

Here, the operation of a focus detection device conventionally provided in a camera will be described.

First, the release button is half-pressed (first
When the stroke is pressed), the gaze detection device operates before the focus detection, and obtains the gaze position of the photographer in the finder visual field. This point is represented by coordinates in the viewfinder field. From the coordinates in the finder field of view, the corresponding focus detection area is determined. The focus state is detected by the focus detection device with respect to the focus detection area thus obtained, and the photographing lens is driven to the focused state based on the information.

As described above, after the focus detection area is determined by the visual axis detection device, the focusing operation is performed only once based on the focus state of the focus detection area.

[0021]

However, in the camera proposed in Japanese Patent Laid-Open No. 5-20040, the line of sight detection is always performed a plurality of times to select the focus detection area. Cannot select the focus detection area. For this reason, it takes time until the focus detection is started, and a quick focus detection operation cannot be performed.

Further, if the focus detection area is always selected by one-time line-of-sight detection, an erroneous focus detection is performed when the line-of-sight position is detected at an intermediate position between one focus detection area and the adjacent focus detection area. There was a case where an area was selected.

These problems are the same even when the target to be selected in the visual line detection is not the focus detection area.

(Object of the Invention) A first object of the present invention is to provide an eye-gaze detecting device and an eye-gaze detecting device capable of quickly and accurately selecting an information detection area, a focus detection area, or an index accurately corresponding to a user's intention. An object of the present invention is to provide a device with a detection function and a camera.

A second object of the present invention is to provide a device with a visual axis detection function and a camera which can accurately focus on an object intended by a user.

[0026]

According to a first aspect of the present invention, there is provided a visual line detecting means for detecting a position of a visual line in a user's observation plane, and a visual line detection means for detecting a user's visual line position in the observation plane. A plurality of first determination ranges including the information detection region and a second determination range at a boundary between the plurality of first determination ranges. The first gaze detection position detected by the first gaze detection operation by the gaze detection means is the second gaze detection position
If not included in the determination range, and is included in any of the first determination range, select at least one information detection region from the plurality of information detection regions based on the first line-of-sight detection position, If the first line-of-sight position is included in the second determination range, line-of-sight detection is performed again to detect a second line-of-sight detection position, and an average of the first and second line-of-sight detection positions is obtained. The gaze detection device includes a selection unit that selects at least one of the plurality of information detection regions based on the average.

Similarly, in order to achieve the first object,
The invention according to claim 2 is a gaze detection unit that detects a gaze position in a user's observation plane, and a gaze detection function device having a plurality of indices provided in the observation plane,
A first determination range including a plurality of first determination ranges including the index and a second determination range at a boundary between the plurality of first determination ranges, the first determination range being detected by a first gaze detection operation by the gaze detection unit; If the gaze detection position is not included in the second determination range and is included in any of the first determination ranges, at least one of the plurality of indices is determined based on the first gaze detection position. Is selected, and if the first line-of-sight position is included in the second determination range, line-of-sight detection is performed again to detect a second line-of-sight detection position, and the first and second line-of-sight detection positions are detected. An apparatus with a line-of-sight detection function having an selecting means for obtaining an average and selecting at least one of the plurality of indices based on the average is provided.

In order to achieve the second object,
According to a fourth aspect of the present invention, there is provided a visual line detection unit that detects a line of sight position of a user in an observation plane, and a focus detection unit that performs focus detection for each of a plurality of focus detection areas provided in the observation plane. In the device with gaze detection function having
It has a plurality of first determination ranges including the focus detection area and a second determination range at a boundary portion between the plurality of first determination ranges, and is detected by a first gaze detection operation by the gaze detection unit. When the first gaze detection position is not included in the second determination range and is included in any of the first determination ranges, the first gaze detection position is included in the plurality of focus detection regions based on the first gaze detection position. If at least one focus detection area is selected and the first line-of-sight position is included in the second determination range, line-of-sight detection is performed again to detect a second line-of-sight detection position, and the first and second line-of-sight detection positions are detected. Selecting means for obtaining an average of the two eye-gaze detection positions and selecting at least one focus detection area from the plurality of focus detection areas based on the average; and focus detection information in the focus detection area selected by the selection means. The focus Obtained from the detection means, it is an visual axis detection function device having a focus adjusting unit that performs focus adjustment of the optical system.

According to a fifth aspect of the present invention, there is provided a gaze detecting apparatus according to the first aspect, or a gaze detecting apparatus according to any one of the second to fourth aspects. The camera is provided with a device with a function.

[0030]

FIG. 1 is a block diagram showing a main part of an electric circuit built in a camera according to an embodiment of the present invention.

A line-of-sight detection circuit 101, a central processing unit (hereinafter referred to as CPU) 100 of a microcomputer, which is a camera control means built in the camera body, is provided.
Photometry circuit 102, focus detection circuit 103, signal input circuit 1
04, LCD drive circuit 105, LED drive circuit 106,
An IRED drive circuit 107, a shutter control circuit 108, and a motor control circuit 109 are connected. Also, a focus adjustment circuit 110 and an aperture driving circuit 1 disposed in the taking lens
Signals are transmitted to the device 11 via the mount contact 37.

An EEPROM 100a as storage means attached to the CPU 100 can store film counters and other photographing information.

The line-of-sight detection circuit 101 outputs an eyeball image from the image sensor 14 (CCD-EYE) to a CPU.
Send to 100. Then, the CPU 100 subjects the eyeball image signal from the image sensor 14 to A / D conversion by an A / D conversion unit inside the CPU, and converts this image information into feature points of the eyeball image necessary for visual line detection as described later. And the rotation angle of the photographer's eyeball is calculated from the position of each feature point. The photometric circuit 102 amplifies the output from the photometric sensor 10, logarithmically compresses the output, and sends the result to the CPU 100 as luminance information of each sensor. The photometric sensor 1
Numeral 0 divides the screen into a plurality of sections, each of which is constituted by a plurality of photodiodes for outputting a photoelectric conversion output.

The line sensor 61 includes five sets of line sensors C corresponding to five focus detection areas 200 to 204 in the screen.
CD-L2, CCD-L1, CCD-C, CCD-R
1, a known CCD line sensor composed of CCD-R2. The automatic focus detection circuit 103 sends the voltages obtained from these line sensors 61 to the CPU 100,
In the PU 100, the line sensor signal is sequentially A / D converted by the built-in A / D converter.

SW1 is O at the first stroke of the release button.
N, a photometry switch for starting photometry, AF, and line-of-sight detection operation, SW2 is a release switch that is turned on by the second stroke of the release button, SW-DIAL1 and SW-DIAL2.
Is a dial switch provided in the electronic dial, which is input to an up / down counter of the signal input circuit 104,
Counts the amount of rotation click of the electronic dial.

Next, the operation of the entire sequence of the camera will be described with reference to the flowchart of FIG. In addition,
In this flow, a normal sequence of the camera is described, but no special operation or trouble handling for an unexpected accident is described.

In step # 601, when the mode dial (not shown) is rotated to set the camera from a non-operation state to a predetermined photographing mode, the power of the camera is turned on. Then, the CPU 100 proceeds to step # 602, and initializes variables and flags. Specifically, it clears the "line-of-sight detection prohibition flag", clears the "AF prohibition flag", prohibits the "release &feed" interrupt, and clears the "release flag".

In step # 604, the state of the switch SW1 which is turned on by the first stroke of the release button is detected. If the switch SW1 is off, the process proceeds to step # 605, and if it is on, the process proceeds to step # 612. Step # 6 assuming that the switch SW1 is OFF
In step 05, variables and flags are initialized. Here, initialization is performed when the switch SW1 is once turned on and a predetermined operation is performed, and then the switch SW1 is turned off again. Specifically, the “flag AFING” indicating that the switch SW1 has not been turned on is cleared. Also, the “eye-gaze detection prohibition flag” is cleared, and the eye-gaze detection is performed.
Perform F. Further, the "release &feed" interrupt is prohibited. Also clear the "release flag".

Next, in step # 606, it is checked whether or not the current camera state is the feed mode setting state. If the feeding mode is set, the process proceeds to step # 608, and if not, the process proceeds to step # 614.
In step # 608, it is checked whether or not the dial has changed. If there is no change in dial, the flow returns to step # 604 without switching the feeding mode. On the other hand, if there is a dial change, the process proceeds to step # 610, and the feed mode is changed every time the dial changes. The feed mode includes a "single shooting mode" in which only one image is shot when the switch SW2 is turned on, and a "continuous shooting mode" in which shooting is continuously performed while the switch SW2 is turned on. The “single shooting mode” and the “continuous shooting mode” are switched every time. When the setting of the feed mode is completed, step # 60
Return to 4.

If the feed mode is not set in step # 606, the process proceeds to step # 614 as described above, and it is determined whether the camera is in the AF mode switching state. As a result, if the AF mode is switched, the process proceeds to step # 616, where the AF mode is switched. When the setting of the AF mode is completed, the process returns to step # 604. If the state is not the switching mode of the AF mode, the process proceeds to step # 618, and the setting other than the feeding mode and the AF mode is performed according to the state of the switch.

At step # 604, the switch SW1
Is ON, the process proceeds to step # 612 as described above,
Here, an "AE control" subroutine such as photometry or camera status display is executed. In this subroutine "AE control", the photometric sensor 10 and the photometric circuit 102 are operated, and the AE control value is calculated for the photometric value by a known algorithm. The calculated AE control value is displayed on an external liquid crystal of the camera. Next, the routine proceeds to step # 613, where a subroutine of "line-of-sight detection and AF (focus detection)" is executed. This subroutine is executed while the switch SW1 is ON.
It is repeated. Thereafter, the flow returns to step # 604, and the operation of the camera is repeatedly performed.

Next, using the flowchart of FIG.
The operation of “release & feeding” will be described. This operation is performed in an interrupt routine.

When the interruption is permitted, the release button is pressed to the second stroke, and the switch SW2 is turned on.
If N, the "release &feed" subroutine is called by the interrupt processing, and first, in step # 688, the "release feed" interrupt is prohibited. And
In the next step # 692, the control value of the aperture and the shutter speed are calculated. The aperture and the shutter speed are calculated by a predetermined algorithm according to the photographing mode of the camera, the photometric value or the set value. Then, the next step # 69
In step 3, the CPU 100 raises the main mirror and the sub-mirror of the camera, and communicates with the aperture driving circuit 111 to move the aperture 31 provided in the lens to the above step # 6.
Control to the value calculated in 92.

In the next step # 694, the shutter magnet MG-1 is controlled via the shutter control circuit 108.
And exposure is started by running the front curtain of the shutter. Then, after a lapse of the predetermined time calculated in step # 692, the shutter magnet MG-2 is energized, and the rear curtain of the shutter is run to complete the exposure. In the following step # 695, each mirror is lowered to a predetermined position, communication is performed with the aperture driving circuit 111, and the aperture is returned to the open state.

Then, the flow advances to step # 696 to feed one frame of film, and to charge the shutter spring. Then, in the next step # 698, when the photographing of one frame is completed, the "eye gaze detection prohibition flag" is set to perform the eye gaze detection when the "eye gaze detection" subroutine is called next.
Is cleared, and in the next step # 699, a flag "release flag" indicating that the release has been performed is set. Then, in step # 700, the "release &feeding" interrupt subroutine is terminated.

After the "release &feeding" subroutine is completed, the process returns to step # 604 in FIG. 2 instead of returning to the program where the interrupt occurred.

Next, the operation of the "line-of-sight detection &AF" subroutine of step # 613 will be described with reference to the flowchart of FIG.

When "line-of-sight detection &AF" is called, a subroutine is executed from step # 631 in FIG.

First, in step # 630, it is determined whether or not gaze detection is permitted. Prohibition of gaze detection can be determined by a “gaze detection prohibition flag”. At first, the process is permitted, and the process proceeds to step # 631. If the gaze detection is prohibited, the process proceeds to step # 63 without performing the gaze detection operation.
Proceed to 637.

When the flow advances to step # 631, it indicates that the gaze input has been performed before the gaze detection is performed.
A gaze input mark is displayed. Specifically, LED for lighting
(F-LED) 25 is turned on, and the LCD drive circuit 105 is turned on.
Through the LCD 24 in the finder (and the monitor L
The gaze input mark on the CD 26) is turned on. This allows the photographer to confirm that the camera is performing line-of-sight detection. In the next step # 632,
The subroutine "line-of-sight detection & focus detection area selection" is called, and the focus detection area is selected from the line-of-sight detection and the line-of-sight position coordinates obtained by the line-of-sight detection.

Here, the line-of-sight position coordinates and the focus detection area will be described with reference to FIG.

In FIG. 5A, L2, L1, C, R
1, R2 are CCD-L2 of the line sensor 61, respectively.
CCD-L1, CCD-C, CCD-R1, CCD-R
2 is a focus detection area for performing focus detection at each location. EYL2, EYL1, EYC, EYR
1, EYR2 are focus detection areas L2, L1, respectively.
This is a first determination range for selecting C, R1, and R2. EYEL2, EYEL1, EY in the viewfinder
Areas other than C, EYR1 and EYR2 are EYEOTHR
It is.

Each area can be represented as follows by coordinates with the center of the finder as the origin.

Range of EYL2: -x3 ≦ x <−x2 and −y1 ≦ y ≦ y1 Range of EYL1: Range of −x2 ≦ x <−x1 and −y1 ≦ y ≦ y1 Range of EYC: −x1 <x <x1 and -Y1≤y≤y1 Range of EYR1: x1 <x≤x2 and -y1≤y≤y1 Range of EYR2: x2 <x≤x3 and -y1≤y≤y1 Range of EYOTHR: -xa≤x≤xa and- ya ≦ y ≦ ya
Next, a portion excluding -x3≤x≤x3 and -y1≤y≤y1 will be described with reference to FIG. 5B.

EYBN is a first determination range EYL2, EYL
Distance d from the boundary of YL1, EYC, EYR1, and EYR2
Is a region with a width of 2d in the range indicated by hatching in the figure.

When expressed in coordinates with the center of the finder as the origin, -x3-d≤x≤x3 + d and y1-d≤y≤y1 + d and -x3-d≤x≤x3 + d and -y1-d≤y≤-y1 + d and −x3−d ≦ x ≦ −x3 + d and −y1 + d ≦ y ≦ y1−d and −x2−d ≦ x ≦ −x2 + d and −y1 + d ≦ y ≦ y1−d and −x1−d ≦ x ≦ −x1 + d and −y1 + d ≦ y ≦ y1−d and x1−d ≦ x ≦ x1 + d and −y1 + d ≦ y ≦ y1−d and x2−d ≦ x ≦ x2 + d and −y1 + d ≦ y ≦ y1−d and x3−d ≦ x ≦ x3 + d and − y1 + d ≦ y ≦ y1-d.

For example, when (xn, yn) in FIG. 5B is detected as the line-of-sight coordinates, (xn, yn) is represented by “−”.
x2 + d <x <-x1-d and -y1 + d <y <y1-d ". That is, it is included in the range of EYL1 and EYBN
Is not included in the range, the line-of-sight detection is not performed again, and the focus detection area L1 is immediately selected.

As another example, in the first gaze detection, FIG.
The case where the line-of-sight position is detected at the position (xm1, ym1) in (b) will be described.

This position is included in the range of EYR1, that is, "x1 <x≤x2 and -y1≤y≤y1".
YBN range "-x3-d≤x≤x3 + d and -y1-d≤y
.Ltoreq.-y1 + d ", the line-of-sight detection is performed again without immediately selecting the focus detection area.

The line of sight is detected again (xm2, ym2).
Are detected, the first line-of-sight coordinates (xm1, ym1) and the second line-of-sight coordinates (xm2, ym) are detected.
The average (xma, yma) of 2) is obtained.

(Xma, yma) = ((xm1, xm
1) / 2, (ym1, ym1) / 2) Since these coordinates are in the range of “x1 <x ≦ x2 and −y1 ≦ y ≦”, that is, in the area of EYR1, the focus detection area R1 is selected. When the line-of-sight detection is performed again, the first line-of-sight coordinates and the second line-of-sight coordinates are averaged, and the position is set to EYBN.
The determination is made only in the first range without determining whether there are coordinates in the range.

Even if the line-of-sight position obtained by the second line-of-sight detection is (xm1, ym1), since the coordinates are included in EYR1, the focus detection area R1 is selected.

Only when there is a line-of-sight coordinate in the second range,
Re-detecting the line of sight and using the average of the two times is because the human eye, even though he intends to gaze at a single point, moves the eyeball incidentally. This is to solve the problem that the line of sight is shifted and the accuracy of line of sight detection is reduced. Further, even if the second range is included in the first range and the second range (not near the boundary) takes into account the minute movement of the fixation, a result that matches the intention of the photographer can be obtained. The detection speed is also increased by eliminating the necessity of performing.

As described above, when it can be considered that the line-of-sight detection result looks near the focus detection area, the focus detection area is quickly determined by determining the focus detection area immediately instead of uselessly repeating the line-of-sight detection. You can choose. In addition, when the gaze position is detected at the boundary of the determination range for selecting the focus detection area, the gaze detection is performed again and the focus detection area that matches the intention of the photographer is selected by using the average value of a plurality of times. can do.

Next, using the flowchart of FIG.
The “line-of-sight detection & focus detection area selection” subroutine will be described. When this subroutine is called, step # 1
01 starts the execution of the program.

In step # 101, the flag for re-detecting the line-of-sight is cleared.
Then, in the next step # 102, gaze detection is performed according to a predetermined algorithm. Specifically, first, IR
An IRED (infrared light emitting diode) predetermined by the position of the camera
Lighted by the ED drive circuit 107, the eye-gaze detection circuit 10
In step 1, the accumulation operation of the image sensor 14 is started. When the accumulation is completed, the accumulated charges are sequentially read out to the CPU 100, A / D converted, and processed by a predetermined algorithm. The process is performed on all the pixels of the image sensor 14 to obtain the coordinates of the corneal reflection images 52b and 52a of the eyeball illumination light source and the center coordinates of the pupil 51 shown in FIGS.
By calculating these in accordance with the algorithm described above, the coordinates (xn, yn) at which the photographer gazes are obtained.

Next, in step # 103, the "re-gaze flag" is determined, and in the case of re-gaze (detection of gaze again), the flow proceeds to step # 120. At first, since the “revision line flag” is cleared in step # 101, step #
The process proceeds to step 104, where the gaze position coordinates (xm1,
ym1) is in the second determination range EYBN. If it is within EYBN, the process proceeds to step # 118,
The “review line flag” is set to “1”. Then, in the next step # 19, the result of the eye gaze detection performed this time is stored. Assuming that the line-of-sight coordinate position obtained this time is (xm1, ym1), “x0 = xm1, y0 = ym1” is stored. Thereafter, the flow returns to step # 102, and the line-of-sight detection is performed again.

As described above, the boundary area, that is, the second determination range EYBN (FIG.
(A shaded portion in (b)) is provided, and when the gaze position is within this range, the current gaze position is stored and then the gaze detection is performed again. In the case where the gaze detection is performed again, after performing the gaze detection in step # 102, step # 103 is performed.
, The average of the previously stored line-of-sight position (x0, y0) and the current line-of-sight position (xm2, ym2) is calculated as xma = (x0 + xm2) / 2 yma = (y0 + ym2) / 2 is calculated. This calculation does not necessarily need to be a simple average. For example, an arbitrary coefficient α (0 <α <
1) When β = 1−α, a weighted average such as xma = (α · x0 + β · xm2) / 2 yma = (α · y0 + β · ym2) / 2 may be used. (Xn, yn) = (xm
a, yma) and proceeds to step # 105.

If the line-of-sight coordinates (xm1, ym1) obtained first in step # 104 are not within EYBN,
Step # 1 using the value of (xm1, ym1) as it is
The determination after 05 is performed.

At step # 105, (xn, y
It is determined whether n) is within the range of one of the first determination ranges, EYL2 (see FIG. 5). If it is within EYL2, the process proceeds to step # 110, and the focus detection area L2 is selected.
If not within EYL2, the process proceeds to step # 106,
It is determined whether (xn, yn) is within the range of one of the first determination ranges, EYL1. If it is within EYL1, the process proceeds to step # 111, and the focus detection area L1 is selected.

If not in EYL1, step # 1
07, (xn, yn) is one of the first determination ranges,
It is determined whether it is within the range of EYC. If it is within the EYC, the process proceeds to step # 112, and the focus detection area C is selected. If it is not within EYC, the process proceeds to step # 108, where (xn, yn) is one of the first determination ranges, EYR1
Is determined to be within the range. If it is within EYR1, the process proceeds to step # 113, and the focus detection area R1 is selected.

If not within EYR1, step # 1
09, (xn, yn) is one of the first determination ranges,
It is determined whether it is within the range of EYR2. If it is within EYR2, the process proceeds to step # 114, and the focus detection area R2 is selected.

If it is not within EYR2, step # 1
Go to No.15, EYL2, EYL1, EYC, EYR
1 and EYR2, the focus detection area is selected irrespective of the result of the line-of-sight detection. In this case, the central focus detection area may be selected in advance, or the focus detection area selected as the previous result may be selected again. The focus detection area may not be selected in this subroutine, but may be selected later based on the focus detection result.

When the focus detection area is selected in steps # 110 to # 115, the process proceeds to step # 116, where the LED
A signal is transmitted to the drive circuit 106 to blink the focus detection area using the LED 21 for superimposition (it may be turned on). Then, the subroutine returns in step # 117.

Returning to FIG. 4, the description will be continued.

In step # 635, once the line of sight is detected, the line of sight detection is prohibited so as not to repeatedly perform the line of sight detection. Prohibition of gaze detection is performed by setting a “gaze detection prohibition flag”. Further, "AFING = 1" is set here. Thereafter, it is determined whether or not this is the first "line-of-sight detection &AF" after the switch SW1 is turned on by this flag. Next step # 63
In step 7, it is determined whether or not the lens is being driven by communicating with the lens driving circuit 110. Last time, "Gaze detection & A
If the F "subroutine has been executed and the lens drive has been performed, and the lens drive has not been completed, the process proceeds to step # 680. Before performing the lens drive or after the lens drive has been completed, the process proceeds to step # 640. .

In step # 640, whether or not the focus detection is permitted is determined by the "AF prohibition flag". Once focused, switch SW1 is turned on
Is maintained, the "AF prohibition flag" is set. In this case, the process proceeds to step # 680. Otherwise, the process proceeds to step # 642 to perform focus detection.
The focus detection is performed on the focus detection area selected by the line-of-sight detection from the plurality of focus detection areas. And the next step #
In step 644, it is determined whether or not the focus detection of the focus detection area performed in step # 642 has been impossible.
As a result, if the focus cannot be detected, the process proceeds to step # 654. If the focus can be detected, the process proceeds to step # 646.

Assuming that the focus cannot be detected, step # 65
In step 4, AF disabled display is performed to inform the photographer that the focus detection result is disabled. AF disabled display
This is performed by blinking the focus mark on the LCD 24 in the viewfinder. Then, in the next step # 656, the "line-of-sight detection prohibition flag" is cleared to perform line-of-sight detection again. As a result, “Gaze detection &
When the AF "subroutine is called, visual line detection and focus detection are performed. Thereafter, the flow proceeds to step # 680.

If the focus has been detected in step # 644, the process proceeds to step # 646, as described above, and it is determined whether the focus detection result is in focus. If the defocus amount obtained by the focus detection is within a predetermined amount, it is determined that the object is in focus. If in focus, the process proceeds to step # 648, and a focus display is performed to notify the photographer that the focus detection result is in focus. This means that the lighting LED (F-LED) 25 is turned on and L
LCD 24 in viewfinder via CD drive circuit 105
The focus mark is turned on. Then, in the next step # 650, since the focus is achieved, a "release &feed" interrupt is permitted, and when the switch SW2 is turned on, the release operation is performed by the interrupt. Further, an “AF prohibition flag” is set so that focus detection is not performed again. Then, the process proceeds to step # 680.

If the focus has not been reached in step # 646, the flow advances to step # 652 to drive the lens. Specifically, the lens drive amount is obtained from the defocus amount detected in step # 642, and is communicated to the lens drive circuit 110. Then, the lens driving circuit 11
0 monitors the lens driving motor 33 and the pulse plate 36
And the lens driving amount communicated is driven. After communicating the data to the lens drive circuit 110, the CPU 100 does not need to monitor the lens drive amount and can perform another operation while driving the lens. Therefore, when the communication with the lens driving circuit ends, step #
Proceed to 680.

In step # 680, the subroutine "line-of-sight detection &AF" is terminated, and the routine returns.

According to the first embodiment described above, a plurality of first determination ranges EYL1, EYL2, EYC, EYR1, EY including the focus detection areas L1, L2, C, R1, R2.
A second determination range E is provided at the boundary between R2 and the first determination range.
If the first gaze detection position detected by the first gaze detection operation is not included in the second determination range and is included in any of the first determination ranges, At least one focus detection area is selected from the plurality of focus detection areas, and if the first line-of-sight position is included in the second determination range, the line-of-sight detection is performed again to perform the second line-of-sight detection. The position is detected, and the focus detection area is determined according to the first and second gaze detection positions, that is, by an average (including a weighted average).

In this way, when it is not necessary to execute the line-of-sight detection a plurality of times, the focus detection area is selected only once, and only when necessary, the line-of-sight detection is executed again to achieve high accuracy. Obtaining a line-of-sight detection result enables selection of a correct focus detection area.

More specifically, if the line of sight detected by the line of sight is close to the focus detection region, the correct focus detection region can be selected even if there is an error such as fixation, but the line of sight determined by the line of sight determines the focus detection region. Is near the boundary of the first determination range, that is, in the second determination area, the gaze detection is performed again in consideration of an error such as fixation fine movement, and a correct focus detection area is selected from a plurality of gaze detection results. I am trying to do it. For this reason, the gaze position is a gray zone, that is,
In the case of the second determination range, the visual axis detection is performed again, and the focus detection area is selected. For example, when a gaze position is detected in the middle of two focus detection areas, gaze detection is performed again, and an accurate gaze position is detected from a plurality of gaze detection results. If a focus detection area is selected from this result, a focus detection area that accurately corresponds to the user's intention can be selected. Further, if the detected line-of-sight position is not in the gray zone, the selection of the focus detection area is not affected even if an error for the fixation fine movement is included. In this case, since the focus detection area is immediately determined and the line of sight detection is not performed, the focus detection area can be quickly selected.

(Second Embodiment) In the first embodiment, an example in which the focus detection area is selected by the line of sight detection has been described. However, it is not always necessary to select the focus detection area by the line of sight detection. good.

FIG. 7 shows an example of the case where the photographing mode of the camera is selected based on the line of sight. In the viewfinder, a mark "P" indicating each shooting mode: MKP, "T
“v”: MKT, “Av”: MKA, “M”: MKM are displayed, and the photographing mode is changed by looking at the respective marks to set the program AE mode, the shutter speed priority AE mode, the aperture priority AE mode, and the manual photography. Mode can be changed. ARP, ART, ARA,
ARM is a first determination range for selecting MKP, MKT, MKA, and MKM, and ARBN is a second determination range.
(The hatched portion), which is the boundary portion of the first determination range.

For example, (xp1, yp1) in FIG.
Is detected as the line-of-sight coordinates, since these coordinates are included in the second determination range ARBN, the line-of-sight detection is performed again.
The line-of-sight coordinates obtained by the line-of-sight detection performed again are (xp2, yp)
In the case of 2), the average of the two coordinates is obtained, and the determination is performed using the coordinates. Assuming that the averaged coordinates are (xm, ym), the mark MKT, that is, the shutter speed priority AE mode, is selected because it is within the range of ART which is one of the first determination ranges.

Here, the average of the line-of-sight detection results is used because the line-of-sight detection results include an error of the fixation of the eyeball and an error of the detection system. If the line of sight is detected near the boundary of, if the value is used as it is, there is a risk that a mode contrary to the intention of the photographer may be selected. In such a case, by averaging a plurality of line-of-sight detection results, it is possible to reduce the effects of fixed eye movement and noise.

As described above, when the line-of-sight detection result looks near the mark, the line-of-sight detection is not repeated in vain.
By immediately selecting the shooting mode, the shooting mode can be quickly selected. In addition, when the gaze position is detected at the boundary of the determination range for selecting the shooting mode, the gaze detection is performed again and the average value is used to select a focus detection area that matches the intention of the photographer. it can.

Next, a subroutine of "line-of-sight detection & photographing mode selection" according to the second embodiment of the present invention will be described with reference to the flowchart of FIG.

This subroutine is called when the “line-of-sight photographing mode selection” of the camera is set. Since this is not directly related to the description of the present invention, a description thereof will be omitted.

When the subroutine is called, execution of the program is started from step # 501. In step # 501, EYCNT for counting the number of gaze detection is initialized to 1. And the next step #
At 502, gaze detection is performed according to a predetermined algorithm. The coordinates of the photographer's gaze obtained here are (xm, y
m). In the following step # 503, it is determined whether or not EYCNT is 1, and if not, the process proceeds to step # 520. At first, since EYCNT is initialized to 1 in step # 501, the process proceeds to step # 523, where E
It is determined whether YCNT is larger than 3, and if it is larger, the process proceeds to step # 505. If it is 3 or less, the process proceeds to step # 504. This determination is made in order to limit the number of repetitions so that the line of sight detection is not repeated indefinitely.
(However, the judgment of the limit does not necessarily have to be 3,
The number of times does not change the essence of the present invention. When the process proceeds to step # 504, it is determined whether or not the line-of-sight position coordinates (xm, ym) to be determined are within the second determination range ARBN. When the coordinates are within the second determination range ARBN, the process proceeds to step # 518. The result of the visual line detection performed is stored. Assuming that the line-of-sight coordinate position obtained this time is (xm, ym), xp (i) = xm yp (i) = ym (i = EYCNT). Then, in the next step # 519, the EYCN
T is counted up, and the process returns to step # 502 to perform the line-of-sight detection again.

As described above, the boundary area (the second determination range ARBN) is provided at the boundary of the first determination range, and if there is a gaze position within this range, the current gaze position is stored and then the gaze position is stored again. Detection is performed.

In the case of the re-gaze, after the gaze is detected in step # 502, the flow proceeds to step # 503, and the flow proceeds to step # 503.
Proceed to 520. Then, in this step # 520, the line-of-sight position (xp (i), yp
(I)) The average of (i = 1 to EYCNT) is obtained. This average value is defined as (xm, ym). In this calculation, it is not always necessary to store all the past gaze detection coordinates, and it is not necessary to calculate the average value of all of them, so that the average of the detection results up to that point is always obtained at (xm, ym). It may be obtained by a weighted average. For example, an arbitrary coefficient α (0 <α <
1) When β = 1−α, xm = (α · xp + β · xm) / 2 ym = (α · yp + β · ym) / 2 (where xp and yp are the previous values of xm and ym) The weighted average may be used as follows.

When the average coordinates (xm, ym) are obtained, the process proceeds to step # 523 again. If the predetermined number of times is not exceeded, the process proceeds to step # 505. If not, the process proceeds to step # 504. Is determined to be within the determination range ARBN, and if not, the process proceeds to step # 505.
Proceed to.

At step # 505, (xm, y
m) is within one of the first determination ranges, ARP. If it is within the ARP, the process proceeds to step # 510, and the mark MKP is selected. As a result, the shooting mode becomes the program AE mode. If it is not within the ARP, the process proceeds to step # 506, and it is determined whether (xm, ym) is within the ART range, which is one of the first determination ranges. If it is in the ART, go to step # 511,
Select the mark MKT. As a result, the shooting mode becomes the shutter speed priority AE mode.

If not within the above ART, the process proceeds to step # 507, and it is determined whether (xm, ym) is within the ARA range, which is one of the first determination ranges. If it is within the ARA, the process proceeds to step # 512, and the mark MKA is selected. As a result, the shooting mode becomes the aperture priority AE mode. If not in the ARA, step # 50
8, (xm, ym) is one of the first determination ranges, A
It is determined whether it is within the range of RM. If it is within the ARM, the process proceeds to step # 513, and the mark MKM is selected.
As a result, the shooting mode becomes the manual exposure mode.

After changing one of the photographing modes in steps # 510 to # 513, the flow advances to step # 516 to display the photographing mode, and returns to the main routine in step # 517.

If it is determined in step # 508 that the image does not fall within the first determination range, the photographing mode is not changed, and the process returns to the main routine in step # 517.

According to the second embodiment, a plurality of marks (indexes) MKP, MKP provided in the viewfinder are provided.
A plurality of first determination ranges AR including KT, MKA, and MKM
P, ART, ARA, ARM and a second determination range ARBN at a boundary between the first determination range,
If the first gaze detection position detected by the first gaze detection operation is not included in the second determination range and is included in any of the first determination ranges, at least one of the plurality of marks is included. Is selected, and if the first line-of-sight position is included in the second determination range, a line-of-sight detection is performed again to detect a second line-of-sight detection position, and according to the first and second line-of-sight detection positions. To determine the mark.

Therefore, when it is not necessary to perform the gaze detection a plurality of times, the mark (corresponding photographing mode in this example) can be selected only once, and the gaze detection is executed again only when necessary, and each gaze detection result is obtained. (Average (weighted average)), the mark can be selected with high accuracy.

More specifically, if the line of sight detected by the line of sight is close to the mark, a correct mark can be selected even if there is an error such as slight fixation, but the first determination range in which the line of sight determined by the line of sight determines the mark. Is near the boundary, that is, within the second determination region, the gaze detection is performed again in consideration of an error such as a fixation fine movement, and a correct mark can be selected from a plurality of gaze detection results.

(Modification) In the above embodiment, an example in which the present invention is applied to a camera has been described. However, the present invention is not limited to this.
The present invention can also be applied to a distance measuring device that measures the distance to an object in a distance measuring area. Further, the apparatus has a line-of-sight detection device that has a mark (other than a mark corresponding to a shooting mode) for instructing a desired operation start by a line of sight and selects a mark selected by the line of sight, The present invention is also applicable to an optical device (such as a personal computer) that starts an operation corresponding to a mark selected by the line of sight.

[0104]

As described above, according to the first to third or fifth aspects of the present invention, an information detection area, a focus detection area or an index corresponding to the user's intention can be selected quickly and accurately. The present invention can provide a gaze detection device, a gaze detection function-equipped device, or a camera.

According to the fourth or fifth aspect of the present invention, it is possible to provide a device or a camera with a line-of-sight detection function that can accurately focus on an object intended by the user.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an electrical configuration of a camera according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing an operation of a main part of the camera of FIG.

FIG. 3 is a flowchart showing an operation of release and feeding according to the first embodiment of the present invention.

FIG. 4 is a gaze detection & A according to the first embodiment of the present invention.
6 is a flowchart showing the operation of F.

FIG. 5 is a diagram for describing a line-of-sight range according to the first embodiment of the present invention.

FIG. 6 is a flowchart showing an operation of gaze detection and focus detection area selection according to the first embodiment of the present invention.

FIG. 7 is a diagram for describing a line-of-sight range according to a second embodiment of the present invention.

FIG. 8 is a gaze detection & A according to a second embodiment of the present invention.
6 is a flowchart showing the operation of F.

FIG. 9 is an optical layout diagram of the single-lens reflex camera.

FIG. 10 is a diagram schematically illustrating an eyeball image of a photographer and output signals.

FIG. 11 is a diagram for explaining the principle of a general gaze detection method.

[Explanation of symbols]

61 Line sensor 14 Image sensor 100 CPU 101 Eye line detection circuit 103 Automatic focus detection circuit 105 LCD drive circuit 107 IRED drive circuit 110 Focus adjustment circuit L2, L1, C, R1, R2 Focus detection area EYL2, EYL1, EYC, EYR1, EYR2 First determination range EYBN Second determination range MKP, MKT, MKA, MKM mark ARP, ART, ARA, ARM First determination range ARBN Second determination range

Claims (5)

[Claims] 1. A gaze detection device comprising: a gaze detection unit configured to detect a gaze position of a user in an observation plane; and a plurality of information detection areas provided in the observation plane. Has a first determination range and a second determination range at a boundary between the plurality of first determination ranges, and the first gaze detection position detected by the first gaze detection operation by the gaze detection unit is If not included in the second determination range and is included in any of the first determination ranges, at least one information detection region of the plurality of information detection regions is determined based on the first line-of-sight detection position. If the first gaze position is selected and included in the second determination range, gaze detection is performed again to detect a second gaze detection position, and the average of the first and second gaze detection positions is determined. Is calculated, and based on the average, Visual axis detecting apparatus characterized by having a selection means for selecting at least one of the information detection region of the information detection region. 2. An apparatus with a line-of-sight detection function having a line-of-sight detecting means for detecting a line-of-sight position of a user in an observation plane, and a plurality of indices provided in the observation plane, wherein A first gaze detection position detected by the first gaze detection operation by the gaze detection means, the first gaze detection position having a second determination range at a boundary between the first determination range and the plurality of first determination ranges. If not included in the second determination range and is included in any of the first determination ranges, at least one of the plurality of indices is selected based on the first line-of-sight detection position, and the first If the gaze position is included in the second determination range, the gaze detection is performed again to detect the second gaze detection position, and the average of the first and second gaze detection positions is obtained. At least one of the plurality of indices based on Visual axis detection function device characterized by having a selection means for selecting One indicator. 3. The apparatus according to claim 2, wherein the plurality of indices correspond to a plurality of selectable operations of the device, and an operation corresponding to the indices selected by the selecting means is started. A device with a line-of-sight detection function according to 1. 4. A line-of-sight detection function comprising: line-of-sight detection means for detecting a line-of-sight position of a user in an observation plane; and focus detection means for performing focus detection for each of a plurality of focus detection areas provided in the observation plane. A device provided with a plurality of first determination ranges including the focus detection area and a second determination range at a boundary between the plurality of first determination ranges, wherein a first gaze detection operation by the gaze detection unit is performed. If the first gaze detection position detected by the first gaze detection position is not included in the second determination range but is included in any of the first determination ranges, the plurality of focal points are determined based on the first gaze detection position. If at least one focus detection area is selected from the detection areas, and the first line-of-sight position is included in the second determination range, a line-of-sight detection is performed again to detect a second line-of-sight detection position, First and second A means for obtaining an average of the line-of-sight detection positions, selecting means for selecting at least one focus detection area among the plurality of focus detection areas based on the average, and obtaining the focus detection information in the focus detection area selected by the selection means. A device with a line-of-sight detection function, comprising: a focus adjustment unit that is obtained from the focus detection unit and adjusts the focus of the optical system. 5. A camera comprising the gaze detection device according to claim 1 or the gaze detection function-equipped device according to any one of claims 2 to 4.