JPH06138373A - Sight line detecting device - Google Patents

Abstract

PURPOSE:To effectively eliminate a false pupil edge, and to improve accuracy for detecting the line of sight by estimating a pupil circle by using the least square method based on extracted all pupil edge data. CONSTITUTION:Extracted plural pupil edges are divided into the upper side part 71, the lower side part 72, the right side part 73 and the left side part 74 of a pupil circle, and based on data of the remaining sides except edge data of one side, a pupil circle 75' is estimated, and simultaneously, an estimated deviation quantity is also calculated. That is, for instance, in the case an edge of the lower side of the pupil circle is extracted erroneously due to influence of an eyelid, when the pupil circle is calculated by using extracted all edge data, an erroneous pupil circle 75 is obtained. Therefore, by estimating the circle by the data from which the lower side edge in an enclosure 72 is eliminated, a correct pupil circle 75' can be detected. In this case, since the circle is estimated by using the minimum square method, the radius or the diameter of the circle is used for the correction.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an axis of a gazing point observed by an observer (photographer) through a finder system on an observation plane (focus plane) on which a subject image is formed by the photographing system, The present invention relates to a line-of-sight detection device which detects a so-called line-of-sight (visual axis) by utilizing a reflected image of the eyeball obtained when the observer illuminates the spherical surface of the eye and performs various photographing operations.

[0002]

2. Description of the Related Art Conventionally, various line-of-sight detection devices (for example, eye cameras) for detecting what position on an observation plane an observer observes, that is, for detecting a so-called line of sight (visual axis) have been proposed.

For example, in Japanese Unexamined Patent Publication No. 1-274736, a parallel light flux from a light source is projected onto the anterior segment of the eyeball of an observer, and a corneal reflection image by the reflected light from the cornea and an image forming position of the pupil are used. Seeking the visual axis.

10 and 11 are explanatory views of the principle of the visual axis detection method.

First, referring to FIG. 11, each infrared light emitting diode (hereinafter referred to as "IRED") 13a, 1
3b are arranged substantially symmetrically in the x direction with respect to the optical axis A of the light receiving lens 12, and divergently illuminate the eyeball of the photographer.

The infrared light emitted from the IREDs 13a, 13 illuminates the cornea 16 of the eyeball 15, and at this time, the corneal reflection images d, e due to a part of the infrared light reflected on the surface of the cornea 16 are the light receiving lenses. The light is collected from the image sensor 12 and re-imaged at the positions d ′ and e ′ on the image sensor 14.

An image of the pupil of the eyeball illuminated by the IRED is also formed on the image sensor 14. The center C of the circle formed by the boundary between the pupil and the iris (this is called the pupil circle)
When the x coordinate of x is xc, x on the image sensor
The coordinates are xc '(not shown).

FIG. 10A shows the image sensor 1 of FIG.
The eyeball images projected on the four planes are shown, and (b) shows the output of the image signal at the line (I)-(I) ′ in (a).

In FIG. 10A, 50 is a so-called white part of the eyeball, 51 is a pupil part, and 52a and 52b are a pair of IRs.
The corneal reflection image of the ED is shown.

This corneal reflection image is called a "Purkinje image" and will be abbreviated as "P image" hereinafter.

Two maximum points in the signal 60 of FIG. 10B correspond to a pair of P images.

Returning to FIG. 11, the x coordinate of the center of the corneal reflection images (P images) d and e and the x coordinate x of the center of curvature of the cornea 16.
_Since it coincides with _O , x at the generation position d, e of the corneal reflection image
The coordinates are xd and xe, and the pupil 19 from the center of curvature O of the cornea 16.
The standard distance to the center C of the L _OC, the distance L rotation angle of the a consideration factor individual differences and A1 eyeball optical axis v for _{OC θ (A1 * L OC)} * sinθ ≒ xc- (xd + xe ) / 2 ... (1) The relational expression is substantially satisfied. Therefore, the rotation angle θ of the optical axis a of the eyeball is obtained by detecting the positions of the respective feature points (corneal reflection images d and e and the center C of the pupil) projected on a part of the image sensor in the visual line arithmetic processing device. be able to. At this time, the expression (1) can be rewritten as β (A1 * L _OC ) * sin θ≈xc ′ − (xd ′ + xe ′) / 2 (2). However, β is a magnification determined by the position of the eyeball with respect to the light receiving lens 12, and is substantially obtained as a function of the interval | xd′−xe ′ | of corneal reflection images. The rotation angle θ of the eyeball 15 can be rewritten as θ≈ARCSIN {(xc′−xf ′) / β / (A1 * L _OC )} (3). However, xf'≈ (xd '+ xe') / 2. By the way, since the optical axis a of the photographer's eye does not coincide with the visual axis, the horizontal rotation angle θ of the optical axis a of the photographer's eye
When is calculated, the angle of view δ between the optical axis of the eyeball and the visual axis is corrected to obtain the horizontal line of sight θH of the photographer.
Letting B1 be a coefficient considering the individual difference with respect to the correction angle δ between the optical axis a of the eyeball and the visual axis, the horizontal line of sight θH of the photographer
Is calculated as θH = θ ± (B1 * δ) (4). Here, if the angle of rotation to the right of the photographer is positive, the sign ± is selected as + when the eye of the photographer looking through the observation device is the left eye, and when the eye of the photographer is the right eye.

Further, in the figure, the eyeball of the photographer is Z-X.
It shows an example of rotation in a plane (eg horizontal plane),
The same can be detected when the photographer's eyeball rotates in the Z-Y plane (for example, a vertical plane). However, since the vertical component of the line of sight of the photographer coincides with the vertical component θ ′ of the optical axis of the eyeball, the vertical line of sight θV is θV = θ ′.
Becomes Further, from the line-of-sight data θH, θV, the position (Xn, Y
n) is calculated as Xn≈m * θH≈m * [ARCSIN {(xc′−xf ′) / β / (A1 * L _OC )} ± (B1 * δ)] (5) Yn≈m * θV . However, m is a constant determined by the finder optical system of the camera.

Here, the values of the coefficients A1 and B1 for correcting the individual difference of the eyeball of the photographer have the photographer fixate the index arranged at a predetermined position in the viewfinder of the camera, and the position of the index. And the position of the fixation point calculated according to the equation (5).

The calculation for obtaining the line of sight and gazing point of the photographer is
It is executed by the software of the microcomputer of the line-of-sight calculation processing device based on the above equations.

The coefficient for correcting the individual difference of the line of sight is obtained, and the position of the line of sight of the observer looking into the viewfinder of the camera is calculated by using the formula (5), and the line of sight information is used for the focus adjustment or exposure of the photographing lens. It is used for control.

In order to actually obtain the line of sight, the eyeball image on the image sensor is processed by a microcomputer or the like to detect the P image and the pupil circle, and the line of sight is calculated based on the position information.

As a concrete method, the present applicant has already proposed (Japanese Patent Application No. 3-121097). According to this method, as a method of obtaining the pupil circle, the luminance difference at the boundary between the pupil and the iris is extracted as a signal edge while reading the eyeball image signal from the image sensor, and the coordinates thereof are stored. Then, when the reading of the eyeball image is completed, a circle is estimated by using the least-squares method for the stored edge coordinates of the plurality of pupils, and this is set as a pupil circle.

Referring to FIG. 12, it will be described with reference to FIG.
Represents an eyeball image, and the P image is omitted here. A plurality of white circles arranged around the pupil part 51 are pupil edges, and 70-1 represents one of them.

In FIG. 5, (b) shows only the pupil edge of (a) extracted and the points inside the enclosure 71 are extracted as the upper edge of the pupil circle. Enclosures 72, 73,
74 are edges relating to the lower side, the left side, and the right side, respectively.

A minimum of 2 based on these edge data
The circle estimated using multiplication is 75. When the center coordinates of this estimated circle are (xc, yc) and the radius is rc, the result is as shown in (c) of FIG.

[0022]

However, as described in the above-mentioned application (Japanese Patent Application No. 3-121097), the method of estimating the pupil circle using the above least square method is a correct pupil circle. If there is false edge data at a position distant from the position, there is a drawback that a correct circle cannot be estimated, and in the same application, in order to improve it, it is calculated in the process of least squares calculation. The estimated error amount is used as the evaluation scale to eliminate false edge data as much as possible.

However, the above-mentioned estimation error amount is the same when the number of pupil edges is the same, and false edges are also present at positions that are distant from the true circle to the same extent, that is, in such a situation, the estimation error amount is similar. Even when it is expected to be calculated, generally, when the size of the pupil (pupil diameter) increases, the estimation error amount also tends to be calculated large. Therefore, it is also inconvenient from the viewpoint of effectively eliminating false edges, and from the viewpoint of determining the quality of the final estimation result.

An object of the present invention is to solve such a conventional problem.

[0025]

The structure for achieving the object of the present invention is as set forth in the claims. Specifically, as described below, there are obstacles for accurate estimation of the pupil circle. It is intended to eliminate only false false pupil edges.

(1): The pupil circle is estimated using the least squares method based on all the extracted pupil edge data, and if the estimation error amount of the resulting estimated circle is smaller than a predetermined value, it is correct. Assuming that the pupil circle has been estimated, the pupil circle is detected successfully.

(2): The plurality of extracted pupil edges are divided into an upper side portion, a lower side portion, a right side portion, and a left side portion of the pupil circle, and the pupil is based on the data of the remaining side excluding the edge data of any one side. The circle is estimated and the estimation error amount is calculated at the same time.

At this time, since the pupil edge extraction processing is time-series signal processing in the horizontal direction, the right edge data and the left edge data should be more reliable than the upper edge data and the lower edge data. From the viewpoint, the combination excluding the upper side or the lower side, that is, (left side, right side, lower side),
Weighting is performed so that the calculation error amount at the time of circle estimation by the combination of (left side, right side, upper side) is smaller than the actual amount.

(3): The smallest combination is selected from the four sets of estimation error amounts calculated in (2).

(4): To the combination selected in (3), the edge data of the removed sides are added one by one, and the change in the estimated error amount is examined. If it does not increase, the added edge data is added as it is as true edge data, and if it exceeds a predetermined value, a false edge is added and the edge data is removed again.

The above trial and error operation is repeated for all the edge data of the removed edges.

When the estimation error amount of the resulting estimated circle of the pupil is smaller than the predetermined value, it is considered that the correct pupil circle has been estimated, and the pupil circle detection is successful.

If the trial and error is larger than the predetermined value, it is assumed that a false edge still remains, and the process (5) described below is executed.

(5): In the future, the pupil edges forming the current estimated circle will be removed one by one. Specifically, remove each edge one by one, check the change in the estimated circle, and if the error amount decreases below a predetermined rate, the removed edge data is assumed to be false edge data, and it is removed as it is. .

On the contrary, when the decrease in the error amount is small or increases, it is determined that the true edge has been removed, and the edge is restored.

The above operation is repeated for all the edge data remaining until the operation of (4), and if the final estimated error amount is smaller than the predetermined value, the pupil circle detection is successful.

The present invention is intended to perform accurate pupil circle estimation by performing the above-described processing.

[0038]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a schematic view of the essential parts of an embodiment in which the present invention is applied to a single-lens reflex camera.

In the figure, reference numeral 1 denotes a taking lens, which is shown as two lenses for convenience, but in reality it is composed of a larger number of lenses. Reference numeral 2 denotes a main mirror, which is obliquely installed or retreated in the photographing optical path according to the observation state of the subject image by the finder system and the photographing state of the subject image. Reference numeral 3 denotes a sub-mirror, which reflects the light flux transmitted through the main mirror 2 toward a focus detection device 6 described below below the camera body.

Reference numeral 4 denotes a shutter, and 5 denotes a photosensitive member, which comprises a silver salt film, a solid-state image pickup device such as CCD or MOS type, or an image pickup tube such as a vidicon.

Reference numeral 6 denotes a focus detection device, which is a field lens 6a and reflection mirrors 6b and 6 arranged near the image plane.
c, a secondary imaging lens 6d, a diaphragm 6e, a line sensor 6f including a plurality of CCDs, and the like.

The focus detection device 6 in this embodiment uses a well-known phase difference method. 7 is a focusing plate disposed on the planned image forming surface of the taking lens 1, 8 is a penta roof prism for changing the optical path of the finder, and 9 and 10 are an image forming lens and a photometric sensor for measuring the subject brightness in the observation screen. is there. The imaging lens 9 conjugately connects the focusing plate 7 and the photometric sensor 10 via the reflection optical path in the penta roof prism 8.

Next, behind the exit surface of the penta roof prism 8, an eyepiece lens 11 having a light splitter 11a is arranged and used for observing the focusing plate 7 by the eye 15 of the photographer. The light splitter 11a includes, for example, a dichroic mirror that transmits visible light and reflects infrared light.

Reference numeral 12 is a light receiving lens, and 14 is an image sensor in which photoelectric element arrays such as CCDs are two-dimensionally arranged so as to be conjugated with the vicinity of the pupil of the photographer's eye 15 at a predetermined position with respect to the light receiving lens 12. ing. 13a to 13f
Are infrared light emitting diodes which are illumination light sources.

Reference numeral 21 denotes a high-intensity superimposing LED that can be visually recognized even in a bright subject. The emitted light is reflected by the main mirror 2 through the projection prism 22 and is provided on the display portion of the focus plate 7. It is bent in the vertical direction by the micro prism array 7a, passes through the penta prism 8 and the eyepiece lens 11 and reaches the photographer's eye 15.

Therefore, the minute prism array 7a is provided at a plurality of positions (distance measuring points) corresponding to the focus detection area of the focusing plate 7.
Are formed in a frame shape, and five superimposing LEDs 21 (each of which are LED-L1 and LED
-L2, LED-C, LED-R1, LED-R2).

Reference numeral 23 is a visual field mask for forming a finder visual field area. Reference numeral 24 denotes an LCD in the finder for displaying photographing information outside the finder field of view.
-LED) 25.

The light transmitted through the LCD 24 is the triangular prism 2
It is guided into the viewfinder field by 6 and displayed outside the viewfinder field so that the photographer can know the shooting information.

Reference numeral 31 denotes an aperture provided in the taking lens 1, 32
Is a diaphragm driving device including a diaphragm driving circuit 111 described later, 3
Reference numeral 3 is a lens driving motor, 34 is a lens driving member including a driving gear, and 35 is a photocoupler which detects rotation of the pulse plate 36 interlocking with the lens driving member 34 and transmits it to the lens focus adjusting circuit 110. Focus adjustment circuit 110
On the basis of this information and the lens driving amount from the camera side, the lens driving motor is driven by a predetermined amount to move the taking lens 1 to the in-focus position. 37
Is a mount contact that serves as an interface between a known camera and a lens.

FIG. 2 is an explanatory diagram of an electric circuit diagram incorporated in the camera of the present embodiment. The same parts as those in FIG. 1 are designated by the same reference numerals.

A central processing unit (hereinafter referred to as a CPU) 100 of a microcomputer built in the camera body includes a visual axis detection circuit 101, a photometric circuit 102, and an automatic focus detection circuit 10.
3, signal input circuit 104, LCD drive circuit 105, LE
The D drive circuit 106, the IRED drive circuit 107, the shutter control circuit 108, and the motor control circuit 109 are connected. Further, a focus adjustment circuit 1 arranged in the photographing lens
10. Signals are transmitted to and from the diaphragm drive circuit 111 via the mount contact 37 shown in FIG.

EEPROM 10 associated with CPU 100
Reference numeral 0a has a function of storing the line-of-sight correction data as a storage unit for correcting individual differences in line-of-sight.

The line-of-sight detection circuit 101 is an image sensor
The output of the eyeball image from the CCD 14 (CCD-EYE) is A / D converted, and this image information is transmitted to the CPU 100. CPU1
As will be described later, 00 extracts each feature point of the eyeball image required for sight line detection according to a predetermined algorithm, and further calculates the photographer's sight line from the position of each feature point.

The photometric circuit 102 amplifies the output from the photometric sensor 10, performs logarithmic compression and A / D conversion, and sends it to the CPU 100 as brightness information of each sensor. The photometric sensor 10 in the present embodiment is an SPC-L for photometrically measuring four areas,
The photodiode is composed of SPC-C, SPC-R, and SPC-A.

The line sensor 6f in FIG. 2 is composed of five sets of line sensor CCD-L corresponding to five distance measuring points on the screen.
2, CCD-L1, CCD-C, CCD-R1, CCD
It is a known CCD line sensor composed of -R2.

The automatic focus detection circuit 103 A / D-converts the voltage obtained from these line sensors 6f, and the CPU 10
Send to 0. SW-1 is a switch that is turned on by the first stroke of the release button 41 to start photometry, AF, line-of-sight detection operations, etc. SW-2 is turned on by the second stroke of the release button 41.
Release switch for N, SW-AEL for AE lock switch that turns on by pressing AE lock button 43,
SW-DIAL1 and SW-DIAL2 are signal input circuits 1 by a dial switch provided in an electronic dial (not shown).
It is input to the 04 up / down counter and counts the amount of rotation clicks of the electronic dial.

Reference numeral 105 denotes a known LCD drive circuit for driving the liquid crystal display device LCD to display the aperture value, the shutter speed, the set photographing mode, etc. according to a signal from the CPU 100 on the monitor LCD 42 and the viewfinder LCD 24. Both can be displayed at the same time. L
The ED drive circuit 106 is a lighting LED (F-LED) 25.
Then, the superimposing LED 21 is turned on and controlled to blink. The IRED drive circuit 107 is an infrared light emitting diode (I
RED1 to 6) 13a to 13f are selectively turned on according to the situation.

When the shutter control circuit 108 is energized, it controls the magnet MG-1 for moving the front curtain and the magnet MG-2 for moving the rear curtain to expose the photosensitive member with a predetermined amount of light. The motor control circuit 109 is for controlling the motor M1 for winding and rewinding the film and the motor M2 for charging the main mirror 2 and the shutter 4. These shutter control circuits 108,
A series of camera release sequences operate by the motor control circuit 109.

Next, a flow chart of the operation of the camera having the visual axis detecting device is shown in FIG. 3 and will be described below based on these.

When the mode dial (not shown) is rotated to set the camera from the inoperative state to the predetermined photographing mode, the power of the camera is turned on (# 100) and the variables used by the CPU 100 for detecting the line of sight are reset (# 101).

Then, the camera waits until the release button 41 is pressed and the switch SW1 is turned on (# 10).
2). The release button 41 is pressed and the switch SW1 is turned off.
When the signal input circuit 104 detects that N has been input, CP
U100 confirms with the line-of-sight detection circuit 101 (# 10
3).

At this time, if the line-of-sight prohibited mode is set, a specific range-finding point is selected by the range-finding point automatic selection subroutine (# 116) without executing the line-of-sight detection, that is, without using the line-of-sight information. At this focus detection point, the automatic focus detection circuit 103 performs focus detection operation. (# 107).

In this way, there are a photographing mode in which the distance measuring point is selected without using the line-of-sight information (line-of-sight prohibited automatic focus photographing mode) and a photographing mode in which the distance measuring point is selected using the line-of-sight information (line-of-sight automatic focus photographing mode). Both of them are provided so that the photographer can arbitrarily select whether to set the line-of-sight prohibition mode.

Although several methods can be considered as the algorithm for automatically selecting the distance measuring points, the near point priority algorithm in which the central distance measuring point is weighted is effective and is not directly related to the present invention. Therefore, the description is omitted.

When the line-of-sight detection mode is set, line-of-sight detection is executed (# 104). Here, the line of sight detected by the line-of-sight detection circuit 101 is converted into the gazing point coordinates on the focus plate 7. The CPU 100 selects a distance measuring point close to the gazing point coordinates, sends a signal to the LED drive circuit 106, and causes the corresponding distance measuring point mark to blink using the superimposing LED 21 (# 105).

When the photographer sees that the distance measuring point selected by the photographer's line of sight is displayed, he recognizes that the distance measuring point is not correct, and releases the release button 41 to release the switch SW.
When 1 is turned off (# 106), the camera switches SW
It waits until 1 is turned on (# 102).

As described above, the fact that the distance measuring point is selected according to the line-of-sight information is displayed to the photographer by blinking the distance measuring point mark in the field of view of the finder so that the photographer can arbitrarily select the distance measuring point. You can check whether or not.

Further, seeing that the distance measuring point selected by the photographer is displayed by the photographer, the switch SW1 is continuously turned on.
If continued (# 106), the automatic focus detection circuit 103
Performs focus detection of one or more focus detection points using the detected line-of-sight information (# 107).

It is determined whether or not the distance measuring point selected here cannot be measured (# 108). If the distance measuring point is impossible, the CPU 100 sends a signal to the LCD drive circuit 105 to set L in the finder.
The focus mark on the CD24 blinks, and distance measurement is NG (impossible)
That is, the photographer is warned (# 118) and continues until SW1 is released (# 119).

Distance measurement is possible, and the focus adjustment state of the distance measurement point selected by the predetermined algorithm is not in focus (#
109), the CPU 100 sends a signal to the lens focus adjustment circuit 110 to drive the taking lens 1 by a predetermined amount (# 11).
7). After driving the lens, the automatic focus detection circuit 103 performs focus detection again (# 107), and determines whether or not the taking lens 1 is in focus (# 109).

If the taking lens 1 is in focus at a predetermined distance measuring point, the CPU 100 causes the LCD drive circuit 105
Is sent to turn on the focus mark on the LCD 24 in the finder, and also the signal is sent to the LED drive circuit 106 to display the focus on the focusing point 201 in focus (# 110).

At this time, the blinking display of the distance measuring point selected by the line of sight is turned off, but the distance measuring point displayed in focus is often coincident with the distance measuring point selected by the line of sight. The focus range-finding point is set to a lighting state so that the photographer can recognize that the focus has been achieved. When the photographer sees the focusing point displayed in the viewfinder and recognizes that the focusing point is not correct, he releases the release button 41 and turns off the switch SW1 (# 111). The camera waits until the switch SW1 is turned on (# 10
2).

If the photographer looks at the focus detection point and continues to turn on the switch SW1 (# 1
11), the CPU 100 sends a signal to the photometry circuit 102 to perform photometry (# 112).

Further, it is judged whether or not the release button 41 is pushed and the switch SW2 is turned on (#
113), if the switch SW2 is OFF, the state of the switch SW1 is checked again (# 111). or,
When the switch SW2 is turned on, the CPU 100 sends signals to the shutter control circuit 108, the motor control circuit 109, and the diaphragm drive circuit 111, respectively.

First, M2 is energized to raise the main mirror 2 and the diaphragm 31 is narrowed down. Then, MG1 is energized to open the front curtain of the shutter 4. The aperture value of the aperture 31 and the shutter speed of the shutter 4 are determined from the exposure value detected by the photometric circuit 102 and the sensitivity of the film 5.
M after a predetermined shutter time (1/250 second, for example)
Energize G2 and close the rear curtain of the shutter 4. When the exposure of the film 5 is completed, the M2 is energized again, the mirror is down and the shutter is charged, and the M1 is energized to advance the film frame, and the series of shutter release sequence operations is completed (# 114). . After that, the camera waits until the switch SW1 is turned on again (# 102).

4 to 9 are flow charts of the above-mentioned line-of-sight detection.

As described above, the line-of-sight detection circuit 101 is a CPU
When a signal is received from 100, the gaze detection is executed (Fig. 3
# 104).

First, the CPU 100 is an infrared light emitting diode (IRED) 13a to 13f for illuminating the photographer's eyes.
Select an appropriate combination of IREDs from among the above and light them up (# 201). The selection of IRED is made depending on whether the camera is in the horizontal position or the vertical position by an unillustrated posture switch, or whether the photographer wears glasses.

Next, charges are accumulated in the image sensor 14 for a predetermined accumulation time (# 202). When the storage is completed, the IRED is turned off together with it (# 203).

The CPU 100 reads out the eyeball image of the photographer from the image sensor 14 whose accumulation has been completed, and at the same time, sequentially performs the process of extracting the features of the P image and the pupil portion (# 20).
4). The specific method is described in detail in the above-mentioned application by the present applicant (Japanese Patent Application No. 3-121097), and thus detailed description thereof is omitted here.

Now that the reading of the entire eyeball image is complete, P
After the feature extraction of the image and the pupil is completed, a set of P image positions is detected based on the information (# 205). As described above, since the P image is a corneal reflection image of the IRED for eyeball illumination, it appears as a bright spot with a strong light intensity in the image signal like 52a and 52b in FIG. A set of P images is detected with a feature and its position (xd ', y
d '), (xe', ye ') can be obtained.

Next, from the coordinate information of the pupil edge extracted in the sequential processing step of # 204, the center of the pupil circle (xc ', y
c ') and the radius re are detected (# 206). This subroutine will be described in detail later.

If the P image position and the pupil position can be detected from the eyeball image of the photographer, the photographer's line-of-sight direction or the coordinates on the finder can be calculated from equation (5) in step # 207. The detection subroutine is returned (# 208).

FIG. 5 shows the flow of the pupil center / pupil diameter detection subroutine. When the subroutine is called,
In step # 301 through # 300, least square method estimation of a circle is performed using all the extracted pupil edge data. The basic calculation formula is applied by the applicant described above (Japanese Patent Application No. 3-
No. 121097), which will be described again here.

The coordinates of the n pupil edges are (x ₁ , y
₁ ), (x ₂ , y ₂ ), ..., (x _n , y _n ),
The center coordinates (xc, yc) of the estimated circle using the least-squares method based on these data, the radius rc, and the estimated error amount ER are xc = (W1 · V2-W2 · W4- (W6-Y1 · Z1) · W3 ) / 2 * (X2 * V2-W5-W6 * X1 / n) ... (10) yc = (W2 * V1-W1 * W4- (W7-X1 * Z1) * W3) / 2 * (Y2 * V1-) W5-W7 · Y1 / n) (11) rc = √ (W3-2 · ((xc · X1 + yc · Y1) / n) + xc ² + yc ² ) (12) ERX = X4-4 · xc · X3 + 2 ( ^{2 · xc 2 + d) ·} X2 -4 · xc · d · X1 + Y4-4 · yc · Y3 +2 (2 · yc 2 + d) · Y2-4 · yc · d · Y1 +2 (Z4-2 · xc · Z3- 2 · yc · Z2 +4 · xc · yc · Z1) + d 2 · n ... (13) ER = √ (ERX / n ... (14) where, X1 = Σxi, X2 = Σxi 2, X3 = Σxi 3, X4 = Σxi 4 ... (15) ~ (18) Y1 = Σyi, Y2 = Σyi 2, Y3 = Σyi 3, Y4 = Σyi 4 (19) to (22) Z1 = Σxi · yi, Z2 = Σxi ² · yi ² (23), (24) Z3 = Σxi · yi ² , Z4 = xi ² · yi ² (25), (26) ) Further, V1 = X2-X1 ² / n (27) V2 = Y2-Y1 ² / n (28) W1 = X3 + Y3 (29) W2 = Y3 + Z3 (30) W3 = (X2 + Y2) / n ((30)) 31) W4 = Z1-X1 · Y1 / n (32) W5 = (Z1-2 · X1 · Y1 / n) · Z1 (33) W6 = X1 · Y2 (34) W7 = X2 · Y1 ((34) By performing the ^{35) d = xc 2 + yc} 2 -rc 2 ... (36) above numerical calculations, The center (xc, yc), can be obtained radius rc and least squares estimation error amount ER.

Here, the estimated error amount ER, which is acceptable in the mathematical expression, is actually corrected by the pupil diameter rc. The error amount ER is zero when all the edge data is accurately located on the circumference of a circle, and as the number of edges deviated from the circumference increases, the distance from the circumference also increases. The larger the error amount is. However, even when there are the same number of edges and the same number of edges distant from the true circle, the error amount ER generally tends to increase as the pupil diameter increases.

In order to calibrate this, the error amount ER is corrected using the pupil diameter rc as in the following equation.

ER = ER · (a · rc + b) (37) Here, for example, if a has a value of −0.05 and b has a value of about 1.5, the pupil diameter on the image sensor is 10 pixels. , The correction coefficient a · rc + b becomes exactly 1, and ER
Is not corrected. When the photographer's eyes approach the camera or the pupil opens, and the pupil diameter rc increases, for example, when rc = 20, the correction coefficient a · rc + b
Will be 0.5, and the error amount ER will be corrected by half. On the contrary, if the pupil diameter becomes smaller, a correction for increasing the error amount will be applied. Of course, with the values of a and b, when the pupil diameter rc becomes 30 pixels, the correction coefficient a · r
Although c + b becomes 0, the coefficient may be determined to be 0.5 when rc is 20 pixels or more.

FIG. 7A shows the flow of the circle least squares estimation subroutine.

When the subroutine is called, the intermediate variables X1 to X4, Y1 to Y4, and Z1 to Z4 of the least squares method are expressed by the formula (1) in step # 400 through step # 400.
5) to (18), formulas (19 to 22), formulas (23) to (2)
Calculate according to 6). Next, the estimated circle is calculated (# 4
02) and returns (# 403).

FIG. 7B is a subroutine flow for calculating the estimated circle. Step # 50 through Step # 500
In the first example, the variables V1 to W7 in the middle of the calculation are expressed by the equations (27) to (3).
5), and then using these variables, the center coordinates xc and yc of the estimated circle are calculated using equations (10) and (11), respectively.
Calculate with. The radius rc of the estimated circle is the already calculated center x
It is calculated according to the equation (12) using the values of c and yc.

Next, the estimated error amount ER is obtained according to the equations (36), (13) and (14) (# 502), and the error amount ER is corrected by the pupil diameter rc according to the equation (37) (#). 503).

In this way, the center of the estimated circle (xc, y
c), the radius rc, and the estimated error amount ER are obtained, and the subroutine is returned (# 504).

As described above, in step # 301, the pupil circle is estimated based on all the extracted pupil edge data, and the center coordinates (xc, yc) of the pupil circle, the radius rc, and the estimated error amount ER. Then, this ER is compared with the error amount threshold value ERTHR (# 302).

Since the estimation error amount ER when the pupil circle is estimated with sufficient accuracy is usually 10 or less, the threshold value E
RTHR is set to about 15 to 20, for example.

If ER <ERTHR, that is, if the estimation error is small, it is considered that the pupil detection is successful (#
303) and returns from this subroutine (# 30).
4).

If ER ≧ ERTHR, that is, if the estimation error is large, it is determined that the estimation has failed due to the false edge data, and the false edges are eliminated in the subsequent processing.

First, in step # 305, among all the extracted pupil edge data, the minimum except for the edge data extracted as the lower side of the pupil circle, that is, based on the edge data of the upper side, the right side, and the left side, a minimum of 2 is obtained. Let's try to estimate the pupil circle using the multiplication method. The calculation here is the same as the above-described numerical calculation except that the number of edges is small. The center coordinate diameter of the pupil circle and the estimation error obtained as a result are stored.

Similarly, in step # 306, the upper edge is removed, the right edge is removed in step # 307, and the left edge is removed in step # 308, and the pupil circle is estimated and the results are stored. .

FIG. 8A is a flow of a subroutine for estimating the least squares of a circle excluding the lower edge.

First, in step # 601, only the edge data extracted as the pupil edge of the lower side of the pupil circle is used to calculate equations (15) to (18), (19) to (22), (23) to.
According to (26), X1 'to X4', Y1 'to Y4', Z
Calculate 1'-Z4 '.

Next, the intermediate variables X1 to X4 calculated with all the edge data are corrected by the following equation.

X1 ← X1-X1 '(38) X2 ← X2-X2' (39) X3 ← X3-X3 '(40) X4 ← X4-X4' (41) Variables X1 to X4 are expressions ( Since it is a linear arithmetic expression as is clear from (15) to (18), the above expressions (38) to (4)
When 1) is calculated, as a result, X1 to X4 become variables by edges excluding the lower edge. The purpose of shortening the calculation is not to calculate X1 to X4 again but to calculate as described above.

Intermediate variables Y1 to Y4 and Z1 to Z4 are similarly corrected (# 602). If the estimated circle is calculated after this (# 603), the result becomes the estimation result by the edges excluding the lower edge.

The estimation of the pupil circle excluding the upper side edge, the estimation of the pupil circle excluding the left side edge, and the estimation of the pupil circle excluding the right side edge are the same as the case of the lower side except that the target side is different. Therefore, the flow chart and description will be omitted.

Returning to FIG. 5, when the estimation of the pupil circle excluding the edge data of the lower side, the upper side, the right side, and the left side is completed, in step # 309, the best result is searched for. At this time, the left side rather than the edge data of the upper side and the lower side,
The selection is performed so that the edge data on the right side has priority.

This is because the pupil edge extraction processing is horizontal time-series signal processing, so that the edge data on the left side and the right side are inevitably more reliable.

Specifically, the estimated error amount ER obtained in steps # 305 and 306 is weighted so as to be smaller than the actual value, and the result of the smallest ER among the four ERs is selected.

Here, the description will be continued assuming that the result excluding the lower edge is the best.

In the next step # 310, the recalculation 1 of the least square estimation of the circle is performed.

In summary, by adding the edge data of the removed sides one by one to the previously selected calculation result, the change in the estimated error amount is examined, and the error amount is determined to be a predetermined ratio. If it does not increase, the added edge data is regarded as a correct pupil edge and is added as it is. If not, false edge data is regarded as added and the data is removed. The above trial-and-error operation is repeated for all the edge data of the edges removed from the beginning.

FIG. 8B shows a flow of the least square estimation recalculation 1 of the circle.

Since it is assumed that the estimation result in which the lower edge is removed is selected, the process proceeds from step # 701 to step # 800.

At the next step # 801, data (x coordinate, y coordinate) of one of the lower side edge groups is converted into (xk, y).
k), the least square method intermediate variables X1 to X are calculated by the following equation.
Correction of 4 is performed.

X1 ← X1 + xk (42) X2 ← X2 + xk ² (43) X3 ← X3 + xk ³ (44) X4 ← X4 + xk ⁴ (45) The variables Y1 to Y4 and Z1 to Z4 are similarly calculated (#). 801), in short, the data of the operation of the least squares method is 1
It is the same as increased.

Then, the estimated circle is calculated again using the corrected intermediate variables (# 802). Next, the ratio between the new estimated error amount ER and the error amount ER ₀ before adding one data is obtained and used as the change rate of ER, and the increase rate is small, that is, (ER / ER ₀ ) <Ca If (46), the added edge is regarded as a true edge. Ca is, for example, about 1.1.

When the rate of increase is large, the added edge is regarded as a false edge and the intermediate variables X1 to X4.
The results of Y1-Y4, Z1 to Z4 and the estimated circle are returned to the values before adding this data.

After the trial for one edge is finished, the next edge is tried. When the trials for all the edge data that have been initially removed have been completed (# 805), it is determined that the correct edge among the lower edge data that has been uniformly removed was effectively used for the estimation calculation, and this subroutine is returned ( # 806).

The above is the recalculation 1 for the lower edge.
However, since the same applies to the upper side, the left side, and the right side, their flow chart and description will be omitted.

When the correct pupil edge data collection is completed by the recalculation 1 of the least square estimation of the circle, the estimated error amount ER of the final result is compared again with the error threshold ERTHR (# 311), and if ER <ERTHR. If the pupil detection is successful (# 303), the subroutine is returned (# 304).

If the estimated error amount is still large, it is considered that the pupil edge currently in use at this time still contains false edge data, and a process for eliminating the false edge data is executed.

Since it is assumed that a combination of (upper side, left side, right side) except for the lower side edge has been selected, the minimum number of circles for the upper side edge is 2 at step # 312, step # 315 and step # 316. The power estimation recalculation 2B is executed. This subroutine is a process for removing false data from the upper pupil edge data currently used at the present time, as described above.

FIG. 9 shows a flowchart of the recalculation 2A of the least squares estimation of the circle on the lower side. Symbols 2A, 2B, 2
C and 2D correspond to the lower side, the upper side, the right side, and the left side, respectively.

Here, only the subroutine relating to the lower side of the recalculation 2A will be described, but the same applies to other cases.

Now, when the same subroutine is called,
1 of the edge data group extracted as the lower edge
Intermediate variables X1 to X4, Y1 to Y4, Z1
~ Z4 is corrected (# 901). If the coordinates of the edge to be removed are (xk, yk), X1 to X4 are corrected as in the following equation.

X1 ← X1-xk (47) X2 ← X2-xk ² (48) X3 ← X3-xk ³ (49) X4 ← X4-xk ⁴ (50) Y1 to Y4, Z1 to Z4 Is also the same.

After the correction of the intermediate variables, the next step #
At 902, the estimated circle is calculated. The center coordinates, diameter, and estimation error amount of the estimated circle obtained as a result are based on the remaining edge data obtained by removing one piece of edge data.

Next, the rate of change of the estimated error amount is calculated by the above equation (46) (# 903), and if the rate of decrease is larger than the predetermined value, it is assumed that the edge removed earlier is false edge data. If there is not much change in the reduction rate, it is assumed that the result is the result of removing the correct edge data, and the intermediate variables X1 to X4 and Y1 to Y are determined in step # 904.
4, Undo the results of Z1 to Z4 and the estimated circle.

Similar to the case of recalculation 1, when the trial for all the relevant edge data is completed (# 905),
This subroutine is returned (# 906).

Returning to FIG. 5, when the recalculation 2 on the upper edge in # 316 is completed, the estimated error amount ER which is the result is compared with the error threshold value ERTHR (# 317). If the error amount is small, Assuming that the pupil detection is successful (# 303), this subroutine is returned (# 3).
04).

If the estimation error amount is still large, the branch T
The process moves to the flow of FIG. 6 via 1 and the same processing is performed on the right side and the left side. If the estimated error amount ER becomes smaller in the middle of the process, the step # in the flow of FIG.
At 303, the pupil detection is successful, and the subroutine is returned (# 304).

Even if the above processing is executed, if the estimated error amount ER is still large, pupil detection fails (# 324),
The subroutine is returned (# 325).

The above process will be described with reference to FIG.
In the example of this eyeball image (FIG. 13A), the edge of the lower side of the pupil circle is erroneously extracted due to the influence of the eyelids. Therefore, if the pupil circle is calculated using all the extracted edge data, an incorrect pupil circle is obtained as indicated by 75 in FIG. 13B.

Therefore, according to the method of the present invention, FIG.
If the circle is estimated from the data obtained by removing the lower edge in the enclosure 72 as shown in (c), a correct pupil circle can be detected as in 75 '.

Although FIG. 13 shows an example of photographing with the eyelids, the influence of the eyelashes will be described with reference to FIG.

Reference numeral 53 in FIG. 14A represents eyelashes, and therefore false edges are included in the edges extracted as the lower pupil side.

Reference numeral 75 in FIG. 13B represents a circle estimated by using all the extracted edges, which is an erroneous estimation.

According to the method of the present invention, the lower edge is also a correct edge (enclosure 72) as shown in FIG.
By selectively separating b) and the false edge (enclosure 72a), the correct estimated circle 75 'can be obtained based on only the correct edge data.

In the above embodiment, a linear equation is used when correcting the estimation error amount using the pupil diameter, but this is not limited to the linear equation, and a higher-order correction equation is used. Is also good.

Further, in the present embodiment, the circle is estimated by using the least squares method, and therefore the radius or diameter of the circle is used for correction. However, for example, it is assumed that the pupil is an ellipse. If it is set to either the major axis or the minor axis,
Alternatively, the average value of both may be used for correction.

[0141]

As described above, according to the present invention,
When the light intensity difference at the boundary between the pupil and the iris is taken as the edge of the signal and the pupil circle is detected based on the edge coordinates,
It is possible to effectively eliminate false eyelid edges such as eyelids and eyelashes that interfere with correct pupil circle detection,
The accuracy of line-of-sight detection can be significantly improved.

[Brief description of drawings]

FIG. 1 is a schematic view of a main part of a camera capable of effectively implementing the present invention.

FIG. 2 is an electric circuit diagram of the camera of FIG.

FIG. 3 is a flowchart showing the operation of FIG.

FIG. 4 is a flowchart of a line-of-sight detection operation.

5 is a flowchart of detection of a pupil center and a pupil diameter in FIG.

FIG. 6 is a flowchart after T _{1 of} FIG.

7 is a flowchart of the least squares estimation operation of the circle in FIG.

FIG. 8A is a least squares estimation of a circle excluding the lower edge, and FIG. 8B is a flowchart of least squares estimation recalculation 1 of the circle.

FIG. 9 is a flowchart of a circle least-squares estimation recalculation 2A.

FIG. 10 is an example of an eyeball image of an observer.

FIG. 11 is an explanatory view of the principle of the eye gaze detection method.

FIG. 12 is an explanatory diagram of normal pupil circle detection.

FIG. 13 is an explanatory diagram of pupil circle detection in the case of being affected by eyelids.

FIG. 14 is an explanatory diagram of pupil circle detection in the case of being affected by eyelashes.

[Explanation of Codes] 1 ... Imaging lens 2 ... Main mirror 6 ... Focus detection device 6f ... Image sensor 7 ... Focus plate 10 ... Photometric sensor 11 ... Eyepiece lens 13 ... Infrared light emitting diode (IRED) 14 ... Image sensor (CCD- EYE) 15 ... Eyeball 16 ... Cornea 17 ... Iris 21 ... Superimpose LED 24 ... Finder LCD 25 ... Lighting LE
D 31 ... Aperture 50 ... Eyeball white eye portion 51 ... Eyeball pupil portion 52a, b ... P image 53 ... Eyelashes 100 ... CPU 101 ... Line-of-sight detection circuit 103 ... Focus detection circuit 104 ... Signal input circuit 105 ... LCD drive circuit 106 ... LED drive circuit 107 ... IRED
Drive circuit 110 ... Focus adjustment circuit 200-204 ...
AF point mark

Claims (4)

[Claims] 1. A light receiving means for receiving an eyeball image, and a processing means for extracting a boundary between a pupil portion and an iris portion in the eyeball image in each direction based on a photoelectric conversion signal output from the light receiving means.
Selecting means for selecting a combination of the boundary position information for each direction extracted by the processing means, and computing means for estimating the shape of the pupil portion based on the plurality of boundary position information in accordance with the selection of the selecting means. A line-of-sight detection device having. 2. The line-of-sight detection apparatus according to claim 1, wherein the selection means performs selection by giving priority to horizontal boundary position information. 3. A light receiving means for receiving an eyeball image, a processing means for extracting a boundary position between a pupil portion and an iris portion in the eyeball image based on a photoelectric conversion signal output from the light receiving means, and the processing means. Based on the extracted plurality of boundary position information, the shape of the pupil part is estimated, and at the same time, the calculating means for calculating the estimated error amount, and whether the plurality of boundary position information are selectively adopted or not according to the estimated error amount. Identifying means for identifying
A line-of-sight detection device comprising: the calculation means and the control means for repeatedly operating the identification means. 4. The line-of-sight detection apparatus according to claim 1, 2 or 3, wherein the calculation means performs estimation calculation using a least square method.