JPH0735966A - Line-of-sight detecting device and camera equipped therewith - Google Patents

Abstract

PURPOSE:To easily, rapidly and accurately correspond to the state of the eyeball of an observer even when the state of the eyeball of the observer is changed because of the alternation of the observer. CONSTITUTION:This line-of-sight detecting device is provided with an observation means 11 observing the reflected image of the eyeball 10 of the observer, a 1st parameter calculation means 12 previously calculating a 1st parameter based on the observed result by the observation means 11 before detecting the line of sight, and a 2nd parameter calculation means 13 calculating a 2nd parameter based on the observed result by the observation means 11 at the time of detecting the line of sight. In the device, a storage means 15 storing plural sets of 1st and 2nd parameters corresponding to plural observers, and a selection means 16 selecting a set of parameters from the plural sets of parameters stored in the storage means 15 based on the 2nd parameter calculated by the calculation means 13 are installed to calculate the line-of-sight direction of the eyeball based on the set of parameters selected by the selection means 16.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight detection device for detecting the direction of a gazing point observed by an observer, that is, the so-called line-of-sight direction on a visual field screen provided in, for example, an optical device, and the same. Regarding the camera.

[0002]

2. Description of the Related Art Conventionally, as a device for detecting the line-of-sight direction of an observer, for example, Japanese Patent Laid-Open No. 61-172552,
There is a device disclosed in Japanese Patent Laid-Open No. 3-168623. The devices disclosed in these publications illuminate an eyeball of an observer with a light source, read a boundary between a pupil and an iris of the eyeball illuminated by the light source to obtain a position of a pupil center, and illuminate light from the light source. The position of the corneal reflection image was obtained by the method, and the line-of-sight direction was obtained from the relative positional relationship between the center of the pupil and the corneal reflection image. The operation of the line-of-sight detection device in the above publication will be described below with reference to FIG.

FIG. 31 is a sectional view of a human eyeball cut horizontally. In FIG. 31, reference numeral 1 is an eyeball, and this eyeball 1 has a substantially spherical sclera 2 filled with a vitreous body 3,
A crystalline lens 4, an iris 5 and a cornea 6 are formed in a front portion (left portion in the drawing) of the sclera 2 to be generally configured. The iris 5 is a kind of diaphragm, and the opening is called a pupil 7. The curvature of the cornea 6 is different than the curvature of the sclera 2. Hereinafter, as shown in the figure, the distance between the corneal curvature center C and the eyeball rotation center O ″ is set as a distance ρ, and the distance between the pupil center D and the eyeball center O ″ is set as a distance A. deep.

As shown in FIG. 31, the X-axis is taken in the horizontal direction (vertical direction in the figure), and the eyeball 1 is placed at the center of the visual field screen (not shown).
O is the origin of the X-axis position of the eyeball rotation center O '' when
And The position of the pupil center D on the X axis and the position of the corneal reflection image P on the X axis at this time can be defined by defining the eyeball rotation angle θ as shown in the figure.

## EQU1 ## D = L + A × sin θ (1) P = L + ρ × sin θ (2) The corneal reflection image P referred to here is called the Purkinje's first image, and is a virtual image of the light rays reflected on the surface of the cornea 6 when the cornea 6 is considered as a convex lens.

From the equations (1) and (2), the translational component L of the eyeball center can be canceled by subtracting the X-axis position of the pupil center D from the X-axis position of the corneal reflection image P.

## EQU00002 ## Since D-P = (A-.rho.). Times.sin .theta. (3) is obtained, the eyeball rotation angle .theta.

## EQU3 ## θ = sin ^-1 ((DP) / (A-ρ)) (4) Here, it can be assumed that the distance A-ρ between the center C of the corneal curvature and the center D of the pupil is a substantially constant value regardless of each observer. Since the line-of-sight direction of the observer is obtained from the eyeball rotation angle θ, the X-axis position of the corneal reflection image P and the X-axis position of the pupil center D are obtained, and an appropriate value is substituted for A-ρ that can be assumed to be a substantially constant value. If you do this, you can find the line of sight.

Further, a more accurate line-of-sight direction can be obtained by initializing a value peculiar to each eyeball (Japanese Patent Laid-Open No. 3-1686).
23). That is, as described above, the distance A-ρ between the corneal curvature center C and the pupil center D is a value having an individual difference, and the line-of-sight direction of the observer is generally the optical axis of the observer (pupil center D And the corneal curvature center C), there is a deviation angle peculiar to each eyeball. Therefore, the eyeball rotation angle θ is corrected by this deviation angle and A-ρ to calculate an accurate line-of-sight direction. To set A-ρ and the shift angle, the observer looks at the index provided in the finder prior to the eye-gaze detecting operation, and the distance A- from the eyeball rotation angle θ at this time.
This is performed by obtaining ρ and the shift angle. At this time, it is also possible to hold the distances A-ρ and the shift angles for a plurality of observers in a non-volatile memory or the like, and to select a more suitable value by a switch or the like during use.

[0007]

However, in the conventional line-of-sight detection device disclosed in the above publication, the distance A-ρ
Since the shift angle and the shift angle must be selected by using a switch or the like when detecting the line of sight, it is necessary to reselect the distance A-ρ and the shift angle by the switch or the like every time the observer or the state of the eyeball of the observer changes. . Therefore, it is necessary to operate a switch or the like every time the observer changes, etc. to select the distance A-ρ and the deviation angle, which is troublesome and quick to deal with, and the observer has a wrong distance. A-ρ,
There is a problem in that a shift angle may be selected.

An object of the present invention is to provide a line-of-sight detection device which can easily, swiftly and accurately respond to a change in the eyeball state of an observer due to the change of the observer or the like.

[0009]

When the invention of claim 1 is explained by associating it with FIG. 1, which is a diagram corresponding to claims, the observation means 11 for observing a reflected image of an eyeball 10 of an observer and prior to the detection of the line of sight. First parameter calculating means 1 for calculating the first parameter in advance based on the observation result of the observing means 11.
2, a second parameter calculating means 13 for calculating a second parameter based on the observation result of the observing means 11 at the time of sight line detection, and an observation result of the observing means 11 and the first and second parameters at the time of sight line detection. It is applied to a line-of-sight detection device provided with a line-of-sight calculation means 14 for calculating the line-of-sight direction of the eyeball based on the above. Then, the above-mentioned object is to store the plurality of sets of the first and second parameters corresponding to the plurality of observers and the second parameter calculated by the second parameter calculation means 13. Based on the set of parameters selected by the selecting means 16, the line-of-sight calculating means 14 is provided. Is achieved by calculating the line-of-sight direction of the eyeball.

On the other hand, according to the invention of claim 2, a storage means 15 in which a plurality of sets of first and second parameters corresponding to a plurality of observers are stored, and a second parameter calculation means 13
Selection means 16 for selecting one first parameter that forms a pair with the second parameter from the plurality of first parameters stored in the storage means 15 based on the second parameter calculated by And the line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball based on the second parameter calculated by the second parameter calculation unit 13 and the first parameter selected by the selection unit 16. You have achieved your purpose.

Further, the invention of claim 3 is the storage means 15 storing a plurality of sets of first and second parameters corresponding to a plurality of observers, and a plurality of first storage means stored in the storage means 15. If some of the two parameters are substantially equal to the second parameter calculated by the second parameter calculating means 13, the selecting means 16 for selecting the second parameter and the first parameter forming a pair with the second parameter. And the line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball based on the set of parameters when the selection unit 16 selects the set of parameters, and the line-of-sight is calculated when the selection unit 16 does not select the set of parameters. The above-described object is achieved by stopping the calculation operation of the eye gaze direction by the calculation means 14.

Further, in the invention of claim 4, a storage means 15 storing a plurality of sets of first and second parameters corresponding to a plurality of observers, and a second storage means stored in the storage means 15. Second parameter calculation means 1 in the parameters
If there is a parameter substantially equal to the second parameter calculated in step 3, the second parameter and the first parameter paired with this second parameter are selected, and the second parameter calculated by the second parameter calculation means 13 is selected. When there is no parameter that is substantially equal, there is provided a selection means 16 for selecting one of the parameter sets stored in the storage means 15, and based on the parameter set selected by the selection means 16. The line-of-sight calculation means 14 achieves the above-mentioned object by calculating the line-of-sight direction of the eyeball.

According to the invention of claim 5, a storage means 15 storing a plurality of sets of first and second parameters corresponding to a plurality of observers, and a second storage means stored in the storage means 15. Second parameter calculation means 1 in the parameters
If there is a parameter substantially equal to the second parameter calculated in step 3, this second parameter and the first parameter paired with this second parameter are selected, and the second parameter calculated by the second parameter calculation means 13 is selected. If none of the parameters is substantially equal, the gaze detection operation is stopped and the first and second
The first and second parameter calculation means 12 and 13 of
And a selecting means 16 for newly calculating the parameter set of and selecting the parameter set, and the line-of-sight calculating means 14 based on the parameter set selected by the selecting means 16.
Achieves the above object by calculating the line-of-sight direction of the eyeball. Here, if there is no substantially equal second parameter calculated by the second parameter calculating means 13, the first and second parameter calculating means 12, 1
It is also possible to store the set of the first and second parameters newly calculated in step 3 in the storage means 15.

The first parameter includes the angle between the line of sight of the eyeball 10 and the optical axis of the eyeball, and the second parameter is
It includes the distance between the center position of the pupil of the eyeball 10 and the center position of the corneal curvature. The line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball based on the set of parameters selected by the selection unit 16 until the reflected image of the eyeball 10 cannot be observed. In addition, the line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball based on the set of parameters selected by the selection unit 16, or the release until the first stroke of the release button 17 provided in the camera is released. The eye gaze direction is calculated based on the set of parameters selected by the selection means 16 for a predetermined time after the release of the first stroke or the second stroke of the button 17. In addition, the line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball based on the set of parameters selected by the selection unit 16 until the main switch 18 provided in the camera is turned off. Further, the line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball based on the set of parameters selected by the selection unit 16 until the eyeball 10 moves away from the viewfinder eyepiece window of the camera.
Alternatively, the line-of-sight direction of the eyeball is calculated based on the set of parameters selected by the selection unit 16 only for a predetermined time after the eyeball 10 has left the viewfinder eyepiece window of the camera.

[0015]

[Action]

-Claims 1 and 2, the selection means 16 stores the storage means 15 based on the second parameter calculated by the second parameter calculation means 13.
One parameter set is selected from among the plurality of parameter sets stored in the storage unit 15 or is paired with the second parameter from the plurality of first parameters stored in the storage unit 15. Select one first parameter.
That is, the selection of the correction value is automatically performed by the selection unit 16 based on the second parameter calculated from the reflected image of the eyeball 10 without being based on a command from the observer. Then, the line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball based on the set of parameters selected by the selection unit 16, or the second parameter and selection unit 16 calculated by the second parameter calculation unit 13. The line-of-sight direction of the eyeball is calculated based on the first parameter selected by.

-Claim 3- The selection means 16 includes a second parameter calculation means 13 among the plurality of second parameters stored in the storage means 15.
If there is a parameter substantially equal to the second parameter calculated by, the second parameter and the first parameter forming a pair with the second parameter are selected. When there is no substantially equal second parameter calculated by the second parameter calculation means 13, an observer other than the plurality of observers corresponding to the plurality of parameter sets stored in the storage means 13 observes. You can judge that you are doing. Therefore, the line-of-sight calculation means 14
When the selection means 16 selects a parameter set, the eye gaze direction is calculated based on this parameter set, and when the selection means 16 does not select a parameter set, the operation of calculating the eye gaze direction is stopped. .

-Claim 4- When the selection means 16 includes one of the second parameters stored in the storage means 15 that is substantially equal to the second parameter calculated by the second parameter calculation means 13. Selects the second parameter and the first parameter forming a pair with the second parameter, and if there is no substantially equal second parameter calculated by the second parameter calculating means 13, it is stored in the storage means 15. Any one of the set of parameters is selected. When there is no substantially equal second parameter calculated by the second parameter calculation means 13, an observer other than the plurality of observers corresponding to the plurality of parameter sets stored in the storage means 13 observes. Therefore, any one of the parameter sets in the storage means is used instead. Then, the line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball based on the set of parameters selected by the selection unit 16.

-Claim 5- When the selecting means 16 has one of the second parameters stored in the storing means 15 that is substantially equal to the second parameter calculated by the second parameter calculating means 13. Selects the second parameter and the first parameter forming a pair with the second parameter, and if there is no substantially equal second parameter calculated by the second parameter calculating means 13, the eye gaze detection operation is stopped. Then, the first and second parameter calculating means 12 and 13 newly calculate the first and second parameter sets, and select the parameter sets. When there is no substantially equal second parameter calculated by the second parameter calculation means 13, an observer other than the plurality of observers corresponding to the plurality of parameter sets stored in the storage means 13 observes. Since it can be determined that it is performing, the first and second parameters are newly calculated. Then, the line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball based on the set of parameters selected by the selection unit 16.

-Claim 9- When the reflected image of the eyeball 10 cannot be observed, it is necessary for the observer to change or the observer to newly use contact, glasses, etc. to change the set of parameters. Can be judged. Therefore, the line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball 10 based on the set of parameters selected by the selection unit 16 while the reflected image of the eyeball 10 is being detected.

-Claims 10 and 11-When the first stroke of the release button 17 of the camera is released, or when a predetermined time has elapsed after the first stroke or the second stroke of the release button 17 was released, an observer , Or the observer may need to change the set of parameters by newly using contacts, glasses, or the like. Therefore, the line-of-sight calculation means 14
Is the parameter selected by the selection means 16 until the first stroke of the release button 17 of the camera is released, or until a predetermined time elapses after the first or second stroke of the release button 17 is released. The line-of-sight direction of the eyeball 10 is calculated based on the set.

-Claim 12- When the main switch 18 of the camera is turned off, it is necessary for the observer to change or the observer to newly use contacts, glasses or the like to change the set of parameters. I can judge. Therefore, the line-of-sight calculation unit 14 calculates the line-of-sight direction of the eyeball 10 based on the set of parameters selected by the selection unit 16 until the main switch 18 of the camera is turned off.

-Claims 13 and 14-When the eyeball 10 is separated from the viewfinder eyepiece window of the camera,
Alternatively, when the observer is away from the viewfinder eyepiece window 18 for a predetermined time, it can be determined that the observer should change or the observer needs to change the set of parameters by newly using contacts, glasses, or the like. Therefore, the line-of-sight calculating means 14 selects the means 1 until the eyeball 10 leaves the viewfinder eyepiece window, or for a predetermined time after the eyeball 10 leaves the viewfinder eyepiece window.
The eyeball 10 based on the set of parameters selected by
The line-of-sight direction of is calculated.

[0023]

【Example】

(1) Description Common to Embodiments First, the configuration common to each embodiment and the principle of the present invention will be described. Note that in the following description, the observer's eyeball and the inside thereof will be described using the reference numerals used in FIG. 31.

<< Structure of Line-of-sight Detecting Device >> FIG. 2 is a view showing the structure of the line-of-sight detecting device according to the present invention.
Reference numeral 101 denotes an observation surface (field-of-view screen) which is provided in an optical device such as a camera and is observed by an observer. Reference numeral 102 denotes an eyeball of the observer. The observer views a specific point in the left direction in the figure through the observation surface 101. Is watching. Reference numeral 103 denotes a light source drive means for detecting the line of sight,
Reference numerals 104a and 104b denote line-of-sight detection light sources such as IREDs, and these line-of-sight detection light sources 104a and 104b are in the vicinity of the eyeball 102 of the observer and at positions that do not obstruct the field of view of the observation surface 101, or It is arranged so as not to stand out so as not to attract the attention of the person.

105 and 106 are both imaging lenses and 10
Reference numerals 7 and 108 denote photoelectric conversion elements such as a two-dimensional CCD, which have a two-dimensionally spread light receiving surface parallel to the observation surface 101. The imaging lens 105 forms an image of the observer's eyeball 102 on the photoelectric conversion element 107, and the imaging lens 106 forms an image of the observer's eyeball 102 on the photoelectric conversion element 108. The imaging lenses 105 and 106 and the photoelectric conversion elements 107 and 108 form a detection optical system. The detection optical system is located at a position where the observer's eyeball 102 can be observed, and is arranged outside the visual field observed by the observer so as not to interfere with the visual field observed by the observer or within the visual field observed. Even so, it is arranged so as not to stand out so as not to attract the attention of the observer.

Reference numerals 109 and 110 denote photoelectric conversion elements 1 respectively.
Signal processing means for processing the output signals of 07 and 108, 11
1 is a central processing unit, and this central processing unit 111 is
Control of line-of-sight detection light source driving means 103, photoelectric conversion element 1
07, 108 drive control and signal processing means 10
It has a function of calculating the line-of-sight direction from the outputs of 9 and 110. Further, the central processing unit 111 is provided with a memory, and a value peculiar to each eyeball described later is stored in this memory.

FIG. 3 is a block diagram showing another example of the arrangement of the detection optical system. In the example shown in FIG. 3, a semi-transmissive mirror (half mirror) 112 is used to detect an image of the eyeball 102 of the observer outside the field of view of the observer (imaging lenses 105, 1).
06, photoelectric conversion elements 107 and 108). Further, in the example shown in FIG. 2, the image of the eyeball 102 formed by the pair of imaging lenses 105 and 106 is converted into a pair of photoelectric conversion elements 10 corresponding to the respective imaging lenses 105 and 106.
In the example shown in FIG. 3, the images are formed on the eyeballs 7 and 108 by the pair of imaging lenses 105 and 106.
Image is formed on one photoelectric conversion element 113. The light receiving surface of the photoelectric conversion element 113 is arranged so as to be conjugate with a surface parallel to the observation surface 101. Regarding the photoelectric conversion element, as shown in FIG. 2, a pair of photoelectric conversion elements 107, 1
The detection may be performed using 08. Reference numeral 114 is a signal processing means for processing the output signal of the photoelectric conversion element 113.

<< Description of Coordinate System >> FIG. 4 is a view showing a coordinate system introduced for explaining the principle of the present invention. In FIG. 4, 115 is the cornea of the eyeball 102 of the observer, and 116 is the iris of the eyeball 102 of the observer. With respect to the observation surface 101 to which the observer is gazing, the Y axis is set along one direction (in the figure, in the figure) along the direction in which the distance between the rotation center O ″ of the eyeball of the observer and the observation surface 101 changes. (Left and right direction of the observation surface 101)
A three-dimensional coordinate system in which the X axis is along the X axis and the Z axis is along the direction orthogonal to both the X axis and the Y axis (the vertical direction of the observation surface 101 in the figure) is referred to as "real space coordinates". ".

The origin O of the real space coordinates is the eyeball 10 of the observer.
The rotation center O ″ is determined so as to be located on the Y axis when 2 is in a position directly facing the center of the observation surface 101, and is located on the reference surface of the visual axis detection device according to the present invention. Since the positional relationship between the reference plane and the observation plane 101 can be set arbitrarily in design, they are shown on the same plane in the illustrated example for ease of explanation.

Further, consider an XZ plane including O ″ about the center of rotation of the observer's eyeball displaced in the real space coordinate system. This plane is called the "rotation center plane", and the Y coordinate of the rotation center plane is set as y = y 'in real space coordinates. Further, the intersection of the rotation center plane and the Y axis is O ′, and the X ′ axis and the Z ′ axis whose origin is O ′ are parallel to the X axis and the Z axis of the real space coordinate, respectively, on the rotation center plane. Take Although y ′ changes as the observer's eyeball 102 moves back and forth,
The rotation center O ″ of is always included in the rotation center plane. The angle between the line segment O ″ C and the XY plane is φ, and the line segment O ″ is
The angle between the line segment projected from C on the XY plane and the Y axis is defined as θ.

FIG. 5 is a diagram showing the optical principle of the present invention. The line-of-sight detection light sources 104a and 104b (representatively shown by reference numeral 104 in the figure) are located in the vicinity of the eyeball 102 of the observer, and the two lines of sight detection light sources 104a and 104b are provided.
Focusing on any one of the light sources 104,
The two visual line detection light sources 104 connect the virtual image of the corneal reflection image to the eyeball 102 of the observer at point P. This virtual image is called Purkinje's first image. Purkinje's first image is the imaging lens 10
An image is re-formed on the point Pi on the photoelectric conversion element 107 via the image forming element 5 and on a point Pj on the photoelectric conversion element 108 via the image forming lens 106. At this time, the photoelectric conversion elements 107 and 108
A coordinate system in which one direction common to the respective light receiving surfaces is set to the X ″ axis and the Z ″ axis is set to the direction perpendicular to the X ″ axis is set and hereinafter referred to as “observation plane coordinates”. According to the observation plane coordinates common to the photoelectric conversion elements 107 and 108 set on the light receiving surface, the point P is Pi: (P
ix, Piz), Pj (Pjx,
Pjz) is observed. From the observation coordinates of the above two points, P
The spatial position P of the point in the real space coordinate system: (Px, Py,
Pz) is a function f _X , f _Y and f _Z that is uniquely determined between the real space coordinate system and the observation plane coordinate system,

Px = f _X (Pix, Piz, Pjx, Pjz) (5) Py = f _Y (Pix, Piz, Pjx, Pjz) (6) Pz = f _Z (Pix, Piz, Pjx, Pjz) (Pix, Piz, Pjx, Pjz) It is expressed as 7). Therefore, the image forming positions of the pair of corneal reflection images and the iris formed on the light receiving surfaces of the photoelectric conversion elements 107 and 108 can be arbitrarily converted into the real space coordinate system. By forming an image on the pair of photoelectric conversion elements 107 and 108,
The positions of the corneal reflection image and the pupil center position in the real space coordinate system can be known.

What has been described above is the point P, the imaging lens 10
5 and 106 and the photoelectric conversion elements 107 and 108 are changed in position, only the specific shapes of the conversion equations (5) to (7) are changed, and the position of the point P on the real space coordinates is the photoelectric conversion element 107. ,
The conversion formula (5) from the observation plane coordinates Pi and Pj common to 108
There is no change that it can be converted using (7).
The specific shapes of f _X , f _Y , and f _Z are as follows.
It is determined by the magnification of 5, 106, aberration, and the like.

<< Explanation of Detection Principle >> Next, the detection principle of the present invention will be described. In the present invention, as shown in FIG.
The spatial coordinates of the corneal curvature center position C and the pupil center position D, the distance A from the eyeball rotation center position O ″ to the pupil center position D, and the distance ρ from the eyeball rotation center position O ″ to the corneal curvature center position C , The distance A−ρ between the center position D of the pupil and the center position C of the corneal curvature is used to determine the line-of-sight direction of the eyeball of the observer.

Corneal curvature center position C and pupil center position D
The relationship between the spatial coordinates of and the constant A-ρ depending on the eyeball of the observer is expressed by the following relational expression using the above-mentioned real space coordinate system. Here, the corneal curvature center position C: (Cx,
Cy, Cz) and the pupil center position D: (Dx, Dy, Dz), and θ is the eyeball rotation angle in the X-axis direction defined by the above definition, L
Let X be the translation amount of the eyeball rotation center in the X-axis direction, φ be the eyeball rotation angle in the Z-axis direction defined above, and k be the translation amount of the eyeball rotation center in the Z-axis direction. Each coordinate value is

Cx = L + ρcosφsinθ (8) Cy = y ′ + ρcosφcosθ (9) Cz = k + ρsinφ (10) Each coordinate value of the pupil center position D is

[Equation 6] Dx = L + A cos φ sin θ (11) Dy = y ′ + A cos φ cos θ (12) Dz = k + A sin φ (13) If the equations (8) to (11), the equations (9) to (12), and the equations (10) to (13) are subtracted in this order, the parallel displacement amounts L and y'included in these equations. , K can be eliminated, resulting in the following equation:

Cx−Dx = (A−ρ) cosφsin θ (14) Cy−Dy = (A−ρ) cosφcosθ (15) Cz−Dz = (A−ρ) sinφ (16) Therefore, the pupil center position and the cornea Distance from the center of curvature A-
ρ is

A−ρ = √ ((Dx−Cx) ² + (Cy−Dy) ² + (Dz−Cz) ² ) (17), and the required eyeball rotation angles θ and φ are

(9) θ = sin ⁻¹ ((Dx−Cx) / (A−ρ) cosφ) (18) φ = sin ⁻¹ ((Dz−Cz) / (A−ρ)) (19) Then, the eyeball rotation angles θ and φ can be obtained from the distance A−ρ between the actual pupil center position D and the corneal curvature center position C obtained by observing the eyeball of each observer.
Since the observer's line-of-sight direction can be obtained from the eyeball rotation angles θ and φ and the deviation angle γ unique to each observer's eyeball, which will be described later, the spatial coordinates (Cx, Cx,
Cy, Cz) and the spatial coordinates (Dx, Dy,
The line-of-sight direction can be obtained by obtaining Dz) and the shift angle γ.

<< Actual Calculation Procedure >> Hereinafter, the corneal curvature center position C: (Cx, Cy, Cz) and the pupil center position D: (Dx,
A procedure for concretely obtaining Dy, Dz) and the deviation angle γ by measurement will be described. First, a method of obtaining the pupil center position will be described. The boundary 118 between the pupil and the iris is a great circle of a sphere in the real space coordinate system as shown in FIG. Since the normal line of the plane including the great circle may not coincide with the Y axis direction of the real space coordinate system, it is connected as an ellipse on the photoelectric conversion elements 107 and 108 in which the XZ plane and the light receiving surface are arranged in parallel. Although an image is formed, the angle between the normal and the Y-axis is usually small, so it can be regarded as a substantially circular shape.

At this time, as shown in FIG. 8, the coordinates of the four points of the observation plane coordinate system at the left and right ends of the boundary 118 between the iris 116 and the pupil part 117 on the light receiving surfaces of the photoelectric conversion elements 107 and 108, and the upper and lower ends. From the above, the spatial coordinates of the pupil center D in the real space coordinate system are obtained by image processing. The coordinates of the four points are represented by the points D _{R ′} : (D _{R ′} x, D _{R ′} y, D _{R ′} z) and the point D _{L ′} : (D
_{L 'x, D L' y} , D L 'z), the point _{D T': (D T '} x, D T' y, D
_{T 'z),} point _{D B': (D B '} x, D B' y, putting a D _{B 'z),}
The pupil center position D: (Dx, Dy, Dz) is Dx = (D _{R ′} x
+ D _{L ′} x) / 2, Dy = (D _{R ′} y + D _{L ′} y) / 2, or Dy = (D _{T ′} y + D _{B ′} y) / 2, Dz = (D _{R ′} z + D _{L ′} z)
/ 2. This method is effective even when the pupil part 117 is regarded as an ellipse.

Next, how to obtain the corneal curvature center position C will be described. As shown in FIG. 9, the corneal curvature center position C is a line connecting the line-of-sight detection light source S1 and the Purkinje first image P1 generated by the line-of-sight detection light source S1, the line-of-sight detection light source S2, and the line-of-sight detection light source S2. Is a point of intersection with a straight line connecting the Purkinje first image P2, the spatial coordinates (Cx, C) of the corneal curvature center position C in real space coordinates.
y, Cz) is a light source for line-of-sight detection S1: (S1x, S1y,
S1z), light source for line-of-sight detection S2: (S2x, S2y, S2
z), the Purkinje first image P by the eye-gaze detecting light source S1
1: (P1x, P1y, P1z), Purkinje first image P2 by the line-of-sight detection light source S2: Using the respective spatial coordinates of (P2x, P2y, P2z),

(Equation 10) Cx = (S1xP1z (S2x-P2x) -S2xP2z (S1x-P1x) -S1zP1x (S2x-P2x) + S2zP2x (S1x-P1x)) / ((S1x-P1x) × (S2z-P2z)-( (S2x-P2x) × (S1z-P1z)) (20) Cy = (S1yP1x (S2y-P2y) -S2yP2x (S1y-P1y) -S1xP1y (S2y-P2y) + S2xP2y (S1y-P1y)) / (S1y (P1y) × (S2x-P2x)-(S2y-P2y) × (S1x-P1x)) (21) Cz = (S1zP1x (S2z-P2z) -S2zP2x (S1z-P1z) -S1xP1z (S2z-P2z) + S2xP2z (S2z-P2z) + S2xP2z S1z-P1z)) / ((S1z-P1z) × (S2x-P2x)-(S2z-P2z) × (S1x-P1x)) (22). However, when the corneal curvature center C is observed on a straight line passing through the first visual line detection light source S1 and the second visual line detection light source S2, the denominator of the equations (20) to (22) becomes zero. It may end up. At this time, for example (2
In the equations 0) and (22), the corneal curvature center position C is represented only by the X coordinate value and the Z coordinate value, but the corneal curvature center position C is calculated using the X coordinate value and the Y coordinate value.

Here, the line-of-sight detection light source S1: (S1x,
S1y, S1z) and the light source S2 for detecting the line of sight: (S2x,
The spatial coordinates of (S2y, S2z) are fixed values determined in advance in the real space coordinate system, and the Purkinje first image P1:
(P1x, P1y, P1z) and Purkinje's first image P
2: The spatial coordinates of (P2x, P2y, P2z) can be obtained from the positions of the respective Purkinje first images P1 and P2 formed on the light receiving surfaces of the pair of photoelectric conversion elements 107 and 108 as described above. it can. Therefore, since the coordinates of the corneal curvature center position C and the pupil center position D in the real space coordinate system can be obtained, the eyeball rotation angles θ and φ, and the distance A− between the corneal curvature center position C and the pupil center position D−. ρ is obtained.

Further, how to obtain the deviation angle γ will be described. Here, the shift angle γ means an angle formed by the line of sight of the observer and a straight line connecting the pupil center position D and the corneal curvature center C, as shown in FIG. 10. In this example,
Although only the deviation angle γ on the YZ plane of the real space coordinate system is considered, the deviation angle on the XY plane may be considered in the same manner. Therefore, the line-of-sight direction of the observer becomes θ-γ, which is a value obtained by correcting the deviation angle γ from the detected eyeball rotation angle θ.

Reference numeral 120 is an index at the center of the observation plane 101 for the observer to gaze, so that the center thereof coincides with the intersection of the Y axis and the observation plane 101, that is, the origin O on the real space coordinates. It is located in. A straight line 121 passes through the pupil center position D of the observer's eyeball 102 and the corneal curvature center position C. Therefore, in FIG. 10, the angle formed by the Y axis and the straight line 121 is the shift angle γ. As the index 120, for example, one for displaying a focus detection area in a viewfinder such as a camera is used.
There are known ones that illuminate a part of No. 1 and ones in which a liquid crystal is provided on a part of the observation surface 101.

The index 120 is presented only when the shift angle γ is calculated, and seeks the observer's gaze. At this time, since the observer is gazing at the index 120 in the center of the observation surface 101, the detected eyeball rotation angle at this time should be 0 (zero) °. However, the detected eye rotation angle θ of the eyeball of the observer has a certain angle. Since the angle formed by the line of sight of the observer and the straight line 121 connecting the pupil center position D and the corneal curvature center C is defined as the shift angle γ, the detected eyeball rotation angle θ at this time is equal to the shift angle γ. Therefore, index 1
By detecting the eyeball rotation angle θ when gazing at 20, the shift angle γ of the observer's eyeball 102 can be obtained.

In FIG. 10, the eyeball 102 of the observer is drawn so that the eyeball rotation center O ″ is on the Y axis of the real space coordinates for convenience, but as shown in FIG. '' May be located off the Y-axis. Further, the index 120 is arranged at a position that coincides with the origin O for convenience of explanation, but it may be arranged at any position on the observation surface 101. By repeating the above-described work, the values of the distance A-ρ between the pupil center position and the corneal curvature center position and the shift angle γ can be obtained more accurately.

The method for obtaining the corneal curvature center position C, the pupil center position D, and the shift angle γ, which are the parameters necessary for detecting the gaze direction, have been described above. The operation for obtaining the corneal curvature center position C and the pupil center position D is described below. In contrast to the line-of-sight detection operation itself, the operation of obtaining the shift angle γ is, so to speak, a calibration operation and needs to be performed prior to the line-of-sight detection operation. In the line-of-sight detection device of the present invention, prior to the line-of-sight detection operation, the distance A-ρ and the shift angle γ between the corneal curvature center position C and the pupil center position D of the eyeball of the observer are obtained, and this distance A is obtained. A pair of −ρ and shift angle γ is stored in the memory in the central processing unit 111. As described above, the distance A-ρ can be calculated at the same time when the shift angle γ is obtained. At this time, a set of the distance A-ρ and the shift angle γ is obtained in advance for a plurality of observers and stored in the memory.
Then, in the visual axis detection device of the present invention, the distance A-ρ is obtained in the actual visual axis detection operation, and this distance A-ρ is compared with the distance A-ρ stored in the memory and stored in the memory. Which one of the plurality of observers is observing is determined, and the distance A-ρ and the shift angle γ of the corresponding observer are determined.
Is used to calculate the line-of-sight direction. Distance A as described above
Although −ρ can be obtained by the line-of-sight detection operation, the shift angle γ cannot be obtained by the line-of-sight detection operation. Therefore, if the distance A−ρ and the shift angle γ are stored in the memory as a pair, any observer It is possible to determine whether or not the user is observing during the eye gaze detection operation. Hereinafter, in this specification, an operation of obtaining the distance A-ρ and the deviation angle γ prior to the visual axis detection operation is referred to as a preliminary measurement operation.

(2) Specific Description of Each Embodiment-First Embodiment-The first embodiment is applied to the visual axis detecting device shown in FIGS. The feature of the present embodiment is that the distance A-ρ between the corneal curvature center C and the pupil center D, which is a value unique to each eyeball, and the straight line connecting the line-of-sight direction with the pupil center D and the corneal curvature center C. A set of shift angles γ that are formed is stored in advance in the memory of the central processing unit 111 for a plurality of observers by the preliminary measurement operation, and among the sets of the plurality of distances A-ρ and shift angles γ when the line of sight is detected. The optimum one is automatically selected and used for eye gaze detection.

FIG. 11 is a flow chart showing the overall flow of the visual axis detection calculation by the visual axis detection device of this embodiment. First, in step S100, it is determined whether or not the preliminary measurement operation of the distance A-ρ and the deviation angle γ is performed. The details of the determination operation are arbitrary, but as an example, the determination is affirmative if the observer has set in advance that the preliminary measurement operation is to be performed by a switch, dial, etc., and is negative in other cases. Actions are listed. If the determination is affirmative, the program moves to the subroutine SS1, and after the preliminary measurement operation is performed, moves to step S101. On the other hand, if the determination is negative, the process directly proceeds to step S101.

In step S101, it is determined whether or not the visual axis detection operation is performed. Although the details of the determination operation are arbitrary, an example is a determination operation in which the determination is negative when the observer turns off the main switch of the line-of-sight detection device, and the determination is affirmative in other cases. If the determination is affirmative, the program proceeds to the subroutine SS2, and the visual axis detection operation is performed. On the other hand, if the determination is denied, the program ends.

FIG. 12 is a flowchart of the subroutine SS1 showing the preliminary measurement operation by the visual axis detection device of this embodiment. First, in step S110, the index 120 is displayed at a predetermined position on the observation surface 101 as described above, and the observer is requested to gaze. Next, in step S111, the eyeball 102 of the observer is illuminated by the pair of visual line detection light sources 104a and 104b, and the reflected image of the eyeball 102 illuminated by the light sources 104a and 104b is converted into a pair of photoelectric conversion elements 1.
Images are captured on the light-receiving surfaces of 07 and 108 and captured. The output value of the captured image is temporarily stored in the memory of the central processing unit 111.

In step S112, the photoelectric conversion element 10
The reflected image data of the eyeball 102 of the observer received at 7 and 108 is read from the memory of the central processing unit 111, and the boundary 118 between the iris 116 and the pupil portion 117 is read at the left end on the light receiving surface of the photoelectric conversion element 107. Point D _L i, right end point D _R i, upper end point D _T i, and lower end point D _B i are read, and left end point D _L j and right end point on the light receiving surface of photoelectric conversion element 108 are read. The four points, point D _R j, top point _DT j, and bottom point D _B j are read. In step S113,
The boundary 1 between the iris 116 and the pupil part 117 in the real space coordinate system from the points D _L i and D _L j read in step S112.
The spatial position of the left end D _L of 18 is calculated, the spatial position of the right end D _R of the boundary 118 in the real space coordinate system is calculated from D _R i, D _R j, and the real space coordinate system is calculated from D _T i, D _T j. The spatial position of the upper end D _T of the boundary 118 at is calculated, and the spatial position of the lower end D _B of the boundary 118 in the real space coordinate system is calculated from D _B i and D _B j. In step S114, 4 points D
The spatial position of the pupil center from the spatial position of _L , D _R , D _T , D _B
Ask for. The spatial position of the pupil center D in the real space coordinate system is
As described above, Dx = (D _L x + D _R x) / 2, Dy = (D _L y
+ D _R y) / 2 and Dz = (D _T z + D _B z) / 2.

In step S115, the central processing unit 11
The reflected image data of the eyeball 102 of the observer is read from the memory No. 1 and the positions P1i and P2 of the two Purkinje first images are read.
The coordinate values of i, P1j, and P2j in the observation plane coordinate system are read. In step S116, the positions P1i and P1 of the one Purkinje first image read in step S115.
From j, the spatial position P1: (P1x, P1y, P1z) of the Purkinje first image in the real space coordinate system, and from the positions P2i and P2j of the other Purkinje first image, the spatial position P2 of the Purkinje first image in real space coordinates. : (P2x, P2y, P2
z) is calculated. In step S117, step S1
The spatial positions P1 and P2 of the Purkinje's first image in the real space coordinate system calculated in 16 and the spatial positions S1 and S2 of the two predetermined visual line detection light sources 104a and 104b are calculated.
Substituting into equations (20) to (22), the spatial position of the corneal curvature center C in the real space coordinate system is obtained.

In step S118, step S114
And the spatial position of the pupil center D obtained in step S117
The spatial position of the corneal curvature center C obtained in step (17), (18)
Substituting into the formula, the distance A-ρ and the shift angle γ are calculated. The calculated set of the distance A−ρ and the deviation angle γ is stored in the memory of the central processing unit 111. In this embodiment, the memory has a plurality of (for example, three) areas, and each area stores one set of the distance A-ρ and the shift angle γ.

Next, FIG. 13 is a flow chart of a subroutine SS2 showing the visual axis detecting operation by the visual axis detecting device of this embodiment. First, in step S120, the eyeball 102 of the observer is illuminated by the pair of visual line detection light sources 104a and 104b, and the reflected image of the eyeball 102 illuminated by the light sources 104a and 104b is paired with the pair of photoelectric conversion elements 1.
Images are captured on the light-receiving surfaces of 07 and 108 and captured. Then, it is determined whether or not the reflected image of the eyeball 102 can be detected. If the determination is affirmative, the process proceeds to the subroutine SS3,
After the distance A-ρ detection operation is performed, the process proceeds to step S121. On the other hand, if the determination is negative, the line-of-sight detection operation ends. Whether or not the reflection image of the eyeball of the observer can be detected may be determined by, for example, whether or not the Purkinje first image or the boundary reflection image between the iris and the pupil can be detected.

FIG. 14 is a flow chart of the subroutine SS3 showing the operation of calculating the distance A-ρ by the visual axis detection device of this embodiment. The operation of each step of the sub-routine SS3 is almost the same as that of the sub-routine SS1 of the preliminary measurement operation described above, so only the outline will be described. First, in step S130, a pair of line-of-sight detection light sources 104a,
The reflected image of the eyeball 102 illuminated by 104b is imaged and captured on the light receiving surfaces of the pair of photoelectric conversion elements 107 and 108.

In step S131, of the boundary 118 between the iris 116 and the pupil portion 117, the left end point D _L i, the right end point D _R i, the upper end point D _T i, on the light receiving surface of the photoelectric conversion element 107, While reading the four points D _B i at the bottom,
The point D at the left end on the light receiving surface of the photoelectric conversion element 108
Four points are read: _L j, right end point D _R j, upper end point D _T j, and lower end point D _B j. In step S132, step S1
31 points D _L i, D _R i, D _T i, D _B i, D _L
4 points D _L of the boundary 118 between the iris 116 and the pupil part 117 in the real space coordinate system from j, D _R j, D _T j, D _B j,
Calculate the spatial positions of D _R , D _T , and D _B , respectively. In step S133, the spatial position D of the pupil center is obtained from the spatial positions of the four points D _L , D _R , D _T , D _B.

In step S134, the coordinate values of the positions P1i, P2i, P1j, and P2j of the two Purkinje first images in the observation plane coordinate system are read. Step S135
Then, from the positions P1i, P2i, P1j, and P2j of the Purkinje first image read in step S134, the spatial position P1: (P1 of the Purkinje first image in the real space coordinate system is obtained.
x, P1y, P1z), P2: (P2x, P2y, P2
z) is calculated. In step S136, step S1
The spatial positions P1 and P2 of the Purkinje's first image in the real space coordinate system calculated in step 35 and the spatial positions S1 and S2 of the two predetermined gaze detection light sources 104a and 104b are
Substituting into equations (20) to (22), the spatial position of the corneal curvature center C in the real space coordinate system is obtained.

In step S137, step S133
The spatial position of the pupil center D obtained in step S136
The spatial position of the corneal curvature center C obtained in step (17), (18)
Substituting into the equation, the distance A-ρ is calculated. The distance A−ρ and the spatial positions of the corneal curvature center C and the pupil center D in the real space coordinate system are temporarily stored in the memory of the central processing unit 111.

Returning to FIG. 13, steps S121 to S1.
In 26, the distance A-ρ calculated in the subroutine SS3
And the distance A-ρ calculated in the preliminary measurement operation by the above-mentioned subroutine SS1 and stored in the memory are compared,
A set of the distance A-ρ and the shift angle γ used in the visual axis detection operation described later is selected. First, in step S121,
It is determined whether or not the distance A-ρ calculated in the subroutine SS3 is within a range that can be regarded as equal to the distance A-ρ stored in the first area of the memory. If the determination is negative, the process proceeds to step S123. Step S122
Then, the distance A-ρ and the shift angle γ stored in the first area are set to be used for the line-of-sight direction calculation described later.

On the other hand, in step S123, it is determined whether or not the distance A-ρ calculated in the subroutine SS3 is within a range that can be regarded as equal to the distance A-ρ stored in the second area of the memory. If the determination is affirmative, the program proceeds to step S124, and if the determination is negative, the program proceeds to step S125. In step S124, the second
The distance A- [rho] and the shift angle [gamma] stored in the area of are used for the calculation of the line-of-sight direction described later.

Further, in step S125, it is determined whether or not the distance A-ρ calculated in the subroutine SS3 is within a range that can be regarded as equal to the distance A-ρ stored in the third area of the memory. If the determination is positive, the program proceeds to step S126. On the other hand, the determination is negative, that is, the distance A− calculated in the subroutine SS3.
When it is determined that ρ is not equal to any of the distances A-ρ stored in the first to third regions, it is the state of the observer who is not performing the preliminary measurement operation or the state of the eyeball of the observer, or It is determined that the preliminary measurement operation has not performed an appropriate measurement for some reason, and the eye gaze detection operation is stopped. In this case, the subroutine S is reissued if necessary.
The preliminary measurement operation of S1 will be performed. Step S12
In 6, the distance A-ρ and the shift angle γ stored in the third area are set to be used in the line-of-sight direction calculation described later.

In step S127, the spatial positions of the corneal curvature center C and the pupil center D obtained in the distance A-ρ calculation subroutine SS3, and steps S122 and S12.
4 or the distance A-ρ set in S126 is (18),
The line-of-sight rotation angles θ and φ are calculated by substituting into the equation (19), and the line-of-sight direction of the observer is calculated from the line-of-sight rotation angles θ and φ and the shift angle γ set in step S122, S124, or S126.

Therefore, according to the present embodiment, the set of the distance A-ρ and the deviation angle γ is stored in the memory by the preliminary measurement operation, and the optimum distance is selected according to the eyeball of the observer who is currently observing. Since the set of A-ρ and the shift angle γ is selected, the observer does not need to select the set of the distance A-ρ and the shift angle γ by a switch or the like as in the conventional visual line detection device, which is troublesome. The selection operation can be omitted, and it is possible to easily and quickly respond to situations such as changes in the observer.
Accurate gaze detection is possible without the risk of being set incorrectly.

(3) Example of Application to Camera Next, an example in which the visual axis detection device according to the present invention is mounted on a single-lens reflex camera (hereinafter simply referred to as a camera) will be described.

-Second Embodiment- FIG. 15 is a schematic configuration diagram showing an embodiment in which the line-of-sight detection device shown in FIG. 2 is mounted on a camera. In FIG. 15, 201 is shown as one lens for convenience, but in reality, a photographing lens composed of a plurality of lenses, 202 is a quick return mirror, 203 is a display device, 204 is a focusing screen,
Although 205 is a condenser lens, 206 is a pentaprism, and 207 is shown as a single lens for the sake of convenience, it is actually an eyepiece lens that may consist of a plurality of lenses.
Reference numeral 208 denotes an eyepiece glass having no degree, which prevents dust, water droplets, and the like from entering the inside from the eyepiece portion, 209 is an observer's eyeball, and 210a is one of the light sources for eye-gaze detection. It is arranged at a position where the observer's eyeball 209 is directly irradiated from the surroundings. In addition, one or more light sources for sight line detection (not shown) are arranged at positions where the eyes 209 of the observer are directly irradiated from the periphery of the eyepiece glass 208, similarly to the light source for sight line detection 210a. When mounted on a camera as in this embodiment, it is desirable that the light source 210a for line-of-sight detection uses a light source such as an infrared light emitting diode that emits light in a wavelength range invisible to the eyes. Reference numeral 211a denotes one of a plurality of imaging lenses provided corresponding to each of the photoelectric conversion elements described later, 212a denotes one of the plurality of photoelectric conversion elements, and a surface light receiving element such as a CCD is used. Are suitable. In addition, one or more imaging lenses and photoelectric conversion elements (not shown) are similarly arranged around the eyepiece glass 208.

FIG. 16 is an enlarged view of the finder portion of the camera of this embodiment, showing an example of the arrangement of the light source for line-of-sight detection, the imaging lens, and the photoelectric conversion element. In FIG. 16, 210a and 210b are the light sources for line-of-sight detection and 2
Reference numerals 11a and 211b are imaging lenses. For each of the imaging lenses 211a and 211b, one photoelectric conversion element (not shown) is arranged behind (on the back side of the paper in the figure) one by one. Reference numeral 220 is a finder eyepiece window, 221 is an eyepiece eyepiece, 222 is a finder body, 223 is a clip-on type accessory shoe, and 224 is an eyepiece shutter lever. In FIG. 16, one line of sight line detection light sources 210a and 210b are arranged on the left and right sides of the finder eyepiece window 220, and one set of detection optical systems including imaging lenses 211a and 211b and photoelectric conversion elements are arranged. According to the above principle, the arrangement of the visual axis detection light source and the detection optical system is arbitrary, and the hatched portion 225 shown in FIG.
2 or more light sources for line-of-sight detection at any position
It is possible to arbitrarily arrange one or more detection optical systems.

In the illustrated example, the line-of-sight detection light source 210a,
The positional relationship between the imaging lens 211a, the photoelectric conversion element 212a, the eyepiece glass 208, and the eyepiece eyepiece 221 is shown in FIG.
The relationship is as shown in. The eyepiece glass 208 is attached to the eyepiece window 220 of the finder body 222, and the eyepiece eyepiece 221 has a structure having no optical system to prevent the eye gaze detection light source 210a from irradiating the eyepiece glass 208 and a photoelectric conversion element. 212a does not pass through an optical system other than the imaging lens 211a. In the present embodiment, the set of the distance A-ρ and the deviation angle γ obtained in the preliminary measurement operation described later is E provided in the camera.
It is stored in a non-volatile memory such as an EPROM. That is, since it is troublesome to perform the preliminary measurement operation each time the photographing operation is performed, even if the main switch of the camera is turned off, a storage unit that holds the set such as the distance A-ρ once stored, in other words, supply of electricity. EEPR that retains information even when there is no
A set such as the distance A-ρ is stored in a non-volatile memory such as the OM, and an appropriate set is selected during the shooting operation.

Next, a specific procedure of detecting the visual axis by the visual axis detecting apparatus of this embodiment will be described with reference to the flow charts of FIGS. In the flow charts shown in FIGS. 19 to 21, until the eyeball 209 of the observer moves away from the viewfinder eyepiece window 220 so that it becomes meaningless to detect the line of sight or it becomes impossible to detect the line of sight.
The line-of-sight is detected using the set distance A-ρ and the deviation angle γ. In the following description, the steps similar to those in the above-described first embodiment will only be outlined.

FIG. 19 is a flow chart showing the overall flow of the visual axis detection calculation by the visual axis detection device of this embodiment. First, in step S200, it is determined whether or not the preliminary measurement operation of the distance A-ρ and the shift angle γ is performed. The details of the determination operation are arbitrary, but as an example, the determination is affirmative if the observer has set in advance that the preliminary measurement operation is to be performed by the switch, dial, etc. attached to the camera, otherwise the determination is negative. Such a determination operation is performed. If the determination is positive, the program is the subroutine SS
1 is performed, and after the preliminary measurement operation is performed, step S2
Move to 01. The details of the sub-routine SS1 of the preliminary measurement operation are the same as those in the above-described first embodiment, and the description thereof will be omitted. On the other hand, if the determination is denied, the step S
Move to 201.

In step S201, it is determined whether or not the main switch of the camera is turned off. If the determination is affirmative, it is determined that there is no intention to perform the visual axis detection and the photographing operation, and the program ends. On the other hand, if the determination is negative, it is assumed that there is an intention to perform the line-of-sight detection and the photographing operation, and in step S202, the process proceeds to the sub-routine SS4 after waiting for the on-determination of the switch SW1 (not shown) that is turned on by half-pressing the release button of the camera. Then, the sight line detection and the photographing operation are performed.

20 and 21 are flowcharts of the subroutine SS4 showing the visual axis detection and photographing operation by the visual axis detection device of the present embodiment, of which FIG. 20 calculates the distance A-ρ and is nonvolatile. 7 is a flowchart showing an operation of selecting a set of the distance A-ρ and the shift angle γ stored in the memory. First, in step S210, the eyeball 209 of the observer is illuminated by the pair of visual line detection light sources 210a and 210b.
The reflected image of the eyeball 209 illuminated by b is imaged and captured on the light receiving surfaces of the pair of photoelectric conversion elements. Then, it is determined from the reflection image of the eyeball 209 whether or not the observer is looking through the viewfinder eyepiece window 220, and if the determination is affirmative, the process proceeds to the subroutine SS3, and after the distance A-ρ detection operation is performed, the step is performed. The process moves to S211. Distance A-ρ
The details of the detection operation subroutine SS3 are the same as those in the above-described first embodiment, and therefore description thereof will be omitted. On the other hand, if the determination is negative, the process proceeds to step S217 to end the visual axis detection operation, and the control unit that does not perform visual axis detection controls the camera (details will be described later). Whether or not the observer is looking through the viewfinder eyepiece window 220 may be determined by, for example, whether or not the Purkinje first image and the boundary reflection image between the iris and the pupil can be detected.

In steps S211 to S216, the distance A-ρ calculated in the sub-routine SS3 is compared with the distance A-ρ calculated in the preliminary measurement operation by the above-mentioned sub-routine SS1 and stored in the memory. A set of distance A-ρ and shift angle γ used in the detection operation is selected. First, in step S211, the subroutine SS
It is determined whether or not the distance A-ρ calculated in 3 is within a range that can be regarded as equal to the distance A-ρ stored in the first area of the non-volatile memory. If the determination is negative, the process proceeds to step S213. In step S212, the distance A-ρ and the shift angle γ stored in the first area are set to be used in the later-described gaze direction calculation. After this, the program moves to step S218 in FIG.

On the other hand, in step S213, it is determined whether or not the distance A-ρ calculated in the subroutine SS3 is within a range that can be regarded as equal to the distance A-ρ stored in the second area of the nonvolatile memory. If the determination is affirmative, the program proceeds to step S214, and if the determination is negative, the program proceeds to step S215. In step S214, the distance A-ρ and the shift angle γ stored in the second area.
Is used to calculate the line-of-sight direction described below. After this, the program moves to step S218 in FIG.

Further, in step S215, it is determined whether or not the distance A-ρ calculated in the subroutine SS3 is within a range that can be regarded as equal to the distance A-ρ stored in the third area of the nonvolatile memory. If the determination is affirmative, the program proceeds to step S216, and the distance A-ρ and the deviation angle γ stored in the third area are set to be used for the later-described gaze direction calculation. After this, the program is
1 to step S218.

On the other hand, if the determination is negative, step S21.
7, the visual axis detection operation is stopped, and the camera is controlled by the control means that does not perform visual axis detection. Step S21
5 is denied, that is, the distance A-ρ between the pupil center position D and the corneal curvature center position C of the observer looking into the viewfinder eyepiece window 220 of the camera is measured in advance and stored in the nonvolatile memory. This is the case where none of the distances A-ρ stored in 1 is equal. Such a situation occurs when an observer who looks into the camera for the first time or an observer who looks into the camera with an unusual eye state is looking into the viewfinder eyepiece window 220 of the camera. This is the case when a stranger passing by in order to take a picture presses the shutter, or when a person who normally uses contact lenses happens to use eyeglasses. When the program proceeds to step S217, an accurate sight line direction cannot be detected because the preliminary measurement operation for the observer looking through the viewfinder eyepiece window 220 of the camera has not been performed. Therefore, in this case, it is better to stop the line-of-sight detection and automatically switch to the conventional camera control method that does not perform line-of-sight detection.
The observer may be notified that the visual line detection mode has been switched to the non-visual line detection mode.

As another means, it is possible to newly measure the set of the distance A-ρ and the deviation angle γ by executing the above-mentioned subroutine SS1 of the preliminary measurement operation. In this case, if the non-volatile memory has a space, the newly measured set of the distance A-ρ and the deviation angle γ may be stored in the non-volatile memory, or if the non-volatile memory has no space, or Alternatively, regardless of whether the non-volatile memory has a free space, it may be temporarily stored in the volatile memory from the beginning. Further, as another means, when the detected line-of-sight direction is good with coarse accuracy, the line-of-sight detection may be performed by substituting the set of the distance A−ρ and the shift angle γ stored in the nonvolatile memory. Since there are means suitable for each of these so-called photographing modes, it is preferable that the photographer who actually operates the camera can select which means to use by means of a dial or lever.

Next, FIG. 21 is a flow chart showing the visual axis detection and photographing operation using the set of the distance A-ρ and the shift angle γ selected in the flow chart of FIG. 20, and steps S212, S214 of FIG. It is a flowchart following S216. First, in step S218, similarly to step S210, it is determined whether or not the observer is looking through the viewfinder eyepiece window 220 of the camera,
If the determination is affirmative, the program proceeds to step S219, and if the determination is negative, the program proceeds to step S222. Note that in step S218, it is not necessary to calculate the distance A-ρ thereafter, so it is sufficient to detect the reflected image of the eyeball using one of the pair of detection optical systems (imaging lens, photoelectric conversion element). Is.

In step S219, the spatial positions of the corneal curvature center C and the pupil center D obtained in the distance A-ρ calculation subroutine SS3, and steps S212 and S21.
4 or the distance A-ρ set in S216 is (18),
The line-of-sight rotation angles θ and φ are calculated by substituting into the equation (19), and the line-of-sight direction of the observer is calculated from these line-of-sight rotation angles θ and φ and the shift angle γ set in step S212, S214 or S216. Once the line-of-sight direction is calculated, it is possible to perform photometry, distance measurement, lens driving, etc. based on the line-of-sight information. What kind of operation is performed at this point depends on the camera design and the observer's camera settings.

In step S220, a switch S (not shown) is turned on by fully pressing the release button of the camera.
It is determined whether or not W2 is in the ON state, and if the determination is affirmative, the process proceeds to step S221, the release operation of the camera is performed, and the process returns to step S218. On the other hand, step S2
When the determination of 20 is denied, the process directly returns to step S218. On the other hand, in step S222, step S21
2, the distance A-set by S214 or S216
The setting of the set of ρ and the shift angle γ is released, and the visual axis detection and the photographing operation are ended.

In this way, while the observer is looking through the viewfinder eyepiece window 220, it is judged that he / she is willing to continue photographing, and the visual axis detection and photographing operation are continued, and the reflected image of the eyeball of the observer is continued. When is no longer detected, it is determined that there is no intention to continue shooting, and the line-of-sight detection and shooting operations end. Once the line-of-sight detection operation is completed, when the line-of-sight detection operation is started again, a new step S21 is performed.
The program starts from 0, and the set of the distance A-ρ between the new pupil center position D and the corneal curvature center position C and the deviation angle γ is used. Even if the person does not change, it is possible to deal with the case where the distance A-ρ and the shift angle γ change due to the photographer using glasses, using contact lenses, or the like.

Therefore, according to this embodiment as well, it is possible to obtain the same effects as those of the above-mentioned first embodiment. Further, in steps S210 and S218, if the eyeball image cannot be detected in one image capture, the line-of-sight detection operation is ended, but the determination is made by repeatedly capturing the image a predetermined number of times. You can also
Furthermore, even if the eyeball image is not detected immediately after the release is performed in step S221, if the eyeball image is detected within a certain period of time, it is determined that the user intends to continue shooting, and the process proceeds to step S219. good.

-Third Embodiment- FIG. 22 is a view showing an embodiment in which the line-of-sight detection device shown in FIG. 2 is mounted on a camera, and step S in FIG.
It is a flow chart which shows the concrete procedure of eye gaze detection and photography operation following 212, S214, and S216. In the flowchart shown in FIG. 22, while the observer continues to press the release button halfway, it is assumed that the observer has the intention to continue shooting, and the distance A-ρ determined in steps S212, S214, and S216 of FIG. And the shift angle γ are continuously used, the line-of-sight detection and the photographing operation are finished when the half-press operation is finished, and the settings of the distance A-ρ and the shift angle γ are released. Since the contents of each step of the flowchart of FIG. 22 and FIG. 23 described later are almost the same as the contents of each step of the flowchart (FIG. 21) of the second embodiment, the step numbers of the second embodiment will be referred to as appropriate. And the description is omitted.

The program shown in the flow chart of FIG. 22 is executed in step S212, S214 or S21 of FIG.
6 is executed subsequently. First, in step S250, similarly to step S218 of FIG. 21, it is determined whether or not the observer is looking through the viewfinder eyepiece window 220 of the camera, and if the determination is affirmative, the program proceeds to step S2.
If the determination is negative, the process proceeds to step S254. Since it is not necessary to calculate the distance A-ρ in step S250 and thereafter, it is sufficient to detect the reflected image of the eyeball using one of the pair of detection optical systems (imaging lens, photoelectric conversion element). Is.

In step S251, as in step S219 of FIG. 21, the spatial positions of the corneal curvature center C and the pupil center D, and step S212, S214 or S.
The line-of-sight direction of the observer is calculated from the distance A-ρ and the shift angle γ set in 216. In step S252, as in step S220 of FIG. 21, a switch SW2 (not shown) is turned on by fully pressing the release button of the camera.
Is determined to be on, and if the determination is affirmative, the process proceeds to step S253 to perform the release operation of the camera and then the process proceeds to step S254. If the determination is negative, the process proceeds to step S254.

In step S254, the above-described switch S
It is determined whether or not W1 is in the ON state. If the determination is affirmative, it is determined that the observer has the intention to continue shooting, and the process returns to step S250. If the determination is negative, the observer continues. If it is determined that there is no intention to continue shooting, steps S212 and S21 are performed in step S255.
4 or the setting of the set of the distance A-ρ and the shift angle γ set in S216 is canceled, and the visual axis detection and the photographing operation are ended.

Therefore, according to this embodiment, the same operational effect as that of the above-mentioned second embodiment can be obtained. In addition,
In S250, the eye-gaze detecting operation is shown to be terminated if the eyeball image cannot be detected by one-time image capture, but it is also possible to make a determination by repeatedly capturing images a predetermined number of times. Even if the switch SW1 is not turned on immediately after the release is performed in S254, if the switch SW1 is turned on again within a certain period of time, it is determined that the user intends to continue shooting, and S2 is set.
You can proceed to 50. Alternatively, it is determined whether or not the switch SW2 is in the ON state even while the program shifts from step S250 to step S254, and when it is determined to be ON, the camera control means that does not detect the line of sight,
Alternatively, if the release operation is performed using the immediately preceding line-of-sight information, the photo opportunity will not be missed.

-Fourth Embodiment- FIG. 23 is a diagram showing an embodiment in which the line-of-sight detection device shown in FIG. 2 is mounted on a camera, and step S in FIG.
It is a flow chart which shows the concrete procedure of eye gaze detection and photography operation following 212, S214, and S216. In the flowchart shown in FIG. 23, while the main switch of the camera is in the ON state, it is assumed that the observer has the intention to continue the photographing, and steps S212, S214 and S in FIG.
The pair of the distance A-ρ and the deviation angle γ determined in 216 is continuously used, and when the main switch is turned off, the line-of-sight detection and the photographing operation are ended, and the settings of the distance A-ρ and the deviation angle γ are canceled. is doing.

The program shown in the flow chart of FIG. 23 is executed in step S212, S214 or S21 of FIG.
6 is executed subsequently. First, in step S300, as in step S218 of FIG. 21, it is determined whether or not the observer is looking through the viewfinder eyepiece window 220 of the camera. If the determination is affirmative, the program proceeds to step S3.
When the determination is negative, the process proceeds to step S304. In step S300, it is not necessary to calculate the distance A- [rho] after that, and it is sufficient to detect the reflected image of the eyeball by one of the detection optical systems (imaging lens, photoelectric conversion element).

In step S301, as in step S219 in FIG. 21, the spatial positions of the corneal curvature center C and the pupil center D, and step S212, S214 or S.
The line-of-sight direction of the observer is calculated from the distance A-ρ and the shift angle γ set in 216. In step S302, as in step S220 of FIG. 21, a switch SW2 (not shown) is turned on by fully pressing the release button of the camera.
Is determined to be on, and if the determination is affirmative, the process proceeds to step S303 to perform the release operation of the camera, and then the process proceeds to step S304. If the determination is negative, the process directly proceeds to step S304.

In step S304, it is determined whether or not the main switch (not shown) of the camera is on.
If the determination is affirmative, it is determined that the observer is willing to continue shooting, and the process returns to step S300. If the determination is negative, it is determined that the observer is not willing to continue shooting, In step S305, step S2
12, the distance A set by S214 or S216
The setting of the pair of −ρ and the shift angle γ is canceled, and the visual axis detection and the photographing operation are ended.

Therefore, according to this embodiment, it is possible to obtain the same effects as those of the above-mentioned second embodiment. In addition,
In step S300, the eye gaze detection operation is shown to be terminated if the eyeball image cannot be detected by capturing the image once, but it is also possible to repeatedly determine the image by a predetermined number of times. Alternatively, it is determined whether or not the switch SW2 (not shown) that is turned on by fully pressing the release button is in the on state even while the program shifts from step S300 to step S304, and if it is determined that the line of sight is detected. If the release operation is performed by the control means of a camera that does not exist or using the immediately preceding line-of-sight information, it is possible to avoid missing a photo opportunity.

-Fifth Embodiment-In the second to fourth embodiments described above, the operation of calculating the line-of-sight direction was performed while the release button was half pressed, but the operation of calculating the line-of-sight direction after the release button was fully pressed. You may go. If you do not calculate the line-of-sight direction without fully pressing the release button, the cycle of capturing the eyeball image will be shortened by the amount that the step of calculating the line-of-sight in the half-pressed state can be omitted, so it is possible to respond to the faster movement of the observer's eyeball. it can.

FIG. 24 is a flowchart of the flowchart of the second embodiment shown in FIG. 21, in which the operation of calculating the line-of-sight direction is performed after the release button is fully pressed. In the description of FIGS. 24 to 26, each step of the flowchart corresponding to each is cited and the description thereof is omitted.

The program shown in the flowchart of FIG. 24 is executed in step S212, S214 or S21 of FIG.
6 is executed subsequently. First, in step S350, similarly to step S218 in FIG. 21, it is determined whether or not the observer is looking through the viewfinder eyepiece window 220 of the camera. If the determination is negative, the program proceeds to step S351, and if the determination is negative, it is determined that the observer has no intention to continue shooting, and the program proceeds to step S351.
Move to 354.

In step S351, similarly to step S220 in FIG. 21, it is determined whether or not the switch SW2 (not shown) that is turned on by fully pressing the release button of the camera is in the on state, and if the determination is affirmative. Step S
If the determination is negative, the process returns to step S350. In step S352, step S21 of FIG.
The gaze direction of the observer is calculated in the same manner as in 9, and step S35
In step 3, the release operation of the camera is performed as in step S221 of FIG. After this, the program proceeds to step S3
Return to 50. On the other hand, in step S354, as in step S222 of FIG. 24, steps S212 and S214 are performed.
Alternatively, the setting of the set of the distance A-ρ and the shift angle γ set in S216 is canceled, and the visual axis detection and the photographing operation are ended.

In the embodiment shown in FIG. 24, step S35.
If the eyeball image cannot be detected by once capturing the image at 0, the line-of-sight detection operation is ended, but the determination can be made by repeatedly capturing the image a predetermined number of times. Further, even if the eyeball image is not detected immediately after the release is performed in step S353, if the eyeball image is detected within a certain period of time, it is determined that the user intends to continue shooting, and the process may proceed to step S351. .

FIG. 25 is a flowchart of the flowchart of the third embodiment shown in FIG. 22, in which the operation of calculating the line-of-sight direction is performed after the release button is fully pressed. The program shown in the flowchart of FIG.
It is executed subsequent to step S212, S214 or S216 of FIG. First, in step S400,
As in step S250 of FIG. 22, it is determined whether the observer is looking through the viewfinder eyepiece window 220 of the camera, and if the determination is affirmative, the program proceeds to step S401.
If the determination is negative, the process proceeds to step S404.

In step S401, similarly to step S252 in FIG. 22, it is determined whether or not the switch SW2 (not shown) that is turned on by fully pressing the release button of the camera is on, and the determination is affirmative. And the process proceeds to step S402, and if the determination is negative, step S40
Go to 4. In step S402, step S in FIG.
Similar to step 251, the line-of-sight direction of the observer is calculated. In step S403, the camera release operation is performed as in step S253 of FIG. After this, the program proceeds to step S404. In step S404, similarly to step S254 in FIG. 22, it is determined whether or not the switch SW1 (not shown) that is turned on by half-pressing the release button of the camera is in the on state. Judges that he / she still has the intention to continue shooting, and returns to step S400. If the judgment is negative, it is judged that the observer has no intention to continue shooting, and in steps S405, S212, S214 or The setting of the pair of the distance A-ρ and the shift angle γ set in S216 is canceled, and the visual axis detection and the photographing operation are ended.

In the embodiment shown in FIG. 25, the eye-gaze detecting operation is ended if the eyeball image cannot be detected in one image capture in S400, but the image is repeatedly captured a predetermined number of times. You can also make a decision by doing. Even if the switch SW1 is not turned on immediately after the release in S404, if the switch SW1 is turned on again within a certain period of time, it is determined that the user intends to continue shooting, and the process proceeds to S400. May be. Alternatively, it is determined whether or not the switch SW2 is in the ON state even while the program shifts from step S400 to step S404. If the release operation is performed using the information, it is possible to avoid missing a photo opportunity.

FIG. 26 is a flowchart of the fourth embodiment shown in FIG. 23, in which the operation of calculating the line-of-sight direction is performed after the release button is fully pressed. The program shown in the flowchart of FIG. 26 is
It is executed subsequent to step S212, S214 or S216 of FIG. First, in step S450,
Similar to step S300 of FIG. 23, it is determined whether the observer is looking through the viewfinder eyepiece window 220 of the camera, and if the determination is affirmative, the program proceeds to step S451.
If the determination is negative, the process proceeds to step S454.

In step S451, similarly to step S302 in FIG. 23, it is determined whether or not the switch SW2 (not shown) that is turned on by fully pressing the release button of the camera is on, and the determination is affirmative. And the process proceeds to step S452, and if the determination is negative, step S45
Go to 4. In step S451, step S in FIG.
Similar to 301, the line-of-sight direction of the observer is calculated. In step S403, the release operation of the camera is performed as in step S303 of FIG. After this, the program proceeds to step S454. In step S454, similarly to step S304 in FIG. 23, it is determined whether or not the main switch (not shown) of the camera is in the on state, and if the determination is affirmative, the observer has the intention to continue shooting. If the determination is negative, it is determined that the observer has no intention of continuing the photographing, and in step S455, steps S212 and S2 are performed.
The setting of the distance A-ρ and the shift angle γ set by 14 or S216 is canceled, and the visual axis detection and the photographing operation are ended.

Although it is shown in step S450 that the eye-gaze detecting operation is ended if the eyeball image cannot be detected by one-time image capturing, the image is repeatedly captured a predetermined number of times for determination. You can also do it. Alternatively, steps S450 to S454
Even when the program shifts to, it is determined whether or not the switch SW2 (not shown) that is turned on by full-pressing the release button is in the on state, and when it is determined to be on, the control means of the camera that does not detect the line of sight, Alternatively, if the release operation is performed using the immediately preceding line-of-sight information, the photo opportunity will not be missed.

Various methods of selecting the set of the distance A-ρ between the center position of the pupil and the center position of the corneal curvature and the shift angle γ have been described above, but there are methods suitable for each so-called photographing mode. Therefore, it is preferable that the photographer who actually operates the camera can select which method to use by using a dial or a lever. Further, in each of the above-described embodiments, the setting of the distance A-ρ and the shift angle γ between the center position of the pupil and the center position of the corneal curvature used for the calculation is canceled when the line-of-sight detection and the photographing operation are finished. However, if necessary, the line-of-sight detection operation may be reset when the operation is restarted. In each of the above-described embodiments, the half-push switch SW1 that resets the set of the distance A-ρ and the shift angle γ may instead be reset when the release operation is completed. Further, in each of the above-mentioned embodiments, in the non-volatile memory,
Three areas were provided for accommodating the sets of the distance A-ρ and the deviation angle γ between the center position of the pupil and the corneal curvature center, respectively. Regarding the number of sets of the distance A-ρ and the deviation angle γ that can be stored, Since it is determined by the performance of the memory element and the usability of the camera, it may be provided in one area, two areas, or four or more areas. Further, in each of the above-described embodiments, the line-of-sight detection is performed using the set of the distance A-ρ and the shift angle γ stored in the memory, but the distance A-ρ obtained by the subroutine SS3 is used. May be.

-Sixth Embodiment- FIG. 27 is a schematic block diagram showing an embodiment in which the line-of-sight detection device shown in FIG. 3 is mounted on a camera. Although 201 is shown as a single lens for the sake of convenience, in actuality, a photographing lens composed of a plurality of lenses, 202 is a quick return mirror,
Reference numeral 203 is a display device, 204 is a focusing plate, 205 is a condenser lens, 206 is a pentaprism, and 207 is a single lens for the sake of convenience, but in actuality, it is an eyepiece lens that may consist of a plurality of lenses. , And has a dichroic mirror 207a. Reference numeral 208 is an eyepiece glass having no degree, which prevents dust, water droplets, and the like from entering the inside from the eyepiece portion, 209 is an observer's eyeball, and 210a
Is one of the light sources for detecting the line of sight, and the eyepiece glass 20
It is arranged at a position where the observer's eyeball 209 is directly irradiated from around 8. In addition, one or more light sources for sight line detection (not shown) are arranged at positions where the eyes 209 of the observer are directly irradiated from the periphery of the eyepiece glass 208, similarly to the light source for sight line detection 210a. Also in this embodiment, the line-of-sight detection light source 210
It is desirable to use a line-of-sight detection light source that emits light in a wavelength range invisible to the eye, such as an infrared light emitting diode. An image forming lens 211a is paired with another image forming lens (not shown). Reference numeral 212 denotes a photoelectric conversion element, and in this embodiment, an image formed by a pair of imaging lenses is formed at different positions on the light receiving surface of one photoelectric conversion element. A surface light receiving element such as a CCD is suitable for the photoelectric conversion element 212.

FIG. 28 is an enlarged view of the finder portion of the camera of the present invention, showing an example of the arrangement of the light source for detecting the line of sight. In this embodiment, four light sources for sight line detection are arranged around the viewfinder eyepiece window. In FIG. 28, 210a, 210b, 210c, and 210d are sight line detection light sources, 220 is a viewfinder eyepiece window, and 221.
Is an eyepiece, 222 is a finder body, 223 is a clip-on type accessory shoe, and 224 is an eyepiece shutter lever.

Further, the four light sources for line-of-sight detection 210a-2
10d is arranged near the apex of the eyepiece window 220, and a substantially square shape is formed by the line connecting these light sources 210a to 210d. Therefore, the positional relationship between the line-of-sight detection light sources 210a to 210d is kept constant when the camera is held in the horizontal position and the vertical position.

The visual axis detection light source 210a in this embodiment
~ 210d, eyepiece glass 208 and eyepiece eyepiece 221
The positional relationship of is as shown in FIG. The eyepiece glass 208 is attached to the eyepiece window 220 of the finder main body 222, and the eyepiece eyepiece 221 has a structure having no optical system, so that the sightline detection light source 210a
Does not illuminate the eyepiece glass 208.

FIG. 30 is a diagram showing the arrangement of the imaging lenses 211a and 211b and the photoelectric conversion element 212 of the camera of this embodiment. In this embodiment, the reflected image of the eyeball 209 is 2
Images are formed on different regions on the light-receiving surface of one photoelectric conversion element 212 by using the image forming lenses 211a and 211b. It should be noted that instead of one photoelectric conversion element 212 in FIG. 30, two photoelectric conversion elements may be used in correspondence with the respective imaging lenses 211a and 211b. Also in the present embodiment, the arrangement of the line-of-sight detection light source and the detection optical system is arbitrary, and the arrangement and number of two or more lines-of-sight line detection light sources are arbitrary, and the detection optical system also forms an image. If two or more lenses are used to form an image, the arrangement and the number thereof are arbitrary.

[0106]

As described in detail above, according to the present invention, a plurality of sets of parameters corresponding to a plurality of observers are stored in the storage means, and the observer who is currently observing by the selecting means. Since the optimum parameter set is selected according to the eyeballs, the observer does not need to select the parameters in advance as in the conventional eye gaze detection device, and the troublesome selection operation can be omitted and the observer can be changed. In addition to being able to respond to the above situation easily and quickly, it is possible to detect the line of sight accurately without the risk of incorrect setting. Further, claim 3
According to the invention of claim 5, when there is no parameter in the storage means that is substantially equal to the second parameter calculated by the second parameter calculation means, an operation such as stopping the gaze detection operation is performed. Appropriate treatment is possible even when observed by an observer other than the plurality of observers corresponding to the plurality of parameter sets stored in the means. Further, according to the inventions of claims 9 to 14, the change of the observer is determined depending on the operation of the release button, the operation of the main switch, the presence or absence of the detection of the reflected image of the eyeball, and whether or not the eyeball has moved away from the viewfinder eyepiece window. Since it is possible to judge the necessity of a new selection operation of the correction value due to the above, it is possible to respond promptly and accurately even if a situation such as a change of the observer occurs.

[Brief description of drawings]

FIG. 1 is a diagram corresponding to a claim of the present invention.

FIG. 2 is a diagram showing an example of a configuration of a visual line detection device of the present invention.

FIG. 3 is a diagram showing another example of the configuration of the visual line detection device of the present invention.

FIG. 4 is a diagram showing an example of a coordinate system used for explaining the principle of the present invention.

FIG. 5 is an optical explanatory view showing a line-of-sight detection system of the present invention.

FIG. 6 is a diagram for explaining the detection principle of the present invention.

FIG. 7 is a diagram for explaining a position of a boundary between a pupil portion and an iris portion.

FIG. 8 is a diagram for explaining an example of a method of determining a pupil center position according to the present invention.

FIG. 9 is a diagram for explaining an example of a method of determining a corneal curvature center position according to the present invention.

FIG. 10 is a diagram showing an example of a deviation angle determination method according to the present invention.

FIG. 11 is a flow chart for explaining the overall operation of the first embodiment of the present invention.

FIG. 12 is a flowchart for explaining a preliminary measurement operation in the first embodiment.

FIG. 13 is a flow chart for explaining a visual axis detecting operation in the first embodiment.

FIG. 14 is a flowchart for explaining a distance A-ρ calculation operation in the first embodiment.

FIG. 15 is a schematic diagram showing an optical system of a camera to which a second embodiment of the present invention has been applied.

FIG. 16 is a front view showing a viewfinder portion taken out of the camera of the second embodiment.

FIG. 17 is a diagram showing an arrangement range of a visual axis detection light source and a detection optical system in the second example.

FIG. 18 is a cross-sectional view showing a finder portion of the camera of the second embodiment.

FIG. 19 is a flow chart for explaining the overall operation of the second embodiment of the present invention.

FIG. 20 is a flow chart for explaining the line-of-sight detection and photographing operation in the second embodiment.

FIG. 21 is a flowchart following FIG. 20.

FIG. 22 is a flow chart for explaining the operation of the third embodiment.

FIG. 23 is a flow chart for explaining the operation of the fourth embodiment.

FIG. 24 is a flow chart for explaining an example of the operation of the fifth embodiment.

FIG. 25 is a flowchart for explaining another example of the operation of the fifth embodiment.

FIG. 26 is a flow chart for explaining still another example of the operation of the fifth embodiment.

FIG. 27 is a schematic diagram showing an optical system of a camera to which a sixth embodiment of the present invention is applied.

FIG. 28 is a front view showing a finder portion taken out of the camera of the sixth embodiment.

FIG. 29 is a cross-sectional view showing a viewfinder section of a camera of the sixth embodiment.

FIG. 30 is a perspective view of a finder portion according to a seventh embodiment.

FIG. 31 is a horizontal sectional view showing a human eyeball.

[Explanation of symbols]

 C Corneal curvature center D Pupil center 101 Observation surface 102, 209 Eyeball 104, 210 Eye-gaze detecting light source 105, 106, 211 Imaging lens 107, 108, 113, 212 Photoelectric conversion element 109, 110, 114 Signal processing means 111 Central processing Device 115 Cornea 116 Iris 117 Pupil 118 Boundary 220 Viewfinder eyepiece window

Claims (14)

[Claims] 1. An observing means for observing a reflected image of an eyeball of an observer, a first parameter calculating means for calculating a first parameter in advance based on an observation result of the observing means prior to the sight line detection; Second parameter calculating means for calculating a second parameter based on the observation result of the observing means at the time of detection, and the eyeball based on the observation result of the observing means and the first and second parameters at the time of detecting the line of sight. In the visual line detection device including a visual line calculation unit that calculates the visual line direction of the, a storage unit that stores a plurality of sets of the first and second parameters corresponding to the plurality of observers, and the second Based on the second parameter calculated by the parameter calculating means, one parameter set is selected from the plurality of parameter sets stored in the storage means. A-option unit, the line-of-sight calculating means, line-of-sight detecting device and calculates a line-of-sight direction of the eyeball based on the set of the parameter selected by said selection means. 2. An observing means for observing a reflection image of an eyeball of an observer, a first parameter calculating means for calculating a first parameter in advance based on an observation result of the observing means prior to the sight line detection, and a sight line. Second parameter calculating means for calculating a second parameter based on the observation result of the observing means at the time of detection, and the eyeball based on the observation result of the observing means and the first and second parameters at the time of detecting the line of sight. In the visual line detection device including a visual line calculation unit that calculates the visual line direction of the, a storage unit that stores a plurality of sets of the first and second parameters corresponding to the plurality of observers, and the second Based on the second parameter calculated by the parameter calculation means, a set of the second parameter and the second parameter is selected from the plurality of the first parameters stored in the storage means. Selecting means for selecting one of the first parameters, wherein the line-of-sight calculating means is configured to select the second parameter calculated by the second parameter calculating means and the first parameter selected by the selecting means. A line-of-sight detection device, which calculates the line-of-sight direction of the eyeball based on the parameter. 3. An observing means for observing a reflection image of an eyeball of an observer, a first parameter calculating means for calculating a first parameter in advance based on an observation result of the observing means prior to the sight line detection, and a sight line. Second parameter calculating means for calculating a second parameter based on the observation result of the observing means at the time of detection, and the eyeball based on the observation result of the observing means and the first and second parameters at the time of detecting the line of sight. A line-of-sight calculation device that calculates a line-of-sight direction of the storage device, and a storage unit that stores a plurality of sets of the first and second parameters corresponding to the plurality of observers; If there is a parameter that is substantially equal to the second parameter calculated by the second parameter calculation means among the plurality of second parameters stored in the second parameter, the second parameter Data and selection means for selecting a first parameter that forms a pair with the data, and when the line-of-sight calculation means selects the parameter set by the selection means, the eye-gaze calculation means selects the eyeball based on the parameter set. The line-of-sight detection apparatus for calculating the line-of-sight direction of the eyeball and stopping the operation of calculating the line-of-sight direction of the eyeball when the parameter set is not selected by the selection unit. 4. An observing means for observing a reflection image of an eyeball of an observer, a first parameter calculating means for calculating a first parameter in advance based on an observation result of the observing means prior to the sight line detection, and a sight line. Second parameter calculating means for calculating a second parameter based on the observation result of the observing means at the time of detection, and the eyeball based on the observation result of the observing means and the first and second parameters at the time of detecting the line of sight. A line-of-sight calculation device that calculates a line-of-sight direction of the storage device, and a storage unit that stores a plurality of sets of the first and second parameters corresponding to the plurality of observers; If there is a parameter substantially equal to the second parameter calculated by the second parameter calculation means among the second parameters stored in the second parameter, the second parameter And a first parameter forming a pair with the first parameter is selected, and if there is no one substantially equal to the second parameter calculated by the second parameter calculating means, the set of parameters stored in the storing means A line-of-sight direction, wherein the line-of-sight calculation unit calculates the line-of-sight direction of the eyeball based on the set of parameters selected by the selection unit. Detection device. 5. An observing means for observing a reflection image of an eyeball of an observer, a first parameter calculating means for calculating a first parameter in advance based on an observation result of the observing means prior to the sight line detection, and a sight line. Second parameter calculating means for calculating a second parameter based on the observation result of the observing means at the time of detection, and the eyeball based on the observation result of the observing means and the first and second parameters at the time of detecting the line of sight. A line-of-sight calculation device that calculates a line-of-sight direction of the storage device, and a storage unit that stores a plurality of sets of the first and second parameters corresponding to the plurality of observers; If there is a parameter substantially equal to the second parameter calculated by the second parameter calculation means among the second parameters stored in the second parameter, the second parameter And the first parameter forming a pair with this is selected, and if there is no substantially equal second parameter calculated by the second parameter calculation means, the visual axis detection operation is stopped and the first and second parameters are calculated. Selection means for newly calculating the set of the first and second parameters by the second parameter calculation means and selecting this set of parameters, wherein the line-of-sight calculation means is selected by the selection means. A line-of-sight detection apparatus that calculates the line-of-sight direction of the eyeball based on a set of parameters. 6. The line-of-sight detection device according to claim 5, wherein when there is no substantially equal second parameter calculated by the second parameter calculation means, the first parameter is calculated.
And a line-of-sight detection apparatus which stores the set of the first and second parameters newly calculated by the second parameter calculation means in the storage means. 7. The line-of-sight detection device according to claim 1, wherein the first parameter includes an angle formed by a line-of-sight direction of the eyeball and an optical axis of the eyeball. Eye gaze detection device. 8. The line-of-sight detection apparatus according to claim 1, wherein the second parameter includes a distance between a pupil center position of the eyeball and a corneal curvature center position. Characteristic gaze detection device. 9. The line-of-sight detection device according to claim 1, wherein the line-of-sight calculation unit sets the parameters selected by the selection unit until the reflected image of the eyeball cannot be observed. A line-of-sight detection device, wherein the line-of-sight direction of the eyeball is calculated based on a set. 10. A camera equipped with the line-of-sight detection device according to claim 1, wherein the line-of-sight calculation means releases a first stroke of a release button included in the camera. Until then, a camera equipped with a line-of-sight detection device, which calculates the line-of-sight direction of the eyeball based on the set of parameters selected by the selection means. 11. A camera equipped with the line-of-sight detection device according to any one of claims 1 to 8, wherein the line-of-sight calculation means selects the selection means for a predetermined time from the release of the first stroke or the second stroke of the release button. A camera equipped with a line-of-sight detection device, wherein the line-of-sight direction of the eyeball is calculated based on the set of parameters selected by. 12. A camera equipped with the line-of-sight detection device according to claim 1, wherein the line-of-sight calculation means until a main switch included in the camera is turned off. A camera equipped with a line-of-sight detection device, which calculates a line-of-sight direction of the eyeball based on the set of parameters selected by the selection means. 13. A camera equipped with the line-of-sight detection device according to claim 1, wherein the line-of-sight calculation means continues until the eyeball moves away from a viewfinder eyepiece window of the camera. A camera equipped with a line-of-sight detection device, characterized in that the line-of-sight direction of the eyeball is calculated based on the set of parameters selected by the selection means. 14. A camera equipped with the line-of-sight detection device according to claim 1, wherein the line-of-sight calculation means is configured to perform the selection by the selection means only for a predetermined time after the eyeball leaves the viewfinder eyepiece window of the camera. A camera equipped with a visual line detection device, characterized in that the visual line direction of the eyeball is calculated based on the selected set of parameters.