JP2021064928A - Electronic device - Google Patents

Abstract

To provide a technology that makes it possible to detect a situation in which a viewpoint cannot be estimated with high accuracy using a simple configuration.SOLUTION: An electronic device capable of acquiring an eye image obtained by capturing an eye looking at a screen of display means through an eye window frame includes estimation means for estimating a viewpoint of the eye on the screen based on the eye image, and detection means for detecting a misaligned viewing state in which the eye is displaced from a position corresponding to a center of the screen in the eye image based on a position of a pupil image or a Purkinje image in the eye image.SELECTED DRAWING: Figure 12

Description

The present invention relates to an electronic device capable of estimating (detecting) a viewpoint.

In recent years, cameras have become more automated and intelligent. Patent Document 1 proposes a technique of recognizing a subject intended by the photographer and performing focus control based on information of the photographer's viewpoint (line-of-sight position) looking into the finder without manually inputting the subject position. ing. Patent Document 2 proposes a technique for improving the detection accuracy of a viewpoint by considering the rotation angle and position of the head in addition to the rotation angle of the eyeball. In Patent Document 3, there is a technique of having a plurality of eyeball illuminations and switching the eyeball illuminations to be used according to a determination result of whether or not the light from the eyeball illuminations reaches the user's eyeballs to detect the gaze point. Proposed. Patent Document 4 proposes a technique for switching an image display range (range for displaying an image) in a display device in a finder.

Japanese Unexamined Patent Publication No. 2004-8323 JP-A-2009-104524 Special Table 2018-506781 Japanese Unexamined Patent Publication No. 2014-64094

In the technique of Patent Document 1, the user's viewpoint is estimated (detected) by detecting the rotation angle of the eyeball based on the position of the pupil image or the Purkinje image in the eye image obtained by capturing the user's (photographer's) eye. There is. However, if the user shifts his / her face with respect to the finder and the head moves greatly in translation, the pupil image and Purkinje image in the eye image also move in translation significantly, so that the viewpoint cannot be estimated with high accuracy.

In the technique of Patent Document 2, not only the eyes but the entire face is imaged, the position and inclination of the head are detected from the feature points of the face, and the position and inclination of the head are considered in addition to the rotation angle of the eyeball. This improves the estimation accuracy of the viewpoint. However, it is necessary to have a configuration in which the entire face (entire head) is imaged, which leads to complicated equipment and high cost. Further, the estimation accuracy of the viewpoint cannot be improved when the head is hidden and the image cannot be taken, such as when the user looks into the finder of the camera or when the VR glasses (VR goggles) are worn.

In the technique of Patent Document 3, the gaze point detection is performed by switching the eyeball illumination to be used depending on the situation. However, the viewpoint may not be estimated with high accuracy because the ghost caused by the eyeball illumination is reflected in the eye image.

In the technique of Patent Document 4, the image display range in the display device is narrowed from the normal time not for estimating the viewpoint but for power saving. Therefore, when estimating the viewpoint, the image display range may not be switched to an appropriate range, and the viewpoint may not be estimated with high accuracy.

An object of the present invention is to provide a technique capable of detecting a state in which the viewpoint cannot be estimated with high accuracy with a simple configuration.

A first aspect of the present invention is an electronic device capable of acquiring an eye image obtained by capturing an image of an eye viewing the screen of a display means through an eye window frame, and based on the eye image, the viewpoint of the eye on the screen can be obtained. An estimation means for estimating and a detection means for detecting a deviation visual state in which the eye deviates from the position corresponding to the center of the screen in the eye image based on the position of the pupil image or the Pulkinier image in the eye image. It is an electronic device characterized by having.

A second aspect of the present invention is a control method for an electronic device capable of acquiring an eye image obtained by capturing an image of an eye viewing a screen of a display means through an eye window frame, and based on the eye image, the eye on the screen. A step of estimating the viewpoint of the eye, and a step of detecting a visual state in which the eye deviates from the position corresponding to the center of the screen in the eye image based on the position of the pupil image or the Pulkinier image in the eye image. It is a control method characterized by having.

A third aspect of the present invention is a program for making a computer function as each means of the above-mentioned electronic device. A fourth aspect of the present invention is a computer-readable recording medium that stores a program for making a computer function as each means of the above-mentioned electronic devices.

According to the present invention, it is possible to detect a state in which the viewpoint cannot be estimated with high accuracy with a simple configuration.

It is an external view of the camera which concerns on Example 1. FIG. It is sectional drawing of the camera which concerns on Example 1. FIG. It is a figure which shows the state that the virtual image which concerns on Example 1 is formed. It is a block diagram of the camera which concerns on Example 1. FIG. It is a figure which shows the field of view in the finder which concerns on Example 1. FIG. It is a figure for demonstrating the principle of the visual field detection method which concerns on Example 1. FIG. It is a figure which shows the eye image which concerns on Example 1. FIG. It is a flowchart of the line-of-sight detection operation which concerns on Example 1. FIG. It is a figure which shows the visual state which concerns on Example 1. FIG. It is a figure which shows the oblique peep state which concerns on Example 1. FIG. It is a figure which shows the eye image which concerns on Example 1. FIG. It is a flowchart of the camera operation which concerns on Example 1. FIG. It is a figure which shows the index which concerns on Example 2. FIG. It is a figure which shows the image display which concerns on Example 2. FIG. It is a figure which shows the visual state which concerns on Example 2. FIG. It is a flowchart of the camera operation which concerns on Example 2. FIG. It is a figure which shows the image display which concerns on Example 3. FIG. It is a figure which shows the visual state which concerns on Example 3. FIG. It is a flowchart of the camera operation which concerns on Example 4. FIG. It is a rear perspective view of the camera which concerns on Example 5. FIG. It is sectional drawing of the camera which concerns on Example 5. FIG. It is a figure which shows the visual state which concerns on Example 5. FIG. It is a block diagram of the camera which concerns on Example 5. FIG. It is a figure which shows the eye image which concerns on Example 5. FIG. It is a figure which shows the visual state which concerns on Example 5. FIG. It is a figure which shows the ghost generation principle which concerns on Example 5. It is a figure which shows the eye image which concerns on Example 5. FIG. It is a rear perspective view of the camera which concerns on Example 5. FIG. It is a flowchart of the camera operation which concerns on Example 5. It is a figure which shows the eye image which concerns on Example 5. FIG.

As described above, in the prior art, the viewpoint (line-of-sight position) cannot be estimated with high accuracy when the pupil image and the Purkinje image in the eye image move greatly in translation. In particular, when a user wearing eyeglasses looks into the finder of the camera, the eyeball is often not sufficiently close to the finder, and the viewpoint cannot be estimated with high accuracy in many cases. Specifically, when the eyeball is not sufficiently close to the finder, the line of sight is blocked by the eyepiece window frame of the finder, etc., and the visible range in the finder is limited. You may not be able to see the edges of the screen in the finder. In such a case, the user tends to make a large translational movement of the head from the front of the finder and look into the finder from an angle because he / she tries to see the end portion. Since the translational movement distance of the head when looking into the finder from an angle is much larger than when the camera is used in the normal (manufacturer's recommended) usage method, a non-negligible error occurs in the viewpoint estimation result. ..

Therefore, in the present invention, based on the detected pupil image and Purkinje image, the user detects an oblique looking state in which the user looks into the finder from an angle, and adversely affects the adverse effect (adverse effect in viewpoint detection) caused by the oblique looking state. Suppress.

<< Example 1 >>
Hereinafter, Example 1 of the present invention will be described with reference to the accompanying drawings.

<Explanation of configuration>
1 (a) and 1 (b) show the appearance of the camera 1 (digital still camera; interchangeable lens camera) according to the first embodiment. FIG. 1A is a front perspective view, and FIG. 1B is a rear perspective view. As shown in FIG. 1A, the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5 which is an operation member for receiving an imaging operation from a user (photographer) is arranged on the camera housing 1B. As shown in FIG. 1B, on the back surface of the camera housing 1B, there is an eyepiece window frame 121 for the user to look into the display device 10 (display panel) described later contained in the camera housing 1B. An eyepiece lens 12 (eyepiece optical system) is arranged. The eyepiece window frame 121 surrounds the eyepiece lens 12 and projects to the outside (rear side) of the camera housing 1B with respect to the eyepiece lens 12. The eyepiece optical system may include a plurality of lenses. On the back surface of the camera housing 1B, operating members 41 to 43 that receive various operations from the user are also arranged. For example, the operation member 41 is a touch panel that accepts touch operations, the operation member 42 is an operation lever that can be pushed down in each direction, and the operation member 43 is a four-direction key that can be pushed in each of the four directions. The operation member 41 (touch panel) includes a display panel such as a liquid crystal panel, and has a function of displaying an image on the display panel.

FIG. 2 is a cross-sectional view of the camera 1 cut along the YZ plane formed by the Y-axis and the Z-axis shown in FIG. 1A, and shows a rough internal configuration of the camera 1.

In the photographing lens unit 1A, two lenses 101, 102, an aperture 111, an aperture drive unit 112, a lens drive motor 113, a lens drive member 114, a photo coupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. Etc. are included. The lens drive member 114 is composed of a drive gear or the like, and the photocoupler 115 detects the rotation of the pulse plate 116 linked to the lens drive member 114 and transmits the rotation to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 based on the information from the photo coupler 115 and the information from the camera housing 1B (lens drive amount information), and moves the lens 101 to adjust the focus position. change. The mount contact 117 is an interface between the photographing lens unit 1A and the camera housing 1B. Although two lenses 101 and 102 are shown for simplicity, more than two lenses are actually included in the photographing lens unit 1A.

The camera housing 1B includes an image sensor 2, a CPU 3, a memory unit 4, a display device 10, a display device drive circuit 11, and the like. The image sensor 2 is arranged on the planned image plane of the photographing lens unit 1A. The CPU 3 is a central processing unit of the microcomputer and controls the entire camera 1. The memory unit 4 stores an image or the like captured by the image sensor 2. The display device 10 is composed of a liquid crystal or the like, and displays an captured image (subject image) or the like on the screen (display surface) of the display device 10. The display device drive circuit 11 drives the display device 10.

The user can see the screen of the display device 10 through the eyepiece window frame 121 and the eyepiece lens 12. Specifically, as shown in FIG. 3, the eyepiece lens 12 forms a virtual image 300 in which the display device 10 (screen) is magnified at a position about 50 cm to 2 m away from the eyepiece lens 12. In FIG. 3, the virtual image 300 is formed at a position 1 m away from the eyepiece lens 12. The user can visually recognize the virtual image 300 by looking into the eyepiece window frame 121.

The camera housing 1B also includes light sources 13a and 13b, an optical divider 15, a light receiving lens 16, an eye image sensor 17, and the like. The light sources 13a and 13b are light sources conventionally used in a single-lens reflex camera or the like for detecting the line-of-sight direction from the relationship between the reflected image (corneal reflex image) due to the corneal reflex of light and the pupil, and the user's eyeball 14 It is a light source for illuminating. Specifically, the light sources 13a and 13b are infrared light emitting diodes or the like that emit infrared light that is insensitive to the user, and are arranged around the eyepiece lens 12. The optical image of the illuminated eyeball 14 (eyeball image; an image of reflected light emitted from the light sources 13a and 13b and reflected by the eyeball 14) passes through the eyepiece 12 and is reflected by the light divider 15. Then, the eyeball image is formed by the light receiving lens 16 on the eye image sensor 17 in which a array of photoelectric elements such as a CCD is two-dimensionally arranged. The light receiving lens 16 positions the pupil of the eyeball 14 and the image sensor 17 for the eye in a conjugate imaging relationship. The line-of-sight direction of the eyeball 14 (viewpoint on the screen of the display device 10) is detected from the position of the corneal reflex image in the eyeball image formed on the eye image sensor 17 by a predetermined algorithm described later.

FIG. 4 is a block diagram showing an electrical configuration in the camera 1. The line-of-sight detection circuit 201, the photometric circuit 202, the autofocus detection circuit 203, the signal input circuit 204, the display device drive circuit 11, the light source drive circuit 205, and the like are connected to the CPU 3. Further, the CPU 3 transmits a signal to the focus adjustment circuit 118 arranged in the photographing lens unit 1A and the aperture control circuit 206 included in the aperture drive unit 112 in the photographing lens unit 1A via the mount contact 117. To do. The memory unit 4 attached to the CPU 3 has a function of storing an image pickup signal from the image pickup element 2 and the eye image sensor 17, and a function of storing a line-of-sight correction parameter for correcting individual differences in the line of sight, which will be described later.

The line-of-sight detection circuit 201 A / D-converts the output (eye image of the eye) of the eye image sensor 17 in a state where the eyeball image is formed on the eye image sensor 17 (CCD-EYE), and the eye image detection circuit 201 performs A / D conversion thereof. The result is transmitted to the CPU 3. The CPU 3 extracts feature points required for line-of-sight detection from the eye image according to a predetermined algorithm described later, and calculates the user's line of sight (viewpoint on the screen of the display device 10) from the positions of the feature points.

The photometric circuit 202 performs amplification, logarithmic compression, A / D conversion, and the like of a signal obtained from the image sensor 2 that also serves as a photometric sensor, specifically, a luminance signal corresponding to the brightness of the field of view. The result is sent to the CPU 3 as the field brightness information.

The autofocus detection circuit 203 A / D-converts signal voltages from a plurality of detection elements (a plurality of pixels) used for phase difference detection included in the CCD in the image pickup element 2 and sends them to the CPU 3. .. The CPU 3 calculates the distance from the signals of the plurality of detection elements to the subject corresponding to each focus detection point. This is a known technique known as imaging surface phase-difference AF. In the first embodiment, as an example, there are focus detection points at each of the 180 locations on the imaging surface corresponding to the 180 locations shown in the field view image in the finder (screen of the display device 10) of FIG. 5 (a). And.

The signal input circuit 204 is turned on by the first stroke of the release button 5, and is turned on by the switch SW1 for starting the light measurement, distance measurement, line-of-sight detection operation, etc. of the camera 1 and the second stroke of the release button 5. The switch SW2 for starting the shooting operation is connected. The ON signals from the switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU3.

FIG. 5A is a view showing the field of view in the finder, and shows a state in which the display device 10 is operating (a state in which an image is displayed). As shown in FIG. 5A, the field of view in the finder includes a focus detection region 500, 180 AF point indexes 501, a field of view mask 502, and the like. Each of the 180 AF point indexes 501 is displayed superimposed on the through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to the focus detection point on the imaging surface. .. Further, among the 180 AF point indexes 501, the AF point index 501 corresponding to the current viewpoint A (estimated position) is highlighted by a frame or the like and displayed.

<Explanation of line-of-sight detection operation>
The line-of-sight detection method will be described with reference to FIGS. 6, 7 (a), 7 (b), and 8. FIG. 6 is a diagram for explaining the principle of the line-of-sight detection method, and is a schematic view of an optical system for performing line-of-sight detection. As shown in FIG. 6, the light sources 13a and 13b are arranged substantially symmetrically with respect to the optical axis of the light receiving lens 16 and illuminate the user's eyeball 14. A part of the light emitted from the light sources 13a and 13b and reflected by the eyeball 14 is focused on the eye image sensor 17 by the light receiving lens 16. FIG. 7A is a schematic view of an eye image (eyeball image projected on the eye image sensor 17) captured by the eye image sensor 17, and FIG. 7B is a CCD in the eye image sensor 17. It is a figure which shows the output intensity of. FIG. 8 shows a schematic flowchart of the line-of-sight detection operation.

When the line-of-sight detection operation starts, the light sources 13a and 13b emit infrared light toward the user's eyeball 14 in step S801 of FIG. The user's eyeball image illuminated by infrared light is imaged on the eye image sensor 17 through the light receiving lens 16 and photoelectrically converted by the eye image sensor 17. As a result, an electric signal of a processable eye image is obtained.

In step S802, the line-of-sight detection circuit 201 sends an eye image (eye image signal; electrical signal of the eye image) obtained from the eye image sensor 17 to the CPU 3.

In step S803, the CPU 3 obtains the coordinates of the points corresponding to the corneal reflex images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S802.

The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, the corneal reflex images Pd and Pe formed by a part of the infrared light reflected on the surface of the cornea 142 are condensed by the light receiving lens 16 and imaged on the eye imaging element 17 to form an ocular image. It becomes the corneal reflex image Pd', Pe'. Similarly, the luminous flux from the ends a and b of the pupil 141 is also imaged on the image sensor 17 for the eye to become the pupil end images a'and b'in the eye image.

FIG. 7B shows the luminance information (luminance distribution) of the region α'in the eye image of FIG. 7A. In FIG. 7B, the horizontal direction of the eye image is the X-axis direction, the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In the first embodiment, the coordinates of the corneal reflection images Pd'and Pe'in the X-axis direction (horizontal direction) are Xd and Xe, and the coordinates of the pupil end images a'and b'in the X-axis direction are Xa and Xb. As shown in FIG. 7B, at the coordinates Xd and Xe of the corneal reflection images Pd'and Pe', an extremely high level of brightness can be obtained. In the region from the coordinates Xa to the coordinates Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by forming the luminous flux from the pupil 141 on the image sensor 17 for the eye), the coordinates Xd and Xe are excluded. , An extremely low level of brightness is obtained. Then, in the region of the iris 143 outside the pupil 141 (the region of the iris image outside the pupil image obtained by forming the luminous flux from the iris 143), a brightness intermediate between the above two types of brightness can be obtained. Specifically, in a region where the X coordinate (coordinate in the X-axis direction) is smaller than the coordinate Xa and a region where the X coordinate is larger than the coordinate Xb, a brightness intermediate between the above two types of brightness can be obtained.

From the luminance distribution as shown in FIG. 7B, the X-coordinates Xd and Xe of the corneal reflection images Pd'and Pe'and the X-coordinates Xa and Xb of the pupil end images a'and b'can be obtained. Specifically, the coordinates with extremely high brightness can be obtained as the coordinates of the corneal reflection images Pd'and Pe', and the coordinates with extremely low brightness can be obtained as the coordinates of the pupil end images a'and b'. .. Further, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16 is small, the pupil center image c'(obtained by forming a luminous flux from the pupil center c on the eye imaging element 17). The coordinates Xc (center of the pupil image) can be expressed as Xc≈ (Xa + Xb) / 2. That is, the coordinates Xc of the pupil center image c'can be calculated from the X coordinates Xa and Xb of the pupil edge images a'and b'. In this way, the coordinates of the corneal reflex images Pd'and Pe'and the coordinates of the pupil center image c'can be estimated.

In step S804, the CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is a magnification determined by the position of the eyeball 14 with respect to the light receiving lens 16, and can be obtained by using a function of the interval (Xd-Xe) between the corneal reflection images Pd'and Pe'.

In step S805, the CPU 3 calculates the rotation angle of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16. The X coordinate of the midpoint of the corneal reflex image Pd and the corneal reflex image Pe and the X coordinate of the center of curvature O of the cornea 142 substantially coincide with each other. Therefore, assuming that the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Occ, the angle of rotation θx of the eyeball 14 in the ZX plane (plane perpendicular to the Y axis) is as follows. It can be calculated by the formula 1 of. The rotation angle θy of the eyeball 14 in the ZZ plane (plane perpendicular to the X-axis) can also be calculated by the same method as the calculation method of the rotation angle θx.

β × Oc × SINθx≈{(Xd + Xe) / 2} -Xc ・ ・ ・ (Equation 1)

In step S806, the CPU 3 uses the rotation angles θx and θy calculated in step S805 to obtain (estimate) the user's viewpoint (the position where the line of sight is focused; the position where the user is looking) on the screen of the display device 10. To do). Assuming that the coordinates of the viewpoint (Hx, Hy) are the coordinates corresponding to the center of the pupil c, the coordinates of the viewpoint (Hx, Hy) can be calculated by the following equations 2 and 3.

Hx = m × (Ax × θx + Bx) ・ ・ ・ (Equation 2)
Hy = m × (Ay × θy + By) ・ ・ ・ (Equation 3)

The parameters m of the equations 2 and 3 are constants determined by the configuration of the finder optical system (light receiving lens 16 and the like) of the camera 1, and the rotation angles θx and θy are converted into the coordinates corresponding to the pupil center c on the screen of the display device 10. The conversion factor to be used. It is assumed that the parameter m is determined in advance and stored in the memory unit 4. The parameters Ax, Bx, Ay, and By are line-of-sight correction parameters that correct individual differences in the line of sight, and are acquired by performing a calibration operation described later. It is assumed that the parameters Ax, Bx, Ay, and By are stored in the memory unit 4 before the line-of-sight detection operation starts.

In step S807, the CPU 3 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 4, and finishes the line-of-sight detection operation.

<Explanation of calibration work>
As described above, the viewpoint can be estimated by acquiring the rotation angles θx and θy of the eyeball 14 from the eye image in the line-of-sight detection operation and converting the position of the pupil center c to the position on the screen of the display device 10. ..

However, it may not be possible to estimate the viewpoint with high accuracy due to factors such as individual differences in the shape of the human eyeball. Specifically, unless the line-of-sight correction parameters Ax, Ay, Bx, and By are adjusted to values suitable for the user, as shown in FIG. 5 (b), the actual viewpoint B and the estimated viewpoint C There will be a gap. In FIG. 5B, the user is gazing at a person, but the camera 1 erroneously presumes that the background is being gazed at, resulting in a state in which appropriate focus detection and adjustment cannot be performed.

Therefore, before the camera 1 performs imaging, it is necessary to perform calibration work, acquire a viewpoint correction parameter suitable for the user, and store it in the camera 1.

Conventionally, the calibration work is performed by highlighting a plurality of indexes having different positions as shown in FIG. 5C on the screen of the display device 10 and having the user see the indexes before imaging. .. Then, a technique of performing a line-of-sight detection operation when gazing at each index and obtaining a viewpoint correction parameter suitable for the user from a plurality of calculated viewpoints (estimated positions) and the coordinates of each index is known as a known technique. ing. If the position to be viewed by the user is suggested, the display method of the index is not particularly limited, and the graphic which is the index may be displayed, or the brightness or color of the image (captured image, etc.) may be changed. Indicators may be displayed.

<Explanation of the visual state in which the head is greatly translated and looked into the finder>
When looking into the finder of the camera, the user may use the camera without bringing the eyeballs sufficiently close to the finder due to circumstances such as wearing glasses. When the eyeball is not sufficiently close to the finder, the line of sight is blocked by the eyepiece window frame, etc., as described later, and the visible range in the finder is limited. The edge of the screen may not be visible. In such a case, the user tends to make a large translational movement of the head from the front of the finder and look into the finder from an angle because he / she tries to see the end portion. Since the translational movement distance of the head when looking into the finder from an angle is much larger than when the camera is used in the normal (manufacturer's recommended) usage method, a non-negligible error occurs in the viewpoint estimation result. ..

The state of the peep will be described with reference to FIGS. 9 (a), 9 (b), 10 (a), and 10 (b). 9 (a), 9 (b), 10 (a), and 10 (b) show a state in which the user visually recognizes the virtual image 300 of the display device 10 (screen) through the eyepiece window frame 121 and the eyepiece lens 12. It is a top surface schematic view seen from the Y-axis positive direction. For the sake of simplicity, the eyepiece 12 is omitted in FIGS. 9 (a), 9 (b), 10 (a), and 10 (b). As described with reference to FIG. 3, even in the states of FIGS. 9 (a), 9 (b), 10 (a), and 10 (b), the user actually displays the display device 10 with the eyepiece 12 (not shown). The virtual image 300 enlarged from the size is visually recognized. Usually, the adjustment is made so that the virtual image 300 is formed at a distance of about several tens of centimeters to 2 m from the eyepiece lens 12. In FIGS. 9 (a), 9 (b), 10 (a), and 10 (b), the virtual image 300 is formed at a position 1 m away from the eyepiece 12 (not shown).

In FIG. 9A, the user is approximately the virtual image 300 in a state where the center O'of the eyeball 14 is located at a position facing the center of the virtual image 300 (screen of the display device 10) and at a position where the optical axis of the eyepiece 12 passes. I am watching the center. The visual field range of the eyeball 14 (the range that can be visually recognized through the eyepiece window frame 121) is determined by the width of the eyepiece window frame 121 and the like, and in FIG. 9A, a straight line passing through the pupil 141 of the eyeball 14 and the edge of the eyepiece window frame 121. The range α sandwiched between the OA and the straight line OA'is the visual field range. In FIG. 9A, since the entire virtual image 300 is included in the visual field range α, the user can visually recognize the entire screen of the display device 10 (from one end to the other end).

However, in FIG. 9B, the distance between the eyeball 14 and the eyepiece window frame 121 is determined in FIG. 9A because the eyeglasses worn by the user are sandwiched between the eyeball 14 and the eyepiece window frame 121. The distance ΔL is larger than that of. Therefore, in FIG. 9B, the visual field range β1 sandwiched between the straight line OB passing through the pupil 141 and the end of the eyepiece window frame 121 and the straight line OB'is narrower than the visual field range α in FIG. 9A. As a result, only a part of the virtual image 300 is included in the visual field range β1, and the user cannot visually recognize the ranges γ1 and γ2 (ends of the virtual image 300) of the virtual image 300 that are not included in the visual field range β1. ..

In the first embodiment, the eyepiece window frame 121 is shown as a factor that limits the visual field range, but the present invention is not limited to this. For example, the eyepiece lens 12 is provided with a mask that limits light rays. The viewing range may be limited. The field of view may be limited by any structure (factor).

The actions often taken by the user who has fallen into the situation shown in FIG. 9 (b) will be described below with reference to FIGS. 10 (a) and 10 (b).

When the user wants to see the range γ1 that is not visible in the state of FIG. 9B, the user translates the eyeball 14 together with the head in the positive direction of the X-axis (downward of the paper surface) as shown in FIG. 10A. Tends to move. Due to such translational movement, the center O'of the eyeball 14 is displaced in the direction perpendicular to the optical axis of the eyepiece 12, and the visual field range is shifted from the range β1 to the pupil 141 of the eyeball 14 and the eyepiece window frame 121. It changes to the range β2 sandwiched between the straight line OC passing through the end and the straight line OC'. Specifically, the visual field range is closer to the negative direction of the X-axis (direction on the paper surface), and the visual field range β2 including the range γ1 that was not visible before the translational movement. This allows the user to see the range γ1. However, the range that cannot be seen in the downward direction of the paper surface extends from the range γ2 to the range γ2'.

Similarly, when the user wants to see the range γ2 that is not visible in the state of FIG. 9 (b), the user views the eyeball 14 together with the head in the negative direction of the X-axis (on the paper surface) as shown in FIG. 10 (b). ) Tends to make a large translational movement. Due to such translational movement, the center O'of the eyeball 14 is displaced in the direction perpendicular to the optical axis of the eyepiece 12, and the visual field range is shifted from the range β1 to the pupil 141 of the eyeball 14 and the eyepiece window frame 121. It changes to the range β3 sandwiched between the straight line OD passing through the end and the straight line OD'. Specifically, the visual field range is the range β3 including the range γ2 that was not visible before the translational movement because the visual field range is closer to the X-axis positive direction (downward of the paper surface). This allows the user to see the range γ2. However, the range that cannot be seen in the paper surface direction extends from the range γ1 to the range γ1'.

The eye images (eyeball images) before and after the translational movement of the eyeball 14 as described with reference to FIGS. 10 (a) and 10 (b) are schematically shown in FIGS. 11 (a) to 11 (c). Shown.

FIG. 11A corresponds to the state of FIG. 9B, and the position facing the center of the virtual image 300 (the screen of the display device 10) and the position where the optical axis of the eyepiece 12 passes are the centers O'of the eyeball 14. The eye image is shown in a state where the user is gazing at substantially the center of the virtual image 300. In FIG. 11A, the center of the pupil image and the center of the two Purkinje images (P image; corneal reflex image) substantially coincide with the center of the ocular image. In the first embodiment, it is assumed that the center of the eye image corresponds to the center of the screen of the display device 10, specifically, the optical axis of the eyepiece 12, but it is not necessary.

FIG. 11B shows an eye image in a state in which the eyeball 14 is largely translated in the positive direction of the X-axis and then looked into the eyepiece window frame 121, corresponding to the state of FIG. 10A. ..

In FIG. 10A, the amount of movement of the center O'of the eyeball 14 in the positive direction of the X-axis (the direction opposite to the desired direction) from the position of FIG. 9B in order to see the end in the negative direction of the X-axis. The position is translated by ΔB. Further, the eyeball 14 is rotated in the negative direction of the X-axis (the same direction as the desired direction), and the pupil center c is a position moved from the center of the eye O'in the negative direction of the X-axis by the amount of movement ΔW. There is. Assuming that the rotation angle of the eyeball 14 is the angle θt and the radius of rotation of the eyeball 14 is the radius R, the movement amount ΔW of the pupil 141 can be expressed as ΔW = R × sinθt.

Then, due to the movement of the eyeball 14 as described with reference to FIG. 10A, the visual state on the eye image is compared with the normal visual state (visual state recommended by the manufacturer; for example, the visual state in which the finder is viewed from the front). The pupil image and the Purkinje image move in translation with a much larger amount of movement. Specifically, the eyeball 14 rotates in the negative direction of the X-axis and the pupil 141 moves with the amount of movement ΔW, so that the pupil image moves in the negative direction of the X-axis with the amount of movement ΔW × β (“β” is the eyeball). Image image magnification (lens magnification of eyepiece 12). Then, the eyeball 14 translates in the positive direction of the X-axis with the amount of movement ΔB, so that the pupil image moves in the positive direction of the X-axis with the amount of movement ΔB × β. Here, the movement amount ΔB × β corresponding to the translational movement of the eyeball 14 greatly exceeds the movement amount ΔW × β corresponding to the rotation of the eyeball 14. Therefore, as shown in FIG. 11B, the pupil image is in the negative X-axis direction (the same direction as the desired direction) with respect to the center of the eye image (the position corresponding to the center of the screen of the display device 10). It will be located at a location that has moved significantly in the positive direction of the X-axis (rightward of the paper), which is the opposite of (leftward of the paper). Similarly, a large translational movement occurs in the Purkinje image.

FIG. 11 (c) shows an eye image in a state in which the eyeball 14 is largely translated in the negative direction of the X-axis and then the inside of the eyepiece window frame 121 is looked into, corresponding to the state of FIG. 10 (b). ..

In FIG. 10 (b), the amount of movement of the center O'of the eyeball 14 in the negative direction of the X-axis (the direction opposite to the desired direction) from the position of FIG. 9 (b) in order to see the end in the positive direction of the X-axis. The position is translated by ΔB. Further, the eyeball 14 is rotated in the positive direction of the X-axis (the same direction as the desired direction), and the pupil center c is a position moved from the center of the eye O'in the positive direction of the X-axis by the amount of movement ΔW. There is. As described above, assuming that the rotation angle of the eyeball 14 is the angle θt and the radius of rotation of the eyeball 14 is the radius R, the movement amount ΔW of the pupil 141 can be expressed as ΔW = R × sinθt.

Then, even with the movement of the eyeball 14 as described with reference to FIG. 10B, the pupil image and the Purkinje image are translated and moved on the eye image with a much larger amount of movement than in the normal visual state. Specifically, the eyeball 14 rotates in the positive direction of the X-axis and the pupil 141 moves with the amount of movement ΔW, so that the pupil image moves in the positive direction of the X-axis with the amount of movement ΔW × β. Then, the eyeball 14 translates in the negative direction of the X-axis with the amount of movement ΔB, so that the pupil image moves in the negative direction of the X-axis with the amount of movement ΔB × β. Here, too, the movement amount ΔB × β corresponding to the translational movement of the eyeball 14 greatly exceeds the movement amount ΔW × β corresponding to the rotation of the eyeball 14. Therefore, as shown in FIG. 11C, the pupil image is in the positive X-axis direction (the same direction as the desired direction) with respect to the center of the eye image (the position corresponding to the center of the screen of the display device 10). It will be located at a location that has moved significantly in the negative direction of the X-axis (leftward of the paper), which is opposite to the right of the paper. Similarly, a large translational movement occurs in the Purkinje image.

In this way, in the visual state in which the finder is viewed from an angle, the pupil image and the Purkinje image in the eye image are located at places that are not expected in the normal visual state. As a result, a non-negligible error occurs in the estimation result of the viewpoint by the line-of-sight detection operation of FIG.

Therefore, in the first embodiment, an oblique looking state (an oblique viewing state in which the finder is viewed from an angle; a viewing state in which the viewpoint cannot be estimated with high accuracy; a position in which the eyeball 14 corresponds to the center of the screen of the display device 10 in the eye image). (Deviation from the visual state) is detected. Then, when the oblique peeping state is detected, a predetermined process is performed to encourage the user to look into the finder from the front without looking into the finder from an oblique angle.

<Explanation of diagonal peeping detection method>
The distance from the center of the eye image to the pupil image in the eye image is the first feature in the oblique looking state in which the end of the screen (virtual image) of the display device 10 does not fit in the user's field of view and the end is viewed. Is significantly larger than the normal visual field. Therefore, in the first embodiment, the first condition that the difference between the center of the eye image (the position corresponding to the center of the screen of the display device 10) and the position of the pupil image in the eye image is larger than a predetermined threshold value is satisfied. Judge whether or not. Then, the state in which the first condition is satisfied is detected as an oblique looking state.

The predetermined threshold value is determined, for example, from the amount of movement of the pupil image that can occur in the eye image in a normal visual state. In a normal visual state, when looking at the edge of the screen of the display device 10, only the eyeball rotates with almost no movement of the head. Therefore, from the maximum rotation angle θmax of the eyeball 14, the rotation radius R of the eyeball 14 (the length from the eyeball center O'to the pupil center c in FIG. 6), and the imaging magnification β of the eyeball image, the pupil image in the eye image. The maximum movement amount β × R × sin θmax can be calculated. For example, this maximum movement amount β × R × sin θmax can be used as the predetermined threshold value.

In the first condition, attention may be paid to the Purkinje image in the eye image instead of the pupil image in the eye image. That is, it is determined whether or not the first condition that the difference between the center of the eye image and the position of the Purkinje image in the eye image is larger than a predetermined threshold value is satisfied, and the state in which the first condition is satisfied is determined. It may be detected as an oblique looking state. One of the pupil image and the Purkinje image may be focused on, or both may be focused on. A state in which one of the first condition regarding the pupil image and the first condition regarding the Purkinje image is satisfied may be detected as an oblique looking state, and a state in which both of them are satisfied may be detected as an oblique looking state. May be detected as.

<Explanation of camera operation>
The camera operation according to the first embodiment will be described with reference to the flowchart of FIG.

When the power of the camera 1 is turned on, in step S1201, the image sensor 2 starts acquiring a through image (visualization image), transmits an image signal of the through image to the CPU 3, and the CPU 3 receives the acquired through image. Is displayed on the display device 10. The user confirms the subject by viewing the through image displayed on the display device 10 through the eyepiece window frame 121 and the eyepiece lens 12. The power supply of the camera 1 is turned ON / OFF according to a user operation on the camera 1.

In step S1202, the CPU 3 determines whether or not to turn off the power of the camera 1, and if it is turned off, the process flow of FIG. 12 is terminated, and if it is not turned off, the process proceeds to step S1203.

In step S1203, the CPU 3 starts acquiring the eye image of the user who has begun to visually recognize the through image in step S1201, and performs the line-of-sight detection operation of FIG. The line-of-sight detection operation calculates the coordinates of the pupil image in the eye image, the coordinates of the Purkinje image in the eye image, and the coordinates of the viewpoint in the through image.

In step S1204, the CPU 3 determines whether or not the first condition is satisfied, specifically, whether or not the position of the pupil image detected by the line-of-sight detection operation in step S1203 is within a predetermined range. The predetermined range is a part of the eye image, and is a range from the center of the eye image to a position separated by the above-mentioned predetermined threshold value (predetermined distance). The CPU 3 proceeds to step S1205 when it is outside the predetermined range, and proceeds to step S1207 when it is within the predetermined range.

In step S1205, the CPU 3 determines that the current state is the oblique looking state. In step S1206, since the estimation result of the viewpoint by the line-of-sight detection operation in step S1203 contains a non-negligible error, the CPU 3 performs a process for eliminating (improving the visual state) (visual state improvement process). After that, the process is returned to step S1202, and the line-of-sight detection operation is performed again.

In the first embodiment, the CPU 3 gives a predetermined notification such as a warning regarding the visual state to the user by the visual state improvement process. For example, it can be estimated that the direction from the front position of the eyepiece window frame 121 toward the position of the user's head is the same as the direction from the center of the eye image toward the pupil image. Notifies the direction information that moves the head in the direction opposite to the direction of.

The visual state improvement process is not limited to the predetermined notification. For example, the visual state improving process may be any process as long as the user can be prompted to improve the visual state (looking into the eyepiece window frame 121 from the front). Specifically, the visual recognition state improvement process may be a reduction of the through image (visualization image). If the edge of the through image is brought closer to the center of the screen of the display device 10 by reducing the through image, the entire through image can be visually recognized without looking into the eyepiece window frame 121 from an angle, and the through image can be viewed diagonally. Can be resolved.

Since the process of step S1207 is performed after the line-of-sight detection operation of S1203 is performed in a suitable visual state other than the oblique looking state, an accurate estimation result of the viewpoint is obtained at the time of the process of step S1207. Therefore, in step S1207, the CPU 3 superimposes and displays the accurate estimation result (frame showing the viewpoint; viewpoint frame) obtained in step S1203 on the through image. As a result, the display as shown in FIG. 5A is performed, and the current viewpoint A (estimated position) can be transmitted to the user. Instead of the viewpoint frame, a point indicating the viewpoint may be displayed.

In step S1208, the CPU 3 waits for a predetermined time.

In step S1209, the CPU 3 determines whether or not the release button 5 is pressed (half-pressed) by the user to turn on the switch SW1. For example, when the user agrees to focus at the position of the viewpoint frame (frame indicating the estimated viewpoint) displayed overlaid on the through image, the user presses the release button 5 halfway and turns on the switch SW1. To do. The CPU 3 proceeds with the process in step S1210 when the switch SW1 is turned on, and returns the process to step S1203 when the switch SW1 is not turned on to re-estimate the viewpoint.

In step S1210, the CPU 3 performs a distance measuring operation at the position of the current line-of-sight frame, and notifies the user that the distance measuring operation has been performed by highlighting such as changing the color of the line-of-sight frame.

In step S1211, the CPU 3 drives the lens 101 in the photographing lens unit 1A according to the distance measurement result obtained in step S1210. As a result, focusing at the position of the viewpoint frame displayed overlaid on the through image is realized.

In step S1212, the CPU 3 determines whether or not the release button 5 is further pushed (fully pressed) by the user to turn on the switch SW2. For example, when the user agrees to shoot at the current focusing position, the release button 5 is fully pressed to turn on the switch SW2. The CPU 3 proceeds to step S1213 when the switch SW2 is turned on, and returns to step S1209 when the switch SW2 is not turned on.

In step S1213, the CPU 3 performs a shooting operation to store the image signal acquired by the image sensor 2 in the memory unit 4.

In step S1214, the CPU 3 displays the image (captured image) stored in the memory unit 4 in step S1213 on the display device 10 for a predetermined time, and returns the process to step S1202.

<Summary>
As described above, according to the first embodiment, whether or not the first condition that the difference between the center of the eye image and the position of the pupil image (or Purkinje image) in the eye image is larger than a predetermined threshold value is satisfied. It is possible to detect an oblique looking state with a simple configuration of determining whether or not. Then, when the oblique peeping state is detected, the user can be urged to improve the visual state, and an accurate (highly accurate) estimation result of the viewpoint can be obtained.

<< Example 2 >>
Hereinafter, Example 2 of the present invention will be described. In the following, the description of the same points (configuration, processing, etc.) as in the first embodiment will be omitted, and the points different from the first embodiment will be described. In the first embodiment, an example of detecting an oblique looking state while displaying a through image has been described. In the second embodiment, an example of detecting an oblique looking state during the calibration operation will be described. Further, in the second embodiment, an example of determining the image display range (range in which the image is displayed) in the finder based on the detection result of the oblique looking state will be described. Specifically, based on the detection result of the oblique looking state, an example of determining the image display range of the display device 10 so as to encourage the user to look into the finder from the front without looking into the finder from an angle. explain.

As described above, the calibration work is performed by highlighting a plurality of indexes having different positions on the screen of the display device 10 before imaging and having the user see the indexes. For example, as shown in FIG. 13A, indicators are displayed at five locations on the screen of the display device 10, above the center, below the center, on the left side of the center, and on the right side of the center. Will be done. In the second embodiment, it is assumed that the indicators are displayed one by one, but all five indicators may be displayed and the emphasized indicators may be sequentially switched among the five indicators. The user will see the displayed metric (highlighted metric). That is, in the calibration work, the position to be seen by the user is specified by the index.

In the calibration work, when the eyeball 14 is not sufficiently close to the eyepiece window frame 121 and a part of the screen of the display device 10 is not within the user's field of view, the user is positioned at the edge of the screen. It may not be possible to visually recognize the index to be used. For example, FIG. 13 (a)
In the case of the visual field range β1 (the visual field range corresponding to the viewing of the eyepiece window frame 121 from the front), the user can see the three indexes of the center, the upper side, and the lower side of the screen. However, I cannot see the two indicators on the left and right.

In this case, the user takes an oblique peeping state in order to see the index outside the visual field range β1. In the oblique looking state of looking at the left index (first index) shown in FIG. 13 (a), an eye image as shown in FIG. 11 (c) is obtained, and the right side shown in FIG. 13 (a) is obtained. In the oblique looking state of looking at the index (second index) of No. 11 (b), an eye image as shown in FIG. 11 (b) can be obtained.

The eye image of FIG. 11C will be described. In order to see the first index at the left end (the end in the positive direction of the X-axis) of the screen of the display device 10, the user shifts the head in the negative direction of the X-axis (opposite to the desired direction) as shown in FIG. 10 (b). After translating in the direction of (to the right), the inside of the eyepiece window frame 121 is looked into. Therefore, in the eye image of FIG. 11C, the pupil image and the Purkinje image are located on the side opposite to the side of the first index (the side in the positive direction of the X-axis) (the side in the negative direction of the X-axis).

The eye image of FIG. 11B will be described. In order to see the second index at the right end (the end in the negative direction of the X-axis) of the screen of the display device 10, the user shifts the head in the positive direction of the X-axis (opposite to the desired direction) as shown in FIG. 10 (a). After translating in the direction of (to the left), the inside of the eyepiece window frame 121 is looked into. Therefore, in the eye image of FIG. 11B, the pupil image and the Purkinje image are located on the side opposite to the side of the second index (the side in the negative direction of the X axis) (the side in the positive direction of the X axis).

In this way, in the eye image in the oblique looking state, the pupil image and the Purkinje image are located on the opposite side of the index to be viewed by the user.

<Explanation of diagonal peeping detection method>
As described above, the second feature of the oblique looking state is that the pupil image is located on the opposite side of the index to be seen by the user in the eye image. Therefore, in the second embodiment, the direction from the center of the eye image toward the pupil image in the eye image is opposite to the direction from the center of the eye image toward the index corresponding position (position corresponding to the index to be viewed) in the eye image. The second condition that there is is used. Even with a simple configuration of determining whether or not the second condition is satisfied, the oblique looking state can be detected. Specifically, a state in which the second condition is satisfied can be detected as an oblique looking state. In the second embodiment, it is determined whether or not both the first condition described in the first embodiment and the second condition described above are satisfied, and a state in which both the first condition and the second condition are satisfied is determined. Detected as an oblique looking state. By using both the first condition and the second condition, it is possible to detect the oblique looking state with higher accuracy than in the first embodiment. Only one of the first condition and the second condition may be used.

In the calibration work, as described above, the position to be seen by the user is specified by the index. Therefore, it can be said that the second condition is a preferable condition for detecting the oblique looking state during the calibration work. However, if the position to be viewed by the user on the screen of the display device 10 is emphasized, the second condition can be preferably used to detect the oblique looking state other than during the calibration operation.

In the second condition, as in the first condition, the Purkinje image in the eye image may be focused on instead of the pupil image in the eye image. That is, it is determined whether or not the second condition that the direction from the center of the eye image toward the Purkinje image in the eye image is opposite to the direction from the center of the eye image toward the index corresponding position in the eye image is satisfied. You may. Then, a state in which the second condition regarding the Purkinje image is satisfied may be detected as an oblique looking state. One of the pupil image and the Purkinje image may be focused on, or both may be focused on. All conditions for pupillary images (1st and 2nd conditions) and all conditions for Purkinje images (1st and 2nd conditions)
A state in which one of the conditions) is satisfied may be detected as an oblique looking state. A state in which all the conditions related to the pupil image and all the conditions related to the Purkinje image are satisfied may be detected as an oblique looking state. A state in which some conditions related to the pupil image (one of the first condition and the second condition) and the remaining conditions related to the Purkinje image (the other of the first condition and the second condition) are satisfied is defined as an oblique looking state. It may be detected.

In the second embodiment, an example in which the two indicators on the left side and the right side of the screen of the display device 10 cannot be seen and the oblique peeping state is taken will be described, but the upper side of the screen of the display device 10 In some cases, an oblique peeping state may be taken without being able to see the index such as the lower side. Such an oblique looking state can also be detected by using the first condition and the second condition.

<Explanation of how to improve the visual condition>
As described above, when the line of sight is blocked by the eyepiece window frame or the like and the end portion of the display device 10 (screen) cannot be seen, the oblique looking state is taken. Therefore, in the second embodiment, when the oblique looking state is detected, the image display range of the display device 10 is reduced from the current range and set.

FIG. 14A shows an image display state before detecting the oblique looking state. A schematic diagram of the visual state at this time is shown in FIG. 15 (a). The field of view is limited by the eyepiece window frame 121, and only the range β1 which is a part of the virtual image 300 of the display device 10 (screen) is in the field of view. Therefore, the ranges γ1 and γ2, which are a part of the virtual image 300, are not visible to the user. An oblique peeping state is taken to see the ranges γ1 and γ2, and a non-negligible error occurs in the estimation result of the viewpoint.

Therefore, in the second embodiment, as shown in FIG. 14B, the image display range of the display device 10 is reduced to the range β1 visible to the user. FIG. 15B is a schematic view of the visual state after reduction. Since the image display range is reduced to the range β1 which is the user's visual field range, the user can see the entire image displayed on the display device 10 without taking an oblique looking state. In other words, it is possible to create a situation in which it is not necessary to take a diagonal peeping state (a situation in which the head does not need to be translated). Therefore, it is possible to encourage the user to improve the visual state (eliminate the oblique looking state), and to obtain an accurate (highly accurate) estimation result of the viewpoint.

In this way, by reducing and setting the image display range of the display device 10 based on the detection result of the oblique looking state, the oblique looking state can be suppressed and the estimation result of the viewpoint can be improved. An example will be described in which the visual field range β1 (the visual field range in the visual field state in which the oblique exclusion state is not detected; the visual field range in the visual field state in which the finder is viewed from the front) is used as the image display range. It may be narrower than the field of view β1. If at least a part of the visual field range β1 is determined as the image display range, the oblique looking state can be suppressed and the estimation result of the viewpoint can be improved. As shown in FIG. 14 (c), the image display range may be reduced so that the aspect ratio is maintained from the state of FIG. 14 (a).

<Explanation of camera operation>
The camera operation according to the second embodiment will be described with reference to the flowchart of FIG. The processing flow of FIG. 16 is started, for example, in response to a user operation instructing the start of calibration work. The above-mentioned visual field range β1 depends on the position of the eyeball 14 (distance between the eyeball 14 and the eyepiece window frame 121, etc.). Therefore, in the flowchart of FIG. 16, the visual field range β1 is estimated based on the detection result of the oblique looking state, and the image display range of the display device 10 is determined based on the estimated visual field range β1 (estimated visual field range).

In step S1601, the CPU 3 displays on the display device 10 a first index (an index on the left side of the center of the screen of the display device 10) to be watched by the user.

In step S1602, the CPU 3 waits for a predetermined time.

In step S1603, the CPU 3 determines whether or not the release button 5 is pressed (half-pressed) by the user to turn on the switch SW1. For example, the user presses the release button 5 halfway to turn on the switch SW1 in order to indicate that he has gazed at the first index. The CPU 3 proceeds to step S1604 when the switch SW1 is turned on, and returns to step S1602 when the switch SW1 is not turned on.

In step S1604, the CPU 3 performs the line-of-sight detection operation shown in FIG. The line-of-sight detection operation calculates the coordinates of the pupil image in the eye image, the coordinates of the Purkinje image in the eye image, and the coordinates of the viewpoint on the screen of the display device 10.

In step S1605, the CPU 3 determines whether or not the first condition is satisfied, specifically, whether or not the position of the pupil image detected by the line-of-sight detection operation in step S1604 is within a predetermined range. The CPU 3 proceeds to step S1606 when it is outside the predetermined range, and proceeds to step S1609 when it is within the predetermined range.

In step S1606, the CPU 3 determines whether or not the second condition is satisfied, specifically, the direction from the center of the eye image toward the pupil image in the eye image is the negative X-axis direction (from the center of the screen of the display device 10 to the thirth). 1 It is determined whether or not the X-axis is in the reverse direction of the forward direction toward the index. The CPU 3 proceeds to step S1607 when the pupil image is located on the negative side of the X-axis, and proceeds to step S1609 when the pupil image is located on the positive side of the X-axis.

In step S1607, the CPU 3 determines that the current state is an oblique looking state. In step S1608, the CPU 3 visually recognizes because the estimation result of the viewpoint by the line-of-sight detection operation in step S1604 contains a non-negligible error and cannot perform appropriate calibration (obtaining an appropriate line-of-sight correction parameter). Performs state improvement processing. After that, the process is returned to step S1602, and the line-of-sight detection operation is performed again.

In the second embodiment, as shown in FIG. 13B, the CPU 3 brings the first index closer to the center of the screen of the display device 10 by the visual state improvement process in step S1607. Further, the CPU 3 reduces the estimated field of view range (the field of view range estimated as the field of view range β1) from the default range (for example, the entire range of the screen). Specifically, the CPU 3 reduces the estimated visual field range from the left side so that the position on the left side of the estimated visual field range becomes the position of the first index. Even in the state of FIG. 13B, since the first index is arranged outside the visual field range β1, the user takes an oblique looking state. Therefore, the visual field improvement process of step S1607 is performed again, and as shown in FIG. 13C, the first index is arranged inside the visual field range β1, and the estimated visual field range is reduced. As a result, the first index can be visually recognized without looking into the inside of the eyepiece window frame 121 from an angle, and the oblique looking state can be eliminated. As a result, when the oblique looking state is resolved, the left end of the visual field range β1 becomes the left end of the estimated visual field range (the end in the positive direction of the X-axis).

The visual state improvement process is not limited to the above process. For example, the visual state improvement process may be any process as long as the user can be urged to improve the visual state so that appropriate calibration can be performed. Specifically, as described in the first embodiment, the visual recognition state improvement process may be a warning that the state is an oblique looking state.

In step S1609, the CPU 3 displays on the display device 10 a second index (an index on the right side of the center of the screen of the display device 10) to be watched by the user. At this time, the first index is hidden, assuming that the processing related to the first index is completed.

In step S1610, the CPU 3 waits for a predetermined time.

In step S1611, the CPU 3 determines whether or not the release button 5 is pressed (half-pressed) by the user to turn on the switch SW1. For example, the user presses the release button 5 halfway to turn on the switch SW1 in order to indicate that he has gazed at the second index. The CPU 3 proceeds to step S1612 when the switch SW1 is turned on, and returns to step S1610 when the switch SW1 is not turned on.

In step S1612, the CPU 3 performs the line-of-sight detection operation shown in FIG. The line-of-sight detection operation calculates the coordinates of the pupil image in the eye image, the coordinates of the Purkinje image in the eye image, and the coordinates of the viewpoint on the screen of the display device 10.

In step S1613, the CPU 3 determines whether or not the first condition is satisfied, specifically, whether or not the position of the pupil image detected by the line-of-sight detection operation in step S1612 is within a predetermined range. The CPU 3 proceeds to step S1614 when it is outside the predetermined range, and proceeds to step S1617 when it is within the predetermined range.

In step S1614, the CPU 3 determines whether or not the second condition is satisfied, specifically, the direction from the center of the eye image toward the pupil image in the eye image is the X-axis positive direction (from the center of the screen of the display device 10 to the thirth). 2 It is determined whether or not the X-axis is in the opposite direction of the negative direction toward the index). The CPU 3 proceeds to step S1615 when the pupil image is located on the positive side of the X-axis, and proceeds to step S1617 when the pupil image is located on the negative side of the X-axis.

In step S1615, the CPU 3 determines that the current state is an oblique looking state. In step S1616, the CPU 3 visually recognizes because the estimation result of the viewpoint by the line-of-sight detection operation in step S1612 contains a non-negligible error and cannot perform appropriate calibration (obtaining an appropriate line-of-sight correction parameter). Performs state improvement processing. After that, the process is returned to step S1610, and the line-of-sight detection operation is performed again. In the second embodiment, similarly to the visual field improvement process in step S1607, the CPU 3 brings the second index closer to the center of the screen of the display device 10 and updates the estimated visual field range by the visual field improvement process in step S1615. Specifically, the estimated visual field range is reduced from the right side so that the position of the right side of the estimated visual field range becomes the position of the second index. Similar to the visual state improvement process of step S1607, the visual state improvement process of step S1615 is repeated until the oblique looking state is eliminated. As a result, when the oblique looking state is resolved, the right end of the visual field range β1 becomes the right end of the estimated visual field range (the end in the negative direction of the X-axis).

In step S1617, the CPU 3 determines that the processing related to all the indexes has been completed, and notifies the user of the success of the calibration. Further, the CPU 3 calculates the line-of-sight correction parameter from the estimation result of the viewpoint when gazing at each index, and stores the line-of-sight correction parameter in the memory unit 4. Although only the processing related to the first index (index on the left side) and the processing related to the second index (index on the right side) are shown in FIG. 16, the five indexes shown in FIG. 13A are actually shown. Processing is performed for each of.

As described above, in the second embodiment, the control of displaying the index on the edge of the screen of the display device 10 and then bringing the index closer to the center of the screen until the oblique peeping state is no longer detected is performed on a plurality of sides of the screen. Is done about. Then, in the screen of the display device 10, a range of a plurality of positions where the user can see the index without detecting the oblique looking state is estimated as the visual field range β1. The method for estimating the visual field range β1 is not limited to this. For example, while changing the position of the index among a plurality of predetermined positions, a plurality of positions where the oblique looking state is not detected are detected, and the range of the detected multiple positions (the minimum range including the plurality of positions). May be estimated as the field of view β1.

In step S1618, the CPU 3 sets the estimated field of view range as the image display range of the display device 10 and ends the processing flow of FIG.

<Summary>
As described above, according to the second embodiment, by further using the second condition in addition to the first condition, it is possible to detect the oblique looking state with higher accuracy than in the first condition.

If the calibration work is performed in the oblique looking state, the estimation result of the viewpoint contains a non-negligible error, and therefore an appropriate line-of-sight correction parameter cannot be obtained. According to the second embodiment, since the oblique looking state is detected and eliminated during the calibration operation, an appropriate line-of-sight correction parameter can be obtained. As a result, it is possible to improve the estimation accuracy of the viewpoint at the time of shooting after the calibration work.

Further, according to the second embodiment, the index is brought closer to the center of the screen of the display device 10 so that the oblique looking state is eliminated. Then, it is possible to specify the visual field range β1 corresponding to a suitable visual state (viewing in the eyepiece window frame 121 from the front) that is not in the diagonal looking state from a plurality of indexes after the diagonal looking state is resolved. it can. By using the specified field of view β1, the convenience of the camera 1 can be improved. For example, at the time of shooting after the calibration work, by displaying the visual image in a reduced size so as to be within the specified visual field range β1, the oblique looking state can be suppressed and the estimation accuracy of the viewpoint can be improved. Specifically, according to the second embodiment, the oblique peeping state is detected, and the visual field range in the state where the oblique peeping state is not detected is estimated based on the detection result of the oblique peeping state. Then, the image display range of the display device is determined based on the estimated field of view range. With such a simple configuration, it is possible to determine the image display range in which the entire image can be visually recognized without detecting the oblique looking state, and it is possible to eliminate the state in which the viewpoint cannot be estimated with high accuracy.

Although the example in which the field of view is estimated during the calibration work and the image display range is determined based on the estimated field of view during the calibration work or after the calibration work has been described, this may not be the case. The visual field range may be estimated during the period when the reblation work is not performed, and the image display range may be determined based on the estimated visual field range. At least one of the size and position of the image display range is determined based on the detection result of the oblique peeping state (presence or absence of the oblique peeping state, the position of the pupil image or Purkinje image in the eye image, etc.) without estimating the visual field range. You may change it.

<< Example 3 >>
Hereinafter, Example 3 of the present invention will be described. In the following, the description of the same points (configuration, processing, etc.) as in the second embodiment will be omitted, and the points different from the second embodiment will be described. In the second embodiment, an example of reducing the image display range has been described. In the third embodiment, an example of moving the image display range will be described.

<Explanation of how to improve the visual condition>
As described above, when the line of sight is blocked by the eyepiece window frame or the like and the end portion of the display device 10 (screen) cannot be seen, the oblique looking state is taken. Therefore, in the third embodiment, when the oblique looking state is detected, the image display range of the display device 10 is moved from the current range and set.

FIG. 17A shows an image display state before detecting an oblique looking state. FIG. 18A is a schematic view of the visual state at this time. The field of view is limited by the eyepiece window frame 121, and only the range β1 which is a part of the virtual image 300 of the display device 10 (screen) is in the field of view. Therefore, the ranges γ1 and γ2, which are a part of the virtual image 300, are not visible to the user. An oblique peeping state is taken to see the ranges γ1 and γ2, and a non-negligible error occurs in the estimation result of the viewpoint.

Therefore, in the third embodiment, as shown in FIG. 17B, the right end (the end in the negative direction of the X-axis) of the image display range of the display device 10 coincides with the right end of the range β1 visible to the user. , The image display range of the display device 10 is moved to the left (X-axis positive direction). FIG. 18B is a schematic view of the visual state after the movement. By moving the image display range so that the end of the image display range in the negative direction of the X-axis is included in the field of view β1, the user can view the image displayed on the display device 10 without taking an oblique peeping state. You can see the right end (the end in the negative direction of the X-axis). In other words, as a situation when the user wants to see the right edge of the image, it is possible to create a situation in which it is not necessary to take a diagonal looking state (a situation in which the head does not need to be translated). Therefore, it is possible to encourage the user to improve the visual state (eliminate the oblique looking state), and to obtain an accurate (highly accurate) estimation result of the viewpoint.

As shown in FIG. 17C, the image display of the display device 10 is such that the left end (the end in the positive direction of the X-axis) of the image display range of the display device 10 coincides with the left end of the range β1 visible to the user. The range may be moved to the right (negative X-axis direction). FIG. 18 (c) is a schematic view of the visual state after the movement. By moving the image display range so that the edge of the image display range in the positive direction of the X-axis is included in the field of view β1, the user can view the image displayed on the display device 10 without taking an oblique peeping state. You can see the left end (the end in the positive direction of the X-axis). In other words, as a situation when the user wants to see the left edge of the image, it is possible to create a situation in which it is not necessary to take a diagonal looking state (a situation in which the head does not need to be translated). Therefore, it is possible to encourage the user to improve the visual state (eliminate the oblique looking state), and to obtain an accurate (highly accurate) estimation result of the viewpoint.

By moving the image display range in the direction opposite to the direction the user wants to see in this way, it is possible to suppress the oblique looking state and improve the estimation result of the viewpoint. An example of moving the image display range so that the edge of the image display range coincides with the edge of the field of view β1 has been described. However, if the image display range is moved to include at least a part of the field of view β1, the image is displayed. The movement direction and movement amount of the range are not particularly limited. If the field of view β1 is estimated by the method described in Example 2, the image is such that the image display range includes at least a part of the field of view β1 based on the estimated field of view β1 (estimated field of view). The display range can be moved. Then, by moving the image display range in this way, it is possible to suppress the oblique looking state and improve the estimation result of the viewpoint.

The movement of the image display range in the positive direction of the X-axis is preferable when the user wants to see the right end of the image (the end in the negative direction of the X-axis), and the user moves the image display range in the negative direction of the X-axis. This is preferable when you want to see the left edge of the image (the edge in the positive direction of the X-axis). Therefore, when the diagonal peeping state when the user wants to see the right edge of the image is detected, the image display range is moved in the positive direction of the X-axis, and the diagonal peeping state when the user wants to see the left edge of the image. When is detected, the image display range may be moved in the negative direction of the X-axis. That is, the moving direction and the moving amount of the image display range may be determined based on the detection result of the oblique looking state. When the pupil image or Purkinje image is located at a position largely moved in the positive direction of the X-axis with respect to the center of the eye image, the user can determine that he / she wants to see the right edge of the image. Then, the pupil image and the Purkinje image are X with respect to the center of the eye image.
If it is located in a location that has moved significantly in the negative axis direction, it can be determined that the user wants to see the left edge of the image.

<Summary>
As described above, according to the third embodiment, the image display range of the display device is moved based on the estimated field of view range. With such a simple configuration, it is possible to determine the image display range in which the entire image can be visually recognized without detecting the oblique looking state, and it is possible to eliminate the state in which the viewpoint cannot be estimated with high accuracy.

<< Example 4 >>
Hereinafter, Example 4 of the present invention will be described. In the following, the description of the same points (configuration, processing, etc.) as in the second embodiment will be omitted, and the points different from the second embodiment will be described. In the fourth embodiment, another example of detecting the oblique looking state during the calibration operation will be described.

The plurality of indexes displayed in the calibration operation include two indexes that sandwich the center of the screen of the display device 10. Specifically, as shown in FIG. 13A, the first index (index on the left side) and the second index (index on the right side) sandwich the center of the screen of the display device 10. Then, as described in Example 2, in the eye image obtained in the oblique looking state of looking at the first index, the pupil image and the Purkinje image are opposite to the side of the first index (the side in the positive direction of the X-axis). It is located on the side of (the side in the negative direction of the X axis). On the other hand, in the eye image obtained in the oblique looking state of looking at the second index, the pupil image and the Purkinje image are on the side opposite to the side of the second index (the side in the negative direction of the X axis) (in the positive direction of the X axis). Located on the side). That is, the pupil image and the Purkinje image in the eye image when the first index is viewed are located on the opposite side to those when the second index is viewed.

<Explanation of diagonal peeping detection method>
As the third feature of the oblique looking state, as described above, the pupil image and the Purkinje image in the eye image when the first index is viewed are located on the opposite side to those when the second index is viewed. The feature is mentioned. Therefore, in the fourth embodiment, the third condition is used that the direction from the center of the eye image toward the pupil image in the eye image is opposite between the case where the first index is viewed and the case where the second index is viewed. .. Even with a simple configuration of determining whether or not the third condition is satisfied, the oblique looking state can be detected. Specifically, a state in which the third condition is satisfied can be detected as an oblique looking state. In the fourth embodiment, it is determined whether or not both the first condition and the second condition described in the second embodiment and the third condition described above are satisfied, and the first condition, the second condition, and the third condition are satisfied. A state in which all the conditions are satisfied is detected as an oblique looking state. By using all of the first condition, the second condition, and the third condition, the oblique looking state can be detected with higher accuracy than in the second embodiment. The first condition and the third condition may be used without using the second condition. Only the third condition may be used.

In the calibration work, as described above, the position to be seen by the user is specified by the index. Therefore, it can be said that the third condition is a preferable condition for detecting the oblique looking state during the calibration work. However, if the position to be viewed by the user on the screen of the display device 10 is emphasized, the third condition can be preferably used in order to detect the oblique looking state other than during the calibration operation.

In the third condition, as in the first condition and the second condition, the Purkinje image in the eye image may be focused on instead of the pupil image in the eye image. That is, whether or not the third condition that the direction from the center of the eye image toward the Purkinje image in the eye image is opposite between the case where the first index is viewed and the case where the second index is viewed is satisfied. A state in which the third condition is satisfied may be detected as an oblique looking state. One of the pupil image and the Purkinje image may be focused on, or both may be focused on. One of all the conditions for the pupil image (first condition, second condition, and third condition) and all the conditions for the Purkinje image (first condition, second condition, and third condition) are satisfied. The state may be detected as an oblique looking state. A state in which all the conditions related to the pupil image and all the conditions related to the Purkinje image are satisfied may be detected as an oblique looking state. A state in which some conditions relating to the pupil image and the remaining conditions relating to the Purkinje image are satisfied may be detected as an oblique looking state.

In the fourth embodiment, the two indexes on the left side and the right side of the screen of the display device 10 cannot be seen, and a diagonal peeping state is taken, and the first index is used as two indexes sandwiching the center of the screen. An example of using (the index on the left side) and the second index (the index on the right side) will be described. However, the two indexes sandwiching the center of the screen of the display device 10 are not limited to this. For example, the two indexes sandwiching the center of the screen of the display device 10 may be two indexes on the upper side and the lower side of the screen. Thereby, the oblique peeping state taken without being able to see the two indexes on the upper side and the lower side of the screen of the display device 10 can be detected by using the third condition.

<Explanation of camera operation>
The camera operation according to the fourth embodiment will be described with reference to the flowchart of FIG. The processing flow of FIG. 19 is started, for example, in response to a user operation instructing the start of calibration work.

The processes of steps S1901 to S1904 are the same as the processes of steps S1601 to S1604 of FIG. 16, and the processes of steps S1905 to S1908 are the same as the processes of steps S1609 to S1612 of FIG.

In step S1909, the CPU 3 determines whether or not the first condition is satisfied during the display of the first index, specifically, the position of the pupil image during the display of the first index is determined based on the operation result of step S1904. Determine if it is within range. The CPU 3 proceeds to step S1910 when it is outside the predetermined range, and proceeds to step S1915 when it is within the predetermined range. The process of step S1909 is the same as the process of step S1605 of FIG.

In step S1910, based on the operation result of step S1904, the CPU 3 determines whether or not the second condition is satisfied during the display of the first index, specifically, the eye from the center of the eye image during the display of the first index. It is determined whether or not the direction toward the pupil image in the image is the negative X-axis direction. The CPU 3 proceeds to step S1911 when the pupil image is on the negative side of the X-axis, and proceeds to step S1915 when the pupil image is on the positive side of the X-axis. The process of step S1910 is the same as the process of step S1606 of FIG.

In step S1911, the CPU 3 determines whether or not the first condition is satisfied during the display of the second index, specifically, the position of the pupil image during the display of the second index is determined based on the operation result of step S1908. Determine if it is within range. The CPU 3 proceeds to step S1912 when it is outside the predetermined range, and proceeds to step S1915 when it is within the predetermined range. The process of step S1911 is the same process as the process of step S1613 of FIG.

In step S1912, the CPU 3 determines whether or not the third condition is satisfied based on the operation results of steps S1904 and S1908. Specifically, the CPU 3 determines whether or not the direction from the center of the eye image toward the pupil image in the eye image is opposite between the display of the first index and the display of the second index. If the direction from the center of the eye image toward the pupil image in the eye image is opposite between the display of the first index and the display of the second index, the CPU 3 proceeds to step S1913. On the other hand, if the direction from the center of the eye image toward the pupil image in the eye image is the same between the display of the first index and the display of the second index, the CPU 3 proceeds to step S1915.

The determination in step S1912 is to determine whether or not the second condition is satisfied during the display of the second index, specifically, from the center of the eye image during display of the second index to the pupil image in the eye image. It may be a judgment as to whether or not the heading direction is the positive direction of the X-axis. That is, in step S1912, the same processing as that of step S1614 of FIG. 16 may be performed.

In step S1913, the CPU 3 determines that at least one of the visible state of the first index and the visible state of the second index is an oblique looking state. In step S1914, the CPU 3 is in a visual state because at least one of the viewpoint estimated in step S1904 and the viewpoint estimated in step S1908 contains a non-negligible error and cannot perform appropriate calibration. Perform improvement processing. After that, the process is returned to step S1901, and the line-of-sight detection operation is performed again. In the fourth embodiment, similarly to the second embodiment, the CPU 3 brings at least one of the first index and the second index closer to the center of the screen of the display device 10 by the visual inspection state improvement process in step S1914, and updates the estimated visual field range. To do.

In step S1915, the CPU 3 determines that the processing related to all the indexes has been completed, and notifies the user of the success of the calibration. Further, the CPU 3 calculates the line-of-sight correction parameter from the estimation result of the viewpoint when gazing at each index, and stores the line-of-sight correction parameter in the memory unit 4. Although only the processing related to the first index (index on the left side) and the processing related to the second index (index on the right side) are shown in FIG. 19, the five indexes shown in FIG. 13A are actually shown. Processing is performed for each of.

In step S1916, the CPU 3 sets the estimated field of view range as the image display range of the display device 10 and ends the processing flow of FIG.

As described above, according to the fourth embodiment, by further using the third condition in addition to the first condition and the second condition, the oblique peeping state can be detected with higher accuracy than in the second embodiment. Further, an appropriate line-of-sight correction parameter can be obtained more reliably than in the second embodiment. As a result, the estimation accuracy of the viewpoint at the time of shooting after the calibration work can be surely improved as compared with the second embodiment.

<< Example 5 >>
Hereinafter, Example 5 of the present invention will be described. In the following, the description of the same points (configuration, processing, etc.) as in the first embodiment will be omitted, and the points different from the first embodiment will be described. When the user wears spectacles, the light from the light source that illuminates the user's eyeball may be reflected by the surface of the spectacles, pass through the eyepiece, and enter the image pickup element for the eye, and appear as a ghost in the eye image. is there. In the oblique peeping state, the user tilts his head to look into the finder, so that the ghost moves near the center of the eye image according to the tilt of the head. As a result, when the ghost overlaps with the pupil image or Purkinje image, the detection accuracy of the pupil image or Purkinje image is lowered, and the estimation accuracy of the viewpoint is lowered. In the fifth embodiment, an example focusing on such a problem will be described.

<Explanation of configuration>
FIG. 20 is a rear perspective view showing the appearance of the camera 1 (digital still camera; interchangeable lens camera). The front perspective view of the camera 1 is the same as that of the first embodiment (FIG. 1 (a)). As shown in FIG. 20, in the fifth embodiment, four light sources 13a to 13d for illuminating the user's eyeball are provided around the eyepiece 12.

FIG. 21 shows a rough internal configuration of the camera 1. FIG. 22 shows a state in which the user looks into the inside of the eyepiece window frame 121. FIG. 23 is a block diagram showing an electrical configuration in the camera 1. In FIGS. 21 to 23, four light sources 13a to 13d are provided as described above, and other than that, the same as in FIGS. 2 to 4.

In the fifth embodiment, as shown in FIG. 21, the user wears an optical member such as eyeglasses 144, and the optical member is attached to the eyeball 14 in a state where the user looks into the inside of the eyepiece window frame 121. It shall be located between the eyepiece window frames 121.

FIG. 24 is a schematic view of an eye image (eyeball image projected on the eye image sensor 17) captured by the eye image sensor 17. Since four light sources 13a to 13d are used, four Purkinje images (P image; corneal reflex image) are captured. The line of sight can be detected by the same principle as in Example 1 based on a combination of a plurality of Purkinje images (the combination is not particularly limited) by a plurality of light sources shifted in the vertical and horizontal directions with respect to the eyepiece.

<Explanation of the visual state in which the head is greatly translated and looked into the finder>
When the head is translated and viewed from an angle, the user often rotates (tilts) the eyeball together with the head, instead of rotating only the eyeball looking into the eyepiece window frame. If the user is wearing glasses and the user tilts his head to look into the finder, the glasses he is wearing will also tilt in the same direction as his head. As a result, of the infrared light that illuminates the eyeball, the ghost caused by the light that is reflected on the surface of the spectacles and is incident on the eye imager through the eyepiece moves to the vicinity of the center of the eye image according to the inclination of the spectacles. Then, the ghost overlaps with the pupil image and the Purkinje image near the center of the eye image, and hinders the detection of those images. If the detection accuracy of the pupil image or Purkinje image decreases, the estimation accuracy of the viewpoint also decreases.

This phenomenon will be described in more detail with reference to FIGS. 25 (a) to 25 (c). 25 (a) to 25 (c) show a state in which the user visually recognizes the virtual image 300 of the display device 10 (screen) through the eyepiece window frame 121 and the eyepiece lens 12 with the right eye (eyeball on the upper side of the paper surface). It is a top surface schematic view seen from the Y-axis positive direction. In FIG. 25A, the user is approximately the virtual image 300 in a state where the center O'of the eyeball 14 is located at a position facing the center of the virtual image 300 (screen of the display device 10) and at a position where the optical axis of the eyepiece 12 passes. I am watching the center. In FIG. 25 (a), the user cannot visually recognize the ranges γ1 and γ2 (ends of the virtual image 300).

When the user wants to see the range γ1 that is not visible in the state of FIG. 25 (a), the user translates the eyeball 14 together with the head in the positive direction of the X-axis (downward of the paper surface) as shown in FIG. 25 (b). Move. Similarly, when the user wants to see the range γ2 that is not visible in the state of FIG. 25 (a), the user views the eyeball 14 together with the head in the negative X-axis direction (on the paper) as shown in FIG. 25 (c). Make a large translational movement. At this time, not only the rotation of the eyeball 14 but also the head is often tilted from the state shown in FIG. 25 (a). In FIGS. 25 (b) and 25 (c), the head is tilted by an angle θh from the state of FIG. 25 (a). Further, in FIGS. 25 (a) to 25 (c), the user wears the spectacles 144, and the spectacles 144 also tilt in the same direction as the head according to the tilt of the head. As a result, the inclination of the optical axis of the spectacles 144 with respect to the optical axis of the eyepiece 12 changes, and the position of the ghost generated by reflecting the light from the light source on the surface of the spectacles 144 also changes accordingly.

The above-mentioned ghost is, for example, as shown in FIG. 26, the light emitted from the light sources 13a to 13d is reflected on the surface of an optical member such as eyeglasses, and the reflected light is for the eyes as shown by an arrow in FIG. 26. It is assumed that the light is incident on the image sensor 17. Although FIG. 26 illustrates the path of light from the light source 13a or the light source 13b, the light from the light sources 13c and 13d can also enter the eye image sensor 17.

FIG. 27A shows an example of an eye image in which a ghost is reflected. In FIG. 27A, four ghosts Ga to Gd corresponding to the four light sources 13a to 13d are reflected. Ghosts Ga to Gd are ghosts generated by reflecting the light emitted from the light sources 13a to 13d on the surface of the spectacles, and Purkinje generated by reflecting the light emitted from the light sources 13a to 13d on the corneal surface of the eyeball. Appears separately from the statue. The eye image of FIG. 27 (a) corresponds to the state of FIG. 25 (a). As shown in FIG. 27 (a), when the optical member such as eyeglasses worn by the user faces the front, the ghosts Ga to Gd appear substantially symmetrically with respect to the center of the eye image.

Here, when the spectacles are tilted in an oblique peeping state as the head is tilted, the ghost moves and an eye image as shown in FIG. 27B can be obtained. In FIG. 27 (b), all of the ghosts Ga to Gd move from the state of FIG. 27 (a) in the positive direction of the X-axis (to the right of the paper) due to the inclination of the glasses accompanying the inclination of the head. As a result, the ghost Ga by the light source 13a overlaps a part of the pupil image, and the part of the pupil image is hidden. If at least a part of the pupil image is hidden, the detection accuracy of the pupil image is lowered. If the detection accuracy of the pupil image is lowered, the estimation accuracy of the viewpoint is also lowered.

Therefore, in the fifth embodiment, among the plurality of light sources, a light source that generates a ghost that moves to the central portion (center and its vicinity) of the eye image is determined based on the detection result of the visual state, and the determined light source is turned off. To do. As a result, the occurrence of ghosts in the central portion of the ocular image can be suppressed, and the detection accuracy of the pupil image can be improved. As a result, the estimation accuracy of the starting point can be improved.

<Explanation of how to turn off the light source>
Regarding the process of improving various detection accuracy by suppressing the occurrence of ghosts near the center of the eye image by turning off the light source in the direction corresponding to the viewing direction when the oblique looking state is detected. This will be described in detail.

As in the change from the state of FIG. 28 (a) to the state of FIG. 28 (b), if it is possible to turn off the light source 13a to 13d that generates a ghost that moves to the center of the eye image, various types of light sources can be used. Ghosts that interfere with detection can be removed from the eye image.

In the example of FIG. 27 (b), since the ghost Ga caused by the light source 13a hinders various detections, the light source 13a may be turned off as shown in FIG. 28 (b). When the light source 13a is turned off, as shown in FIG. 27 (c), the ghost Ga disappears, and the pupil image including the portion where the ghost overlaps can be detected with high accuracy, and the detection accuracy of the pupil image and the line of sight deteriorates. Can be suppressed. At this time, the Purkinje image formed by the light source 13a also disappears, but since there are a plurality of other light sources forming the Purkinje image, the line-of-sight detection is performed using the plurality of Purkinje images by the plurality of light sources (light sources 13b to 13d). Can be done with high precision. In Example 5, four light sources 13a to 13d are used so that highly accurate line-of-sight detection can be performed even when some of the light sources are turned off. The number of light sources is not limited to four, and may be three or more. If two or more light sources are turned on, the plurality of light sources may be turned off. Line-of-sight detection is possible if two or more light sources are turned on.

However, the above-mentioned processing (appropriate extinguishing of the light source) is impossible unless the light source that generates the ghost that moves to the central portion of the ocular image is known.

Therefore, in the fifth embodiment, the tilting direction of the head, that is, the tilting direction of the eyeglasses is determined by detecting the oblique looking state by the method described in another embodiment, and the tilting direction is used on the eye image. The moving direction of the ghost is determined, and the light source to be turned off is determined from the moving direction.

As shown in FIG. 25 (b), both the head and the spectacles 144 are tilted in the clockwise direction on the paper surface in the oblique looking state in which the head is translated in the positive direction of the X-axis. On the other hand, as shown in FIG. 25 (c), in the oblique looking state in which the head is translated in the negative direction of the X-axis, both the head and the spectacles 144 are tilted in the counterclockwise direction on the paper surface. That is, in FIGS. 25 (b) and 25 (c), the tilt directions of the head and the glasses are opposite. Therefore, the tilting direction of the head and the glasses can be specified from the viewing direction. Then, the moving direction of the ghost can be known from the tilting direction of the spectacles, and the light source that generates the ghost approaching the vicinity of the center of the eye image can be specified. By turning off the specified light source as shown in FIG. 27 (b), it is possible to eliminate the ghost that hinders the detection of the pupil image and improve the detection accuracy of the pupil image.

<Explanation of camera operation>
The camera operation according to the fifth embodiment will be described with reference to the flowchart of FIG. Here, an example of detecting an oblique looking state by a method using the first condition that the difference between the center of the eye image and the position of the pupil image in the eye image is larger than a predetermined threshold value will be described. The oblique peeping state may be detected by another method such as the method using. When the power of the camera 1 is turned on, the camera operation of FIG. 29 starts.

The processes of steps S2901 to S2905 and S2908 to S2925 are the same as the processes of steps S1201 to S1205 and S1207 to S1214 of FIG. 12 (Example 1), respectively.

In step S2906 in which the oblique peeping state is detected, the CPU 3 specifies the peeping direction (head or spectacles) from the moving direction of the pupil image in the eye image, and specifies the light source to be turned off. After identifying the light source to be turned off, the process proceeds to step S2907.

A specific example of the process of step S2906 will be described. In the eye image of FIG. 30A, the pupil image is moving from a predetermined range to the right of the paper surface. In this case, it can be determined that the head is translated in the positive direction of the X-axis to look diagonally, that is, the state shown in FIG. 25 (b). Further, it can be determined that the tilting direction of the head and the glasses is the clockwise direction of the paper in FIG. 25 (b). Then, the light source to be turned off can be specified from the tilting direction of the glasses.

In the eyeball image of FIG. 30B, contrary to FIG. 30A, the pupil image is moving from a predetermined range to the left of the paper. In this case, it can be determined that the head is translated in the negative direction of the X-axis to look diagonally, that is, the state shown in FIG. 25 (c). Further, it can be determined that the tilting direction of the head and the glasses is the counterclockwise direction on the paper surface of FIG. 25 (c). Then, the light source to be turned off can be specified from the tilting direction of the glasses.

In step S2907, the CPU 3 turns off the light source specified in step S2906. Then, the process returns to the line-of-sight detection operation in step S2903. When returning to step S2903 through steps S295 to S2907, the line-of-sight detection operation is performed using a light source other than the light source extinguished in step S2907 among the plurality of light sources. Therefore, the line of sight can be detected with high accuracy without any ghost that hinders the detection of the pupil image.

<Summary>
As described above, according to the fifth embodiment, the light source that causes ghost generation near the center of the eye image is selected from the plurality of light sources and turned off based on the detection result of the oblique looking state. be able to. As a result, it is possible to suppress the occurrence of ghosts near the center of the eye image and improve the detection accuracy of the line of sight.

It should be noted that Examples 1 to 5 are merely examples, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of Examples 1 to 5 within the scope of the gist of the present invention. A configuration obtained by appropriately combining the configurations of Examples 1 to 5 is also included in the present invention.

In addition, the finder of the camera is taken as an example, but the present invention is not limited to this. For example, when performing line-of-sight detection on an HMD (head-mounted display) worn on the head to experience VR (virtual reality) or the like, the HMD shifts with respect to the head (eyeball) using the present invention. It is possible to detect the visible state. Such a visual state may occur when an unfamiliar user wears the HMD, or may occur due to the movement of the user after wearing the HMD. Similarly, the present invention can be applied to a glasses-type line-of-sight detection device such as an AR (augmented reality) glass. It is possible to acquire an eye image of an eye that sees a visual image through an eyepiece window frame such as an eyepiece window frame or a spectacle frame that limits the visual field, and for all electronic devices that estimate the viewpoint using the acquired eye image. Therefore, the present invention is applicable.

As described above, according to the first to fifth embodiments, in the camera, the HMD, and the eyeglass-type eye-gaze detection device, the user mainly involves the translational movement of the head and the tilt of the head, and the screen is viewed from an oblique direction. It is possible to provide an electronic device that detects the visual state of looking into it with a simple configuration. Further, it is possible to provide an electronic device that eliminates the oblique looking state by a simple configuration in which the image display range is determined based on the detection result of the oblique looking state. Further, by appropriately turning off the light source based on the detection result of the inclination of the optical member such as eyeglasses in the oblique looking state, it is possible to provide an electronic device that improves the accuracy of various detections.

<Other Examples>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1: Camera 3: CPU

Claims (24)

An electronic device capable of acquiring an eye image of an eye looking at the screen of a display means through an eye window frame.
A first estimation means for estimating the viewpoint of the eye on the screen based on the eye image,
Based on the position of the pupil image or the Purkinje image in the eye image, it is characterized by having a detecting means for detecting a deviation visual state in which the eye is deviated from a position corresponding to the center of the screen in the eye image. Electronics. The detection means determines a state in which a predetermined condition including a condition that the difference between the position of the pupil image or the Purkinje image and the position corresponding to the center of the screen is larger than a predetermined threshold value is satisfied. The electronic device according to claim 1, wherein the electronic device is detected as. The screen can be viewed through the eye window frame and the eyepiece optical system.
The electronic device according to claim 1 or 2, wherein the position corresponding to the center of the screen is a position corresponding to the optical axis of the eyepiece optical system. The electronic device according to any one of claims 1 to 3, further comprising the display means, the eye window frame, and an imaging means for imaging the eye. On the screen, the index to be seen by the eye is displayed while changing the position of the index.
In the eye image, the direction from the position corresponding to the center of the screen toward the pupil image or Purkinje image corresponds to the position where the index is displayed from the position corresponding to the center of the screen. The electron according to any one of claims 1 to 4, wherein a state in which a predetermined condition including a condition opposite to the direction toward the position is satisfied is detected as the shift visual state. machine. On the screen, a plurality of indicators to be seen by the eyes are displayed.
The plurality of indicators include two indicators that sandwich the center of the screen.
The detection means has opposite directions from the position corresponding to the center of the screen toward the pupil image or the Purkinje image between the case where one of the two indexes is viewed and the case where the other is viewed. The electronic device according to any one of claims 1 to 5, wherein a state in which a predetermined condition including the above condition is satisfied is detected as the shift visual state. The electron according to any one of claims 1 to 6, wherein the detection means detects the deviation visual state during a calibration operation for obtaining a parameter used for estimating the viewpoint. machine. The electronic device according to any one of claims 1 to 7, further comprising a processing means for performing a predetermined process when the misaligned visual state is detected. The electronic device according to claim 8, wherein the predetermined process is a process for urging the user to eliminate the misaligned visual state. The electronic device according to claim 8 or 9, wherein the predetermined processing is a predetermined notification. The electronic device according to claim 8 or 9, wherein the predetermined process is a reduction of an image to be displayed on the screen. On the screen, the index to be seen by the eye is displayed.
The electronic device according to claim 8 or 9, wherein the predetermined process is a process of bringing the position of the index closer to the center of the screen. Claim 8 or 9 is characterized in that the predetermined process is a process of determining an image display range, which is a range for displaying an image in the screen, based on a shift visual state detected by the detection means. Electronic equipment described in. A control means for controlling the index to be seen by the eye so as to be displayed on the screen while changing the position of the index.
A second estimation of the screen, in which a range of a plurality of positions where the index can be seen by the eye without detecting the shift visual state is estimated as a visual field range in a state where the shift visual state is not detected. Means and
Have more
The electronic device according to claim 13, wherein the processing means determines the image display range based on the field of view range estimated by the second estimation means. The electronic device according to claim 14, wherein the index is an index for a calibration operation for obtaining a parameter used for estimating the viewpoint. The electronic device according to claim 14 or 15, wherein the processing means determines at least a part of the visual field range as the image display range. The electronic device according to claim 14 or 15, wherein the processing means determines the image display range so that the image display range moves including at least a part of the field of view. After displaying the index on the edge of the screen, the control means controls the plurality of sides of the screen so that the index is brought closer to the center of the screen until the deviation visual state is no longer detected. The electronic device according to any one of claims 14 to 17, which is characteristic. The predetermined process is
Based on the deviation visual recognition state detected by the detection means, the tilt direction of the optical member located between the eye window frame and the eye is detected from the state in which the deviation visual observation state is not detected.
The electronic device according to claim 8, further comprising a process of turning off a part of the plurality of light sources that illuminate the eyes according to the detected tilt direction. The electronic device according to claim 19, wherein the processing means detects the tilting direction based on the position of the pupil image or the Purkinje image in the eye image in the shifted visual state detected by the detecting means. .. The plurality of light sources are three or more light sources provided at three or more positions deviated from the optical axis of the eyepiece optical system provided on the eye window frame.
The electronic device according to claim 19 or 20, wherein the processing means turns off one or more light sources so that the two or more light sources turn on. It is a control method of an electronic device capable of acquiring an eye image obtained by capturing an image of an eye viewing the screen of a display means through an eye window frame.
A step of estimating the viewpoint of the eye on the screen based on the eye image, and
A control characterized by having a step of detecting a deviation visual state in which the eye deviates from a position corresponding to the center of the screen in the eye image based on the position of a pupil image or a Purkinje image in the eye image. Method. A program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 21. A computer-readable storage medium in which a program for operating a computer as each means of the electronic device according to any one of claims 1 to 21 is stored.

Images