JP2021043368A - Electronic apparatus and control method thereof - Google Patents

Abstract

To provide an electronic apparatus capable of precisely identifying the detection result of line-of-sight representing which frame a user looks at.SOLUTION: The electronic apparatus includes: display means that displays images; imaging means that picks up an image of an eye looking at a display image displayed on the display means; detection means that detects a line-of-sight using an image of eye picked up by the imaging means; and identification means that identifies the detection frame of line-of-sight that is a display frame corresponding to the line-of-sight detection result of the detection means. The identification means is configured to identify the line-of-sight detection frame based on drive mode information of the imaging means and drive mode information of the display means.SELECTED DRAWING: Figure 11

Description

The present invention relates to an electronic device, and more particularly to an electronic device that detects a line of sight.

In recent years, cameras have become more automated and intelligent. Patent Document 1 proposes a technique for recognizing a subject intended by a photographer based on information on the line-of-sight position of a photographer looking into a finder and performing focus control without manually inputting the subject position. Patent Document 2 describes a technique for recording a recorded image and line-of-sight information by associating the time information of the image to be recorded with the time information of line-of-sight detection.

Japanese Unexamined Patent Publication No. 2004-8323 Japanese Unexamined Patent Publication No. 2005-252732

With the prior art, it is difficult to identify the display image used for line-of-sight detection. In other words, it is difficult to specify which frame image the line-of-sight detection result represents the line-of-sight.

Patent Document 1 does not disclose to specify the display image used for the line-of-sight detection in the first place. Patent Document 2 associates an image with a line-of-sight detection result using time information, but the display image displayed at the line-of-sight position when the eyeball is imaged moves back and forth depending on the synchronization relationship between the image display and the eyeball imaging. there is a possibility. That is, Patent Document 2 may fail to specify the display image used for the line-of-sight detection.

For example, when the line-of-sight detection result is used for subject tracking or focus control, if the frame image for which the line-of-sight detection result is not specified is not specified, the subject unintended by the user is targeted for tracking or focusing. In particular, if the subject moves quickly, the subject tracking and focus detection accuracy may decrease.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an electronic device capable of accurately specifying which frame the line-of-sight detection result represents the line-of-sight.

The first aspect of the present invention is
Display means for displaying images and
An imaging means for capturing an eye for viewing a display image displayed on the display means, and an imaging means.
A detection means that detects the line of sight using an eye image captured by the imaging means, and
A specific means for specifying the line-of-sight detection frame, which is a display frame corresponding to the line-of-sight detection result of the detection means, and
With
The specific means identifies the line-of-sight detection frame based on the drive mode information of the image pickup means and the drive mode information of the display means.
It is an electronic device.

The second aspect of the present invention is
A display step that displays an image on a display means,
An imaging step in which an eye for viewing a display image displayed on the display means is imaged by the imaging means,
A detection step of detecting the line of sight using the eye image captured in the imaging step, and a detection step.
A specific step for specifying the line-of-sight detection frame, which is a display frame corresponding to the line-of-sight detection result of the detection step,
Including
In the specific step, the line-of-sight detection frame is specified based on the drive mode information of the image pickup means and the drive mode information of the display means.
This is a control method for electronic devices.

According to the present invention, it is possible to accurately identify which frame the line-of-sight detection result represents the line-of-sight.

The external view of the camera which concerns on embodiment. Sectional drawing of the camera which concerns on embodiment. The figure which shows how the virtual image which concerns on embodiment is imaged. The block diagram of the camera which concerns on embodiment. The figure which shows the field of view in the finder which concerns on embodiment. The figure for demonstrating the principle of the visual field detection method which concerns on embodiment. The figure which shows the eye image which concerns on embodiment. The flowchart of the line-of-sight detection operation which concerns on embodiment. The relationship diagram of the display frame and the eyeball imaging frame in an embodiment. The relationship diagram of the display frame and the eyeball imaging frame in an embodiment. The timing chart until the line-of-sight detection result output in the embodiment. The flowchart of the line-of-sight detection frame identification process in an embodiment. The relationship diagram of the display frame and the line-of-sight detection position in an embodiment. The flowchart of the gazing point focusing process in an embodiment. The flowchart of the tracking correction processing in an embodiment.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. The present invention can be applied to any electronic device, but an image pickup device (camera) will be described below as an example.

<Explanation of configuration>
1 (A) and 1 (B) show the appearance of the camera 1 (digital still camera; interchangeable lens camera) according to the present embodiment. FIG. 1 (A) is a front perspective view, and FIG. 1 (B) is a rear perspective view. As shown in FIG. 1A, the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5 which is an operation member for receiving an imaging operation from a user (photographer) is arranged on the camera housing 1B. As shown in FIG. 1 (B), on the back surface of the camera housing 1B, there is an eyepiece window frame 121 for the user to look into the display device 10 (display panel) described later contained in the camera housing 1B. An eyepiece lens 12 (eyepiece optical system) is arranged. The eyepiece window frame 121 surrounds the eyepiece lens 12 and projects to the outside (rear side) of the camera housing 1B with respect to the eyepiece lens 12. The eyepiece optical system may include a plurality of lenses. On the back surface of the camera housing 1B, operating members 41 to 43 that receive various operations from the user are also arranged. For example, the operation member 41 is a touch panel that accepts touch operations, the operation member 42 is an operation lever that can be pushed down in each direction, and the operation member 43 is a four-direction key that can be pushed in each of the four directions. The operation member 41 (touch panel) includes a display panel such as a liquid crystal panel, and has a function of displaying an image on the display panel.

FIG. 2 is a cross-sectional view of the camera 1 cut along the YZ plane formed by the Y-axis and the Z-axis shown in FIG. 1A, and shows a rough internal configuration of the camera 1.

In the photographing lens unit 1A, two lenses 101, 102, an aperture 111, an aperture drive unit 112, a lens drive motor 113, a lens drive member 114, a photo coupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. Etc. are included. The lens drive member 114 is composed of a drive gear or the like, and the photocoupler 115 detects the rotation of the pulse plate 116 linked to the lens drive member 114 and transmits the rotation to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 based on the information from the photocoupler 115 and the information from the camera housing 1B (lens drive amount information), and moves the lens 101 to adjust the focus position. change. The mount contact 117 is an interface between the photographing lens unit 1A and the camera housing 1B. Although two lenses 101 and 102 are shown for simplicity, more than two lenses are actually included in the photographing lens unit 1A.

The camera housing 1B includes an image sensor 2, a CPU 3, a memory unit 4, a display device 10, a display device drive circuit 11, and the like. The image sensor 2 is arranged on the planned image plane of the photographing lens unit 1A. The CPU 3 is a central processing unit of the microcomputer and controls the entire camera 1. The memory unit 4 stores an image or the like captured by the image sensor 2. The display device 10 is composed of a liquid crystal or the like, and displays an captured image (subject image) or the like. The display device drive circuit 11 drives the display device 10.

The display device 10 has arrived at the display means, and the image pickup device 2 corresponds to the second image pickup means.

The user can see the image (visual image) displayed on the display device 10 through the eyepiece window frame 121 and the eyepiece lens 12. Specifically, as shown in FIG. 3, the eyepiece 12 forms an image of a virtual image 300 in which the display device 10 is magnified at a position about 50 cm to 2 m away from the eyepiece 12. In FIG. 3, the virtual image 300 is formed at a position 1 m away from the eyepiece lens 12. The user can visually recognize the virtual image 300 by looking into the eyepiece window frame 121.

The camera housing 1B also includes light sources 13a and 13b, an optical divider 15, a light receiving lens 16, an eyeball image sensor 17, and the like. The light sources 13a and 13b are light sources conventionally used in a single-lens reflex camera or the like for detecting the line-of-sight direction from the relationship between the reflected image (corneal reflex image) due to the corneal reflex of light and the pupil, and the user's eyeball 14 It is a light source for illuminating. Specifically, the light sources 13a and 13b are infrared light emitting diodes or the like that emit infrared light that is insensitive to the user, and are arranged around the eyepiece lens 12. The optical image of the illuminated eyeball 14 (eyeball image; an image of reflected light emitted from the light sources 13a and 13b and reflected by the eyeball 14) passes through the eyepiece 12 and is reflected by the light divider 15. Then, the eyeball image is formed by the light receiving lens 16 on the eyeball image pickup element 17 in which a array of photoelectric elements such as a CCD is two-dimensionally arranged. The light receiving lens 16 positions the pupil of the eyeball 14 and the eyeball image sensor 17 in a conjugate imaging relationship. The line-of-sight direction (viewpoint in the visual recognition image) is detected from the position of the corneal reflex image in the eyeball image formed on the eyeball image pickup device 17 by a predetermined algorithm described later.

FIG. 4 is a block diagram showing an electrical configuration in the camera 1. The line-of-sight detection circuit 201, the photometric circuit 202, the autofocus detection circuit 203, the signal input circuit 204, the display device drive circuit 11, the light source drive circuit 205, and the like are connected to the CPU 3. Further, the CPU 3 transmits a signal to the focus adjustment circuit 118 arranged in the photographing lens unit 1A and the aperture control circuit 206 included in the aperture drive unit 112 in the photographing lens unit 1A via the mount contact 117. To do. The memory unit 4 attached to the CPU 3 has a function of storing an image pickup signal from the image pickup element 2 and the eyeball image pickup element 17, and a function of storing a line-of-sight correction parameter for correcting individual differences in the line of sight, which will be described later.

The line-of-sight detection circuit 201 corresponds to the detection means, the eyeball image sensor 17 corresponds to the image pickup means, and the CPU 3 corresponds to the specific means.

The line-of-sight detection circuit 201 A / D-converts the output (eye image obtained by imaging the eye) of the eyeball image sensor 17 in a state where the eyeball image is formed on the eyeball image sensor 17 (CCD-EYE), and obtains the result. It is transmitted to the CPU 3. The CPU 3 extracts feature points required for line-of-sight detection from the eye image according to a predetermined algorithm described later, and calculates the user's line of sight (viewpoint in the visual recognition image) from the positions of the feature points.

The photometric circuit 202 performs amplification, logarithmic compression, A / D conversion, and the like of a signal obtained from the image sensor 2 that also serves as a photometric sensor, specifically, a luminance signal corresponding to the brightness of the field of view. The result is sent to the CPU 3 as the photometric brightness information.

The autofocus detection circuit 203 A / D-converts signal voltages from a plurality of detection elements (a plurality of pixels) used for phase difference detection included in the CCD in the image pickup element 2 and sends them to the CPU 3. .. The CPU 3 calculates the distance from the signals of the plurality of detection elements to the subject corresponding to each focus detection point. This is a known technique known as imaging surface phase-difference AF. In the present embodiment, as an example, it is assumed that there are focus detection points at each of the 180 locations on the imaging surface corresponding to the 180 locations shown in the visual field image (visual field image) in the finder of FIG. 5 (A). ..

The signal input circuit 204 is turned on by the first stroke of the release button 5, is turned on by the switch SW1 for starting the photometry, distance measurement, line-of-sight detection operation, etc. of the camera 1, and is turned on by the second stroke of the release button 5. The switch SW2 for starting the shooting operation is connected. The ON signals from the switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU3.

The tracking circuit 207 is a circuit that tracks a subject in an input image, and transmits information on a tracking frame indicating a subject position to the CPU 3. The tracking process is performed by, for example, determining the similarity between two images by SAD (Sum Of Absolute Difference). Further, the tracking circuit 207 may use a tracking process other than SAD.

The recognition circuit 208 is a circuit that recognizes a subject in an input image, and for example, detects a person's face and an animal.

Further, the operation members 41 to 43 transmit the operation signal to the CPU 3. The CPU 3 controls the movement of the estimated gazing point frame position according to the operation signals transmitted from the operation members 41 to 43.

FIG. 5A is a view showing the field of view in the finder, and shows a state in which the display device 10 is operating (a state in which a visual image is displayed). As shown in FIG. 5A, the visual field in the viewfinder includes a focal point detection region 500, 180 AF point indexes 501, a visual field mask 502, and the like. Each of the 180 AF point indexes 501 is displayed superimposed on the through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to the focus detection point on the imaging surface. .. Also, of the 180 AF point indexes 501, the current estimated gazing point A (
The AF point index 501 corresponding to the estimated position) is highlighted by a frame or the like.

<Explanation of line-of-sight detection operation>
The line-of-sight detection method will be described with reference to FIGS. 6, 7 and 8. FIG. 6 is a diagram for explaining the principle of the line-of-sight detection method, and is a schematic view of an optical system for performing line-of-sight detection. As shown in FIG. 6, the light sources 13a and 13b are arranged substantially symmetrically with respect to the optical axis of the light receiving lens 16 and illuminate the user's eyeball 14. A part of the light emitted from the light sources 13a and 13b and reflected by the eyeball 14 is focused on the eyeball image sensor 17 by the light receiving lens 16. FIG. 7 shows a schematic view 701 of an eye image (eyeball image projected on the eyeball image sensor 17) captured by the eyeball image sensor 17, and a distribution 702 of the output intensity of the CCD in the eyeball image sensor 17. FIG. 8 shows a schematic flowchart of the line-of-sight detection operation.

When the line-of-sight detection operation starts, the light sources 13a and 13b emit infrared light toward the user's eyeball 14 in step S801 of FIG. The user's eyeball image illuminated by infrared light is imaged on the eyeball image sensor 17 through the light receiving lens 16 and photoelectrically converted by the eyeball image sensor 17. As a result, an electric signal of a processable eye image is obtained.

In step S802, the line-of-sight detection circuit 201 sends an eye image (eye image signal; electrical signal of the eye image) obtained from the eyeball image sensor 17 to the CPU 3.

In step S803, the CPU 3 obtains the coordinates of the points corresponding to the corneal reflex images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S802.

The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, the corneal reflex images Pd and Pe formed by a part of the infrared light reflected on the surface of the cornea 142 are condensed by the light receiving lens 16 and imaged on the eyeball imaging element 17 to be formed in the eye image. The corneal reflex images are Pd'and Pe'. Similarly, the luminous flux from the ends a and b of the pupil 141 is also imaged on the eyeball image sensor 17 to become the pupil end images a'and b'in the eye image.

The distribution 702 indicates the luminance information (luminance distribution) of the region α'in the eye image 701. In the distribution 702, the horizontal direction of the eye image 701 is the X-axis direction, the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In the present embodiment, the coordinates of the corneal reflection images Pd'and Pe'in the X-axis direction (horizontal direction) are Xd and Xe, and the coordinates of the pupil end images a'and b'in the X-axis direction are Xa and Xb. As shown in the luminance distribution 702 of FIG. 7, extremely high levels of luminance can be obtained at the coordinates Xd and Xe of the corneal reflection images Pd'and Pe'. In the region from the coordinates Xa to the coordinates Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by forming the luminous flux from the pupil 141 on the eyeball image sensor 17), the coordinates Xd and Xe are excluded. Extremely low levels of brightness are obtained. Then, in the region of the iris 143 outside the pupil 141 (the region of the iris image outside the pupil image obtained by forming the luminous flux from the iris 143), a brightness intermediate between the above two types of brightness can be obtained. Specifically, in a region where the X coordinate (coordinate in the X-axis direction) is smaller than the coordinate Xa and a region where the X coordinate is larger than the coordinate Xb, a brightness intermediate between the above two types of brightness can be obtained.

From the luminance distribution 702, the X-coordinates Xd and Xe of the corneal reflection images Pd'and Pe'and the X-coordinates Xa and Xb of the pupil end images a'and b'can be obtained. Specifically, the coordinates with extremely high brightness can be obtained as the coordinates of the corneal reflection images Pd'and Pe', and the coordinates with extremely low brightness can be obtained as the coordinates of the pupil end images a'and b'. .. Further, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16 is small, the pupil center image c'(pupil center image c'(pupil) obtained by forming a light beam from the pupil center c on the eyeball imaging element 17. The coordinates Xc (center of the image) can be expressed as Xc≈ (Xa + Xb) / 2. That is, the coordinates Xc of the pupil center image c'can be calculated from the X coordinates Xa and Xb of the pupil edge images a'and b'. In this way, the coordinates of the corneal reflex images Pd'and Pe'and the coordinates of the pupil center image c'can be estimated.

In step S804, the CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is a magnification determined by the position of the eyeball 14 with respect to the light receiving lens 16 and can be obtained by using a function of the interval (Xd-Xe) between the corneal reflection images Pd'and Pe'.

In step S805, the CPU 3 calculates the rotation angle of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16. The X coordinate of the midpoint of the corneal reflex image Pd and the corneal reflex image Pe and the X coordinate of the center of curvature O of the cornea 142 substantially coincide with each other. Therefore, assuming that the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Occ, the angle of rotation θ _X of the eyeball 14 in the ZX plane (plane perpendicular to the Y axis) is It can be calculated by the following formula (1). The rotation angle θy of the eyeball 14 in the ZZ plane (plane perpendicular to the X-axis) can also be calculated by the same method as the calculation method of the rotation angle θx.

β × Oc × SINθ _X ≒ {(Xd + Xe) / 2} -Xc ・ ・ ・ (1)

In step S806, the CPU 3 uses the rotation angles θx and θy calculated in step S805 to display the user's viewpoint (the position where the line of sight is poured; the position where the user is looking) in the visual image displayed on the display device 10. ) Is obtained (estimated). Assuming that the coordinates of the viewpoint (Hx, Hy) are the coordinates corresponding to the center of the pupil c, the coordinates of the viewpoint (Hx, Hy) can be calculated by the following equations (2) and (3).

Hx = m × (Ax × θx + Bx) ・ ・ ・ (2)
Hy = m × (Ay × θy + By) ・ ・ ・ (3)

The parameters m of the equations (2) and (3) are constants determined by the configuration of the finder optical system (light receiving lens 16 and the like) of the camera 1, and the rotation angles θx and θy are coordinates corresponding to the pupil center c in the visual image. It is a conversion coefficient to be converted into, and is determined in advance and stored in the memory unit 4. The parameters Ax, Bx, Ay, and By are line-of-sight correction parameters that correct individual differences in the line of sight, are acquired by performing the calibration work described later, and are stored in the memory unit 4 before the line-of-sight detection operation starts. And.

In step S807, the CPU 3 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 4, and finishes the line-of-sight detection operation.

<Explanation of calibration work>
As described above, the viewpoint can be estimated by acquiring the rotation angles θx and θy of the eyeball 14 from the eye image in the line-of-sight detection operation and converting the position of the pupil center c to the position on the visual recognition image.

However, it may not be possible to estimate the viewpoint with high accuracy due to factors such as individual differences in the shape of the human eyeball. Specifically, unless the line-of-sight correction parameters Ax, Ay, Bx, and By are adjusted to values suitable for the user, as shown in FIG. 5B, the actual viewpoint B and the estimated viewpoint C There will be a gap. In FIG. 5B, the user is gazing at a person, but the camera 1 erroneously presumes that the background is being gazed at, resulting in a state in which appropriate focus detection and adjustment cannot be performed.

Therefore, before the camera 1 performs imaging, it is necessary to perform calibration work, acquire a viewpoint correction parameter (eyeball characteristic) suitable for the user, and store it in the camera 1. The viewpoint correction parameter (eyeball characteristics) includes delay information indicating the degree of delay in eye movement. The CPU 3 that executes the calibration process corresponds to the eyeball characteristic acquisition means.

Conventionally, the calibration work has been performed by highlighting a plurality of indexes having different positions as shown in FIG. 5C on a visual image and having the user see the indexes before imaging. Then, a technique of performing a line-of-sight detection operation when gazing at each index and obtaining a viewpoint correction parameter suitable for the user from a plurality of calculated viewpoints (estimated positions) and the coordinates of each index is known as a known technique. ing. If the position to be viewed by the user is suggested, the index may not be displayed and the position may be emphasized by changing the brightness or color.

Further, during the calibration work, the CPU 3 stores the number of delay frames (delay information) C of the human eye movement. The delay frame number C is the number of frames between the frame on which the calibration work index of FIG. 5C is displayed and the frame in which the line of sight is detected at the index position in the displayed image. The delay frame number C is stored in the memory unit 4 for use in S1110 of the line-of-sight detection frame identification process described later in FIG. Since it is sufficient to acquire the delay information indicating the delay of the eye movement, the delay time may be acquired and stored instead of the number of delay frames.

<Explanation of display frame with line-of-sight detected>
As mentioned at the beginning, it is desirable to identify which display frame the user is actually looking at when the gaze point of the line-of-sight detection result is. In the present disclosure, the display frame actually viewed by the user is referred to as a display frame used for line-of-sight detection or a line-of-sight detection frame. The line-of-sight detection frame can also be referred to as a display frame corresponding to the line-of-sight detection result or a display frame corresponding to the gazing point.

The relationship between the frame (display frame) displayed on the display device 10 and the frame of the eye image (eyeball imaging frame) will be described with reference to FIGS. 9A to 9F. In these figures, the vertical direction represents the rows in the frame and the horizontal direction represents the time.

In FIGS. 9A to 9F, the combination of the synchronization relationship between the display frame and the eyeball image pickup frame and the readout time of the eyeball image pickup element 17 is different. In order to compare the display scanning time of the display device 10 with the reading time of the eyeball image sensor 17, the display scanning time of the display device 10 is constant in all cases, but the display scanning time of the display device 10 can be changed. May be good. 9 (A) to 9 (D) and 9 (G) assume imaging by the rolling shutter method by the eyeball image sensor 17, and FIGS. 9 (E) and 9 (F) show the global shutter. It is supposed to be imaged by the method.

In FIGS. 9A to 9F, 900 and 901 indicate a display frame, and the display frame 901 is a frame image displayed next to the display frame 900. In 904 and 905, the synchronization timing of the display is indicated by a dotted line. 902 is the exposure time of the eyeball image sensor 17, and 903 is the eyeball image pickup frame. The image captured by the eyeball image sensor 17 during the exposure period 902 is the eyeball image pickup frame 903.

906 and 907 represent the positions of the user's corneal reflex images (Purkinje images), and when there are four corneal reflex images, they represent the positions of the corneal reflex images detected at the lowermost side of the image. Since the gazing point (viewpoint coordinates calculated in S807) is detected based on the position of the corneal reflex image and the center position of the pupil, 906 and 907 are also referred to as gazing points for the sake of simplicity. The tip of the arrow extending from the corneal reflex image positions 906 and 907 represents the gaze point calculated based on the corneal reflex image positions 906 and 907, that is, the position in the display frame actually gazed by the user. If necessary, the position of the corneal reflex image in the eye image may be referred to as the gazing point in the ocular image, and the position actually gazing by the user may be referred to as the gazing point in the display frame. For convenience of explanation, an example is shown in which there are two gazing points in the eye image during one exposure period, but the actual gazing point is one.

FIG. 9A shows a case where the display device 10 and the eyeball image sensor 17 are synchronized and the read time of the eyeball image sensor 17 is equal to or less than the display scanning time of the display device 10.

The fact that the display device 10 and the eyeball image sensor 17 are synchronized means that in the present embodiment, the display start timing of the frame image by the display device 10 and the read start timing of the eyeball image pickup frame by the eyeball image sensor 17 coincide with each other. It means that it is. The display device 10 starts displaying a new frame image according to the input of the display synchronization signal. The eyeball image sensor 17 starts reading the signal charge accumulated during the exposure period in accordance with the input of the image pickup synchronization signal. The fact that the display device 10 and the eyeball image sensor 17 are synchronized can be said to mean that the input of the display synchronization signal and the image pickup synchronization signal are at the same timing.

In the case of FIG. 9A, the display frame 900 is displayed at the gazing point at the timing when the corneal reflex image is taken, regardless of the position of the gazing point 906 in the eye image. Therefore, the line-of-sight detection frame is the display frame 900 when the exposure of the eyeball imaging frame 903 is started regardless of the position of the gazing point. The display frame 900 can also be identified as a display frame immediately before the display frame 901 displayed (display starts) at the input timing of the imaging synchronization signal.

FIG. 9B shows a case where the display device 10 and the eyeball image sensor 17 are synchronized and the read time of the eyeball image sensor 17 is longer than the display scanning time of the display device 10. In this case, the line-of-sight detection frame may differ depending on the position of the gazing point in the eye image. In this example, the gaze point 906 corresponds to the display frame 900 and the gaze point 907 corresponds to the display frame 901.

FIG. 9C shows a case where the display device 10 and the eyeball image sensor 17 are asynchronous and the read time of the eyeball image sensor 17 is equal to or less than the display scanning time of the display device 10. In this case, the line-of-sight detection frame may differ depending on the position of the gazing point in the eye image. In this example, the gazing point 906 corresponds to the display frame 901 and the gazing point 907 corresponds to the display frame 900.

FIG. 9D shows a case where the display device 10 and the eyeball image sensor 17 are asynchronous and the read time of the eyeball image sensor 17 is longer than the display scanning time of the display device 10. In this case, the line-of-sight detection frame differs depending on the position of the gazing point in the eye image. The display frame corresponding to the line-of-sight detection result is the same as in FIG. 9B, except that the display frame and the imaging frame are asynchronous, that is, adjustment must be made according to the difference between the display start timing and the imaging start timing. Is required.

FIG. 9E shows a case where the display device 10 and the eyeball image sensor 17 are synchronized with each other and the eyeball image sensor 17 takes an image by the global shutter method. In this case, it can be equated with the case where the readout time of the eyeball image sensor 17 is zero in FIG. 9A. Therefore, regardless of the position of the gazing point 906 in the eye image, it can be determined that the line-of-sight detection frame is the display frame 900.

FIG. 9F shows a case where the display device 10 and the eyeball image sensor 17 are asynchronous, and the eyeball image sensor 17 takes an image by the global shutter method. In this case, it can be equated with the case where the readout time of the eyeball image sensor 17 is zero in FIG. 9C. Therefore, the line-of-sight detection display frame differs depending on the position of the gazing point in the eye image.

FIG. 9 (G) shows a case where the display device 10 and the eyeball image sensor 17 are synchronized, and the image pickup frame rate of the eyeball image sensor 17 and the display frame rate of the display device 10 are 2: 1. If the eyeball imaging frames 908 are not used for the line-of-sight detection and the eyeball imaging frames 903 and 909 are used for the line-of-sight detection, the line-of-sight detection frame can be obtained in the same manner as in FIG. 9A. Further, when only the eyeball imaging frame 908 is used for the line-of-sight detection, the line-of-sight detection frame can be obtained in the same manner as in FIG. 9C. That is, when the frame rate of the eyeball image sensor 17 and the frame rate of the display device 10 are n: 1 (n is an integer of 2 or more), the line of sight is detected using only the eye image captured in synchronization with the display. Then, the line-of-sight detection frame can be easily specified.

9 (A) to 9 (G) are only some examples. From these examples, the display frame corresponding to the line-of-sight detection position is determined according to the synchronization relationship between the display frame and the eyeball imaging frame, the display scanning time of the display device 10, the readout time of the eyeball imaging element 17, and the vertical position of the gazing point. You can see that they are different.

<Explanation of line-of-sight detection frame identification processing>
Hereinafter, the line-of-sight detection frame identification process will be described with reference to FIGS. 10 and 11. The line-of-sight detection frame identification process is a process for specifying the line-of-sight detection frame based on the drive mode information of the display device 10 and the eyeball image sensor 17.

FIG. 10 shows a timing showing a series of flow of displaying the image data captured by the image sensor 2 on the display device 10, imaging the eyeball of the user who is viewing the display by the eyeball image sensor 17, and detecting the line of sight of the eye image. It is a figure. The horizontal direction of FIG. 10 is the time axis. 1000-1003 indicates a display frame, and 1004 indicates a user's gaze point. T0 is the synchronization timing of the image sensor 2. T1 is a synchronization timing to the display device 10, and is a timing (display scanning start timing) at which a display synchronization signal is input. T2 is the synchronization timing of the eyeball image pickup device 17, and is the timing at which the imaging synchronization signal is input (reading start timing; drive start timing). T3 is the display scanning completion timing of the display device 10. T4 is the read completion timing of the eyeball image sensor 17. T5 is the output timing of the line-of-sight detection result, that is, the timing at which the position of the gazing point in the displayed image is output. The difference between T3 and T1 corresponds to the display scanning time of the display device 10, and the difference between T4 and T2 corresponds to the reading time of the eyeball image sensor 17. Α, Td, Ts, and Tr in FIG. 10 will be described below.

FIG. 11 is a flowchart showing the flow of the line-of-sight detection frame identification process (hereinafter, also referred to as frame identification process). This process will be described as being controlled by the CPU 3.

In S101, the CPU 3 determines whether or not the eyeball image sensor 17 and the display device 10 are in a synchronous relationship and the read time of the eyeball image sensor 17 is shorter or the same as the display scanning time of the display device 10. When the eyeball image sensor 17 takes an image by the global shutter method (FIGS. 9 (E) and 9 (F)), the readout time of the eyeball image sensor 17 is regarded as zero. If the determination condition is satisfied, the CPU 3 proceeds to S108, and if not, the process proceeds to S102.

In S102, the CPU 3 determines whether or not the eyeball image sensor 17 has been exposed during the display scan of the display device 10. If the exposure is performed, the process proceeds to S103, and if not, the process proceeds to S108.

In S103, the CPU 3 calculates the difference time α between the read start time T2 of the eyeball image sensor 17 and the display start time T1 of the display device 10 by the equation (4).

α = T2-T1 ・ ・ ・ (4)

In S104, the CPU 3 calculates the time Ts from the start of reading by the eyeball image sensor 17 to the start of reading the line in which the gazing point (corneal reflex image) is captured by the equation (5). However, at the time of global exposure, Ts = 0.

Ts = (reading time of one horizontal line of the eyeball image sensor 17) × (number of lines in the vertical direction up to the gazing point (including the gazing point) on the eyeball image sensor 17) ... (5)

In S105, the CPU 3 calculates the time Td from when the display device 10 starts displaying the frame image until the line including the gazing point is displayed by the equation (6).

Td = (time of one horizontal line of display scan) × (total number of vertical lines of display device 10) × (number of vertical lines to the gazing point of eyeball image sensor 17) / (total number of vertical lines of eyeball image sensor 17) ... (6)

In S107, the CPU 3 calculates the frame difference X (adjustment information) between the line-of-sight detection frame and the display frame at the start of driving the eyeball image sensor 17. The line-of-sight detection frame is considered to be a display frame displayed at the start of exposure at the line position of the eyeball image sensor 17 on which the corneal reflex image is captured. Therefore, the timing (Td) at which the display device 10 starts displaying the frame image and then the line including the gazing point is started, and the timing (α + Ts−) of the exposure of the line of the eyeball image sensor 17 on which the corneal reflex image is taken. The value X can be calculated according to the difference in exposure time). Specifically, the CPU 3 calculates the value of X by the following equation (7). In the example of FIG. 10, X = 0 is calculated.

X = Floor [(Ts + α-Td-exposure time) / display cycle] ・ ・ ・ (7)

Here, Floor (x) is a floor function, that is, a function that returns the maximum integer that does not exceed the real number x. The exposure time is the exposure time of one line of the eyeball image sensor 17. The display cycle is an input interval of the display synchronization signal of the display device 10.

Since the exposure time is used in the equation (7), the value X represents a display frame displayed at the start of exposure and a display frame displayed at the start of driving the eyeball image sensor 17. Here, if a value that is half of the exposure time is used, the value X for the display frame displayed at an intermediate point between the start of exposure and the end of exposure can be obtained. Further, if the exposure time is not used, the value X for the display frame displayed at the end of the exposure can be obtained. Instead of the equation (7), the value X may be calculated in this way.

In S108, the CPU 3 sets X = -1. This means that the line-of-sight detection frame is the display frame immediately before the frame displayed at the drive timing of the eyeball image sensor 17.

In the flowchart of FIG. 11, the value X is calculated by which of the processes S107 and S108 is calculated according to the determination results of S101 and S102. However, the same result can be obtained even if the value X is always calculated by the processing of S107 without determining S101 and S102. For example, when an affirmative judgment is made in S101, α = 0 and Ts ≦ Td, and since the difference between Ts and Td and the exposure time are sufficiently smaller than the display cycle, X = according to the equation (7). It is calculated as -1.

However, in the case of FIGS. 9 (A) and 9 (G), the value of X can be determined without performing the processes of S103 to S107, which is convenient. That is, when the display device 10 and the eyeball image sensor 17 are synchronized and the image pickup frame rate of the eyeball image sensor 17 is an integral multiple of the display frame rate of the display device 10, the following may be performed. Specifically, the line-of-sight detection circuit 201 performs line-of-sight detection using only an eye image whose reading is started at the same timing as the display start timing of the display device 10. In this way, X = -1, that is, the line-of-sight detection frame of the display frame displayed on the display device 10 at the timing when the eyeball image sensor 17 starts reading, without performing the processes of S103 to S107. It can be identified as the previous frame.

In S109, the CPU 3 calculates the number of frames Y between the display frame of the drive start timing (T2) of the eyeball image sensor 17 and the display frame of the gaze detection completion timing (T5). Specifically, since the value Y is the number of frames corresponding to the total of the time Tr and the time α, it is calculated by the equation (8). In the example of FIG. 10, Y = 2 is calculated.

Y = Floor ((Tr + α) / display cycle) ・ ・ ・ (8)

Here, Tr = T5-T2, and Floor (x) is a floor function.

In S110, the CPU 3 calculates a value Z indicating how many frames before the line-of-sight detection display frame is before the display frame of the line-of-sight result output timing (T5) by the equation (9). Specifically, the value Z is calculated by adjusting the value Y using the adjustment information X based on the drive mode information and the delay information C of the eyeball characteristics.

Z = Y − (X + C) ・ ・ ・ (9)

Here, C is the number of delay frames acquired at the time of calibration.

By the above processing, the line-of-sight detection frame is identified as a frame before the Z frame from the display frame displayed at the timing when the line-of-sight detection result is obtained. The value Z adjusts the frame difference Y obtained based on the timing at which the line-of-sight detection result is obtained by using the adjustment information X based on the drive mode information of the eyeball image sensor 17 and the display device 10 and the eyeball delay information C. This is the frame difference after adjustment. In the adjustment based on the drive mode information (S107), the synchronization timing (T2), the image readout time from the image sensor 17 (T4-T2), and the exposure time per line are used as the drive mode information of the eyeball image sensor 17. .. Further, the synchronization timing (T1) and the display scanning time (T3-T1) are used as the drive mode information of the display device 10.

By adjusting the value Y according to the equation (8) using the value X (adjustment information) calculated by using the drive mode information of the display device 10 and the eyeball image sensor 17, the line-of-sight detection frame can be specified more accurately.

If the display frame corresponding to the line-of-sight detection result can be specified, the object that the user is actually gazing at can be specified, so that various processes for the object that the user is gazing at can be performed more appropriately. This is especially effective when the subject moves quickly. In the following, focusing control and tracking correction processing will be described as examples of processing for an object that the user is actually gazing at.

<Explanation of focusing control>
Hereinafter, focusing control using the gaze point detected by the line of sight will be described with reference to FIGS. 12 and 13. The image pickup apparatus 1 performs focusing control so as to focus on the subject that the user is actually gazing at.

FIG. 12 is a diagram showing images 1201 to 1204 when an image taken by the image pickup apparatus 1 according to the present embodiment is displayed on the display device 10. Images 1201 to 1203 are in chronological order in this order, in which the car 1200 is moving from the upper right to the lower left. The image 1204 is an image of the same frame as the image 1203, and the position and size of the automobile 1200 are the same. The tracking frames 121 to 1214 are images showing a region in which the automobile 1200 to be tracked exists. The information of the tracking frames 121 to 1214 is output from the tracking circuit 207, superimposed on the captured image, and displayed on the display device 10.

1221, 1222, 1223 indicate the gazing point. The gazing point is actually a point, but as will be described later, a rectangular image centered on the gazing point is used for the focusing control process. Therefore, in FIG. 12, the gazing points 1221, 1222, 1223 are shown as rectangular images. In the present disclosure, a rectangular image centered on the gazing point is also referred to as a gazing point image. In the following, the gazing point and the gazing point image will be referred to using the same reference numerals. For example, a rectangular image centered on the gazing point 1221 is referred to as a gazing point image 1221. Although the gazing points 1221, 1222, and 1223 are drawn in FIG. 12, they may or may not be displayed on the display device 10. When the gazing point is displayed, a display frame having the same size as the gazing point image (rectangular image) is displayed on the image centering on the gazing point.

The gazing point 1221 represents the gazing point of the user when the frame of the image 1201 is displayed on the display device 10. The image 1202 is the frame next to the image 1201, and the automobile 1200 is moving. Images 1203 and 1204 are the next frames of image 1202. Here, it is assumed that the gazing point 1221 when the user is looking at the image 1201 is detected two frames later, that is, Z = 2 described above.

The gazing point 1222 has the same horizontal and vertical positions as the gazing point 1221 in the image. The image pickup apparatus 1 performs focusing control so as to focus on the subject that the user is gazing at. However, focusing on the subject at the gazing point 1222 in the image 1203 is contrary to the user's intention. This is because the user intends to focus on the subject at the gazing point 1221 in the image 1201. Therefore, the image pickup apparatus 1 corrects the gaze point 1222 to the gaze point 1223 corresponding to the gaze point 1221 by the correction process based on the detection frame specified by the line-of-sight detection frame identification process. By performing focusing control on the gazing point 1223, it is possible to focus on the position originally desired by the user. The gazing point correction process will be described in the gazing point focusing process.

FIG. 13A is a flowchart showing the flow of focusing control processing. The focusing control process is executed by the CPU 3. When the focusing control process is started, the CPU 3 executes the processes after S201.

In S201, the CPU 3 performs the gazing point correction process. The gaze point correction process is a process of correcting the gaze point 1222 obtained as a line-of-sight detection result to a position in the current display frame corresponding to the gaze point 1221 of the line-of-sight detection frame 1201 that the user was actually gazing at. .. More specifically, the gaze point correction process is a process of correcting the position in the current display frame 1204 (gaze point 1223) where the subject located at the gaze point 1221 of the line-of-sight detection frame 1201 is damaged. ..

In S202, the CPU 3 controls the focus adjustment circuit 118 so as to focus on the subject at the corrected gazing point (corrected gazing point) position.

FIG. 13B is a flowchart showing the flow of the gazing point correction process. The gaze point correction process of S201 will be described in more detail with reference to FIG. 13 (B).

In S211 the CPU 3 extracts a rectangular image centered on the gazing point position from the line-of-sight detection frame specified by the line-of-sight detection frame identification process. Hereinafter, this image will be referred to as a gaze point image A. In the example of FIG. 12, the gazing point image A is the gazing point image 1221.

In S212, the CPU 3 extracts the image at the gazing point position as a rectangular image with respect to the currently displayed display frame. Hereinafter, this image will be referred to as a gaze point image B. In the example of FIG. 12, the gazing point image B is the gazing point image 1222. The CPU 3 sets the size of the gazing point image 1222 so that the size of the gazing point image 1221 with respect to the tracking frame 1211 and the size of the gazing point image 1222 with respect to the tracking frame 1213 match.

In S213, the CPU 3 calculates the SAD (Sum Of Absolute Difference) calculation, that is, the sum of the absolute values of the differences between the pixel values, in order to calculate the similarity between the gazing point frame images A and B. The smaller the SAD value, the higher the similarity. If the sizes of the gazing point images A and B are different, the SAD calculation is performed after the sizes are the same. In the present embodiment, SAD is used to calculate the similarity of the images, but the similarity may be calculated by another calculation. For example, SSD (Sum of Squared Difference) or NCC (Normalized Cross-Correlation) may be used.

In S214, the CPU 3 determines whether the SAD value calculated in S203 is within the threshold value, proceeds to S205 if it is within the threshold value, and proceeds to S206 if it is above the threshold value. It can be said that S204 is a branching process in which the process proceeds to S205 if the similarity between the gaze frame images A and B is higher than the threshold value, and proceeds to S206 if the similarity is less than the threshold value.

In S215, the CPU 3 ends the process without performing the gazing point correction process. If the degree of similarity between the gaze point images A and B is high, the subject has not moved, and the gaze point image A (1221) and the gaze point image B (1222) match, and no correction process is required.

On the other hand, if the degree of similarity between the gazing point images A and B is low, the subject is moving. Therefore, the position of the object that the user was actually gazing at is searched from the current frame by the processing after S216, and the result is obtained. The corrected position is set as the gaze point 1223 after correction.

In S216, the CPU 3 sets the area around the gazing point 1222 as the search area. The search area 1222 may be, for example, a rectangular area centered on the gazing point 1222, or a position where an object at the gazing point 1221 of the line-of-sight detection frame 1201 is likely to exist in consideration of the movement of the subject. You may meet in the central rectangular area.

In S217, the CPU 3 sets an evaluation frame for the image in the tracking frame 1213 of the current display frame 1203 (1204). The image in the evaluation frame is called an evaluation frame image. The size of the evaluation frame is the same as that of the gazing point image B. In S218, the CPU 3 calculates the SAD value between the evaluation frame image and the gazing point image A. In S219, the CPU 3 proceeds to S220 if the calculated SAD value is within the threshold value (similarity is equal to or higher than the threshold value), and proceeds to S221 otherwise.

In S220, the CPU 3 sets the position of the current evaluation frame as the corrected gazing point position. As a result, among the current frames, the position where the object at the gazing point 1221 of the display frame 1201 that the user was actually gazing at exists can be set as the gazing point after correction.

In S221, the CPU 3 determines whether the similarity evaluation of the evaluation frame images at all the positions in the tracking frame 1213 is completed, and if it is completed, proceeds to S222, and if not, proceeds to S223.

In S222, the CPU 3 controls the focus adjustment circuit 118 so as to focus on the subject at the center position of the tracking frame 1213. This is because when there is no evaluation frame higher than the threshold value, it is considered that the center of the tracking frame 1213 is being watched, and the error is considered to be the smallest. In S222, the focus may be on a position other than the center position of the tracking frame 1213. For example, the CPU 3 obtains the corrected gazing point position 1223 by moving the gazing point position 1222 based on the difference in the positions of the tracking frame 1221 and the tracking frame 1223, and focuses on the corrected gazing point position 1223. You may.

In S223, the CPU 3 sets a frame in which the evaluation frame is shifted by a certain pixel as a new evaluation frame image with respect to the image in the tracking frame 1213, and controls to return to S207.

By the above processing, it is possible to focus on the subject position that the user is actually gazing at.

The gaze point correction process also corrects the size of the gaze point image after correction. Specifically, the corrected gazing point so that the ratio of the size of the gazing point image 1221 to the tracking frame 1211 in the line-of-sight detection frame 1201 and the ratio of the gazing point image 1223 to the tracking frame 1213 in the current frame 1204 are the same. The size of the image is set.

In the flowchart of FIG. 13B, if an evaluation frame having a similarity equal to or higher than the threshold value is found, the process proceeds to S209. Alternatively, evaluation frames are set at all positions in the tracking frame 1213, which is the highest. The focus may be on the position of the evaluation frame where the similarity can be obtained. In this alternative example, if the highest similarity obtained is less than the threshold value, the processing of S211 may focus on the center position of the tracking frame.

<Explanation of tracking correction processing>
Hereinafter, the tracking correction process using the gaze point detected by the line of sight will be described with reference to FIG. When the deviation between the tracking position (tracking frame) obtained by the tracking circuit 207 and the gazing point is large, the image pickup apparatus 1 resets the subject actually gazing by the user to the tracking position.

FIG. 14 is a flowchart showing a flow of correction processing of the tracking position of the subject using the gazing point detected by the line of sight. The flow of the tracking correction processing of FIG. 14 will be described as being controlled by the CPU 3, but the tracking correction processing may be performed by the tracking circuit 207.

In S301, the CPU 3 compares the gazing point position of the display frame specified by the line-of-sight detection frame specifying process with the tracking position of the latest tracking image, and obtains the position deviation. If the positional deviation is within the threshold value, the CPU 3 proceeds to S302, and if not, proceeds to S303. The threshold used is a predetermined value.

In S302, the CPU 3 determines that the reliability of the latest tracking position is high, maintains the tracking position, and finishes the process. That is, if the position deviation is determined within the threshold value in S301, the tracking position is not corrected.

In S303, the CPU 3 obtains the moving speed of the tracking subject. The moving speed can be calculated from the difference in tracking position between the line-of-sight detection frame and the frame before that (that is, the amount of movement of the subject between frames) and the image frame rate. The moving speed of the tracking subject may be determined by other known methods.

In S304, the CPU 3 determines (updates) the threshold value of the deviation amount based on the subject speed. Specifically, the CPU 3 adopts a larger threshold value as the subject speed is faster. The relationship between velocity and threshold may be linear or non-linear.

In S305, the CPU 3 determines whether the deviation between the gazing point position and the tracking position is within the threshold value determined in S304. The CPU 3 proceeds to S302 within the threshold value, otherwise proceeds to S306.

In S306, the CPU 3 re-searches the subject using the gazing point image of the line-of-sight detection frame. The search range is a region in which there is a high possibility that a subject predicted based on the subject speed exists. The CPU 3 evaluates the similarity of the evaluation frame within the search range with the gazing point image of the line-of-sight detection frame, and obtains the position of the evaluation frame having the highest degree of similarity. For example, SAD can be adopted for the similarity evaluation, but other similarity criteria may be used. If the highest similarity is equal to or higher than the threshold value, the CPU 3 sets this evaluation frame as the tracking frame. If the CPU 3 is less than the highest similarity threshold value, the CPU 3 has no tracking frame (tracking lost). When the tracking is lost, the CPU 3 may detect a subject (for example, a face) by the recognition circuit 208 and update the tracking frame using the result.

As described above, according to the present embodiment, it is possible to accurately identify which display frame the line-of-sight detection result is when the user is looking at. As a result, the subject image at the gazing point position in the line-of-sight detection frame can be specified, so that the user can control the object actually gazing. For example, in the example applied to focusing control, it is possible to focus on the subject that the user is actually gazing at. Further, in the example applied to the tracking correction, the subject actually being watched by the user can be set to the tracking position.

Although the present invention has been specifically described above based on the examples, it goes without saying that the present invention is not limited to the above examples and various modifications can be made without departing from the gist thereof.

For example, in the above embodiment, the display device 10 and the eyeball image sensor 17 are arranged in the viewfinder of the camera, but the present invention can be applied to any electronic device other than the camera. For example, the display device 10 may be a monitor output from a personal computer, and the eyeball image sensor 17 may be a camera attached to the monitor. The display device 10 is an HMD (head-mounted display) worn on the head to experience VR (virtual reality) or the like, and the eyeball image sensor 17 may be a camera attached to the HMD. Further, the display device 10 may be a glasses-type device such as an AR (augmented reality) glass, and the eyeball image sensor 17 may be a camera attached to the glasses-type device. The display device 10 may be either a virtual image projection method or a retinal projection method.

<Other Embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1: Camera 3: CPU 10: Display device 17: Eyeball image sensor 201: Line-of-sight detection circuit

Claims (19)

Display means for displaying images and
An imaging means for capturing an eye for viewing a display image displayed on the display means, and an imaging means.
A detection means that detects the line of sight using an eye image captured by the imaging means, and
A specific means for specifying the line-of-sight detection frame, which is a display frame corresponding to the line-of-sight detection result of the detection means, and
With
The specific means identifies the line-of-sight detection frame based on the drive mode information of the image pickup means and the drive mode information of the display means.
Electronics. The drive mode information of the image pickup means includes synchronization timing, reading time of image data from the image pickup device, and exposure time.
The drive mode information of the display means includes a synchronization timing and a display scanning time.
The electronic device according to claim 1. The specific means
Difference between the display start timing of the frame image by the display means and the read start timing of the image pickup means,
The time from when the imaging means starts reading to when reading the line in which the corneal reflex image appears.
The exposure time of the imaging means, and the time from when the display means starts displaying the frame image until the line including the gazing point is displayed.
Is used to obtain adjustment information indicating a frame difference between the line-of-sight detection frame and the display frame displayed at the read start timing of the imaging means, and the line-of-sight detection frame is specified using the adjustment information.
The electronic device according to claim 1 or 2. The specific means
Corresponds to the total of the time from the start of displaying the frame image by the display means to the start of reading by the imaging means and the time from the start of reading by the imaging means until the line-of-sight detection result is obtained. Adjust the number of frames to be performed using the adjustment information,
From the display frame at the timing when the line-of-sight detection result is obtained, the display frame that is the number of frames before the adjustment is determined as the line-of-sight detection frame.
The electronic device according to claim 3. The drive mode information of the imaging means includes a synchronization timing and a frame rate.
The drive mode information of the display means includes a synchronization timing and a frame rate.
The imaging means and the display means are synchronized, and the frame rate of the imaging means is an integral multiple of the frame rate of the display means.
The detection means detects the line of sight using only the eye image whose reading is started at the same timing as the display start timing of the display means, and displays the line of sight detection frame at the timing when the imaging means starts reading. Identify the frame immediately before the display frame displayed in the means,
The electronic device according to claim 1 or 2. Further provided with a characteristic acquisition means for acquiring eye characteristics including at least delay information of the user's eye movements,
The identifying means also identifies the line-of-sight detection frame using the delay information.
The electronic device according to any one of claims 1 to 4. Further provided with a gazing point correction means for determining a position in the current display frame corresponding to the gazing point of the line-of-sight detection frame as a correction gazing point.
The electronic device according to any one of claims 1 to 6. The position in the current display frame corresponding to the gazing point of the line-of-sight detection frame is the position in the current display frame where the subject located at the gazing point of the line-of-sight detection frame exists.
The electronic device according to claim 7. The gaze correction means is
Set the area around the gazing point of the current display frame as the search area, and set it as the search area.
In the search area, the evaluation frame is moved to obtain the similarity between the image in the evaluation frame of the current display frame and the frame image including the gazing point of the line-of-sight detection frame.
The position of the evaluation frame in which the similarity of the line-of-sight detection frame with the frame image at the gazing point is equal to or higher than a predetermined threshold value is determined as the corrected gazing point.
The electronic device according to claim 7 or 8. The gazing point correction means determines the position of the evaluation frame having the highest degree of similarity as the corrected gazing point.
The electronic device according to claim 9. The search area is set according to the moving speed of the subject located at the gazing point.
The electronic device according to claim 9 or 10. The display means displays a display frame for representing the position of the user's gaze at the position of the correction gaze point of the display frame.
The electronic device according to any one of claims 7 to 11. The display frame has a size based on the ratio of the size of the object detected at the position of the gazing point of the line-of-sight detection frame to the size of the object detected at the position of the corrected gazing point of the display frame. Have,
The electronic device according to claim 12. With the second imaging means
A focus adjusting means for controlling the focus of the second imaging means and
With more
The display means displays an image captured by the second imaging means, and displays the image.
The focus adjusting means controls the focus so as to focus on the corrected gazing point.
The electronic device according to any one of claims 7 to 13. When the similarity between the image of the gazing point position of the line-of-sight detection frame and the image of the gazing point position of the current display frame is lower than a predetermined threshold value, the focus adjusting means is applied to the corrected gazing point. Control the focus so that it is in focus,
When the similarity between the image of the gazing point position of the line-of-sight detection frame and the image of the gazing point position of the current display frame is higher than a predetermined threshold value, the focus adjusting means focuses on the gazing point. The electronic device according to claim 14, wherein the focus is controlled so as to match. With additional tracking means to track the subject,
If the deviation between the tracking position and the position of the gazing point detected by the detecting means is equal to or greater than a predetermined threshold value, the tracking means corrects the tracking position to the position of the corrected gazing point.
The electronic device according to any one of claims 7 to 15. A display step that displays an image on a display means,
An imaging step in which an eye for viewing a display image displayed on the display means is imaged by the imaging means, and an imaging step.
A detection step of detecting the line of sight using the eye image captured in the imaging step, and a detection step.
A specific step for specifying the line-of-sight detection frame, which is a display frame corresponding to the line-of-sight detection result of the detection step,
Including
In the specific step, the line-of-sight detection frame is specified based on the drive mode information of the image pickup means and the drive mode information of the display means.
How to control electronic devices. A program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 16. A computer-readable storage medium containing a program for causing the computer to function as each means of the electronic device according to any one of claims 1 to 16.

Images