JP2023155585A - Electronic device - Google Patents

Abstract

To provide a technique that enables high-precision eye-tracking regardless of the user's eye condition.SOLUTION: An electronic device has control means for controlling the amount of light incident on a user's eye. The control means, during calibration of eye tracking function for obtaining eye gaze information on the user's eye gaze, controls the amount of light incident on the user's eye split to each of a plurality of amounts of light.SELECTED DRAWING: Figure 9

Description

The present invention relates to an electronic device that performs control for calibrating a line of sight detection function.

In recent years, head-mounted displays (HMDs), such as eyeglass-type devices that utilize MR (Mixed Reality) and AR (Augmented Reality), that have a function of detecting the user's line of sight have become increasingly automated and intelligent. In such an HMD, a deviation occurs between the detected point of gaze and the actual point of gaze (the point of gaze intended by the user), and the pointer may not be displayed at the actual point of gaze. Patent Documents 1 and 2 disclose that the line of sight detection function is calibrated.

Japanese Patent Application Publication No. 2017-138609 JP2019-086556A

However, since the condition of the user's eyeballs changes from moment to moment, even if conventional calibration is performed, line of sight detection may not be performed with high accuracy.

An object of the present invention is to provide a technology that enables highly accurate line-of-sight detection regardless of the state of the user's eyeballs.

A first aspect of the present invention includes a control means for controlling the amount of light incident on the user's eyes, and the control means includes: during calibration of a line of sight detection function that obtains line of sight information regarding the user's line of sight; The electronic device is characterized in that the amount of light that enters the user's eyes is controlled to each of a plurality of light amounts.

A second aspect of the invention comprises the step of controlling the amount of light incident on the user's eyes, the amount of light incident on the user's eyes during calibration of a line of sight detection function for obtaining line of sight information regarding the user's line of sight. This is a method for controlling an electronic device, characterized in that the amount of light to be transmitted is controlled to each of a plurality of light amounts.

A third aspect of the present invention is a program for causing a computer to execute a step of controlling the amount of light incident on the user's eyes, wherein the step includes a line-of-sight detection function that obtains line-of-sight information regarding the user's line of sight. The program is characterized in that during calibration, the amount of light incident on the user's eyes is controlled to each of a plurality of light amounts.

A fourth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to execute a step of controlling the amount of light incident on the user's eyes, wherein The storage medium is characterized in that during calibration of a line-of-sight detection function for obtaining line-of-sight information regarding the user's line of sight, the amount of light incident on the user's eyes is controlled to each of a plurality of light amounts.

According to the present invention, highly accurate line of sight detection is possible regardless of the state of the user's eyeballs.

FIG. 2 is an external view of a display device. FIG. 2 is a block diagram of a display device. FIG. 2 is a diagram for explaining the principle of a line of sight detection method. It is a figure showing an eye image. It is a flowchart of line of sight detection operation. FIG. 3 is a diagram showing a field of view. FIG. 7 is a diagram showing the correspondence between pupil size and appropriate line of sight correction parameters. It is a figure showing the installation state of a light control device. FIG. 3 is a diagram showing changes in transmittance of a light control device. It is a flowchart of a calibration operation. FIG. 3 is a diagram showing the inside of the eyeball.

<<Example 1>>
Example 1 of the present invention will be described. An example in which the present invention is applied to an optical see-through display device (an optical see-through head mounted display (HMD)) will be described below. A user wearing an optical see-through type display device can directly view the outside world (real space) through an optical member (lens). Furthermore, the user can view the virtual object displayed on the optical see-through display device.

Note that the present invention is also applicable to other display devices. For example, the present invention is also applicable to a video see-through type display device (video see-through type HMD). A video see-through type display device displays a virtual space in which images of the outside world are captured in substantially real time. A user wearing a video see-through display device cannot directly see the outside world, but can indirectly see the outside world by viewing the virtual space displayed on the video see-through display device. The present invention is also applicable to an HMD that displays a virtual space unrelated to the outside world. Further, the present invention is applicable to display devices and various control devices that control display devices. For example, the present invention is not limited to display devices, but can also be applied to various electronic devices (for example, controllers of display devices, personal computers (PCs), etc.) that perform control for calibration of line-of-sight detection functions. .

<Explanation of configuration>
1(A) and 1(B) show the appearance of a display device 100 according to a first embodiment. FIG. 1(A) is a front perspective view, and FIG. 1(B) is a rear perspective view. The display device 100 is an optical see-through type display device, which is a type of head-mounted display (HMD) that can be attached to and detached from the head, and is a glasses-type device that uses MR (Mixed Reality) or AR (Augmented Reality). It is. The display device 100 can separately detect the line of sight of the right eye and the line of sight of the left eye of the user who wears the display device 100 on the head. Hereinafter, a user who wears the display device 100 on his or her head will be simply referred to as a user.

Lens 10 is an optical member facing the user's eyes. The user can view the outside world through the lens 10. The display device 11 displays a virtual object (virtual image of the virtual object) to both eyes (both the right eye and the left eye) of the user under control (display control) from the CPU 2, which will be described later. The user can view the displayed virtual object as if it existed in the outside world. The light source drive circuit 12 drives the light sources 13a and 13b. Each of the light sources 13a and 13b is a light source that illuminates the user's eyes, and is, for example, an infrared light emitting diode that emits infrared light that is insensitive to the user. A portion of the light emitted from the light sources 13a and 13b and reflected by the user's eyes is focused on the eye imaging device 17 by the light receiving lens 16. These members are provided for each of the left eye and right eye. For example, as the eye image sensor 17, a right image sensor that images the right eye and a left image sensor that images the left eye are provided. The light control device 19 (light control panel) adjusts light from the outside world. The light control circuit 18 is a circuit that changes the transmittance of the light control device 19, and is, for example, a circuit that controls the voltage applied to the light control device 19. The light control device 19 transmits light from the outside at a transmittance controlled by the light control circuit 18 .

FIG. 2 is a block diagram showing the electrical configuration of the display device 100. The CPU 2 is a central processing unit of a microcomputer built into the display device 100, and controls the entire display device 100. A display device 11, a light source drive circuit 12, a line of sight detection circuit 15, a dimming control circuit 18, a memory section 3, and the like are connected to the CPU 2.

The memory unit 3 has a function of storing a video signal from the eye image sensor 17 and a function of storing a line-of-sight correction parameter for correcting individual differences in line-of-sight, which will be described later.

The line of sight detection circuit 15 performs A/D conversion on the output of the eye image sensor 17 (eye image captured by the eye) in a state where an optical image of the eye is formed on the eye image sensor 17, and sends the result to the CPU 2. do. The CPU 2 extracts feature points necessary for line-of-sight detection from the eye image according to a predetermined algorithm described later, and detects the user's line-of-sight from the position of the feature point. For example, the CPU 2 acquires right line of sight information regarding the line of sight of the right eye based on the right eye image obtained by the right image sensor, and acquires right line of sight information regarding the line of sight of the left eye based on the left eye image obtained by the left image sensor. Obtain left gaze information.

<Explanation of gaze detection operation>
The line of sight detection method will be explained using FIGS. 3, 4(A), 4(B), and 5. Both the right eye's line of sight and the left eye's line of sight are detected by the following line of sight detection method. FIG. 3 is a diagram for explaining the principle of the line-of-sight detection method, and is a schematic diagram of an optical system for detecting the line-of-sight. 4(A) is a schematic diagram of an eye image captured by the eye image sensor 17 (an optical image projected onto the eye image sensor 17), and FIG. 4(B) is a schematic diagram of the output intensity of the CCD in the eye image sensor 17. FIG. FIG. 5 shows a schematic flowchart of the line of sight detection operation.

When the line of sight detection operation starts, in step S101 in FIG. 5, the CPU 2 controls the light sources 13a and 13b using the light source drive circuit 12 to emit infrared light toward the user's eyeball 14. An optical image of the user's eye illuminated by the infrared light is formed on the eye image sensor 17 through the light receiving lens 16, and is photoelectrically converted by the eye image sensor 17. This provides a processable electrical signal of the eye image.

In step S102, the CPU 2 acquires an eye image (eye image signal, electric signal of the eye image) from the eye image sensor 17 via the line of sight detection circuit 15.

In step S103, the CPU 2 detects the coordinates of a point corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S102.

The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, corneal reflection images Pd and Pe formed by a portion of the infrared light reflected on the surface of the cornea 142 are collected by the light receiving lens 16 and imaged on the eye imaging device 17, so that the corneal reflection images Pd and Pe are formed by a portion of the infrared light reflected on the surface of the cornea 142. The corneal reflection images become Pd' and Pe'. Similarly, the light beams from the ends a and b of the pupil 141 also form images on the eye imaging device 17, forming pupil end images a' and b' in the eye image.

FIG. 4(B) shows brightness information (brightness distribution) of area α in the eye image of FIG. 4(A). In FIG. 4(B), the horizontal direction of the eye image is the X-axis direction, the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In the first embodiment, the coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction (horizontal direction) are Xd and Xe, and the coordinates of the pupil edge images a' and b' in the X-axis direction are Xa and Xb. As shown in FIG. 4B, an extremely high level of brightness is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the area from the coordinate Xa to the coordinate Xb, which corresponds to the area of the pupil 141 (the area of the pupil image obtained when the light flux from the pupil 141 forms an image on the eye imaging device 17), except for the coordinates Xd and Xe, An extremely low level of brightness is obtained. In the area of the iris 143 outside the pupil 141 (the area of the iris image outside the pupil image obtained by imaging the light flux from the iris 143), a brightness intermediate between the above two types of brightness is obtained. Specifically, a region where the X coordinate (coordinate in the X-axis direction) is larger than the coordinate Xa and a region where the X coordinate is smaller than the coordinate Xb have a brightness intermediate between the above two types of brightness.

From the brightness distribution as shown in FIG. 4(B), the X coordinates Xd, Xe of the corneal reflection images Pd', Pe' and the X coordinates Xa, Xb of the pupil edge images a', b' can be obtained. Specifically, coordinates with extremely high brightness can be obtained as the coordinates of the corneal reflection images Pd', Pe', and coordinates with extremely low brightness can be obtained as the coordinates of the pupil edge images a', b'. . Furthermore, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the light-receiving lens 16 is small, the pupil center image c' (pupil The coordinate Xc of the center of the image can be expressed as Xc≈(Xa+Xb)/2. That is, the coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa, Xb of the pupil edge images a', b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

In step S104, the CPU 2 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of the eyeball 14 with respect to the light-receiving lens 16, and can be calculated using a function of the interval (Xd-Xe) between the corneal reflection images Pd' and Pe'.

In step S105, the CPU 2 calculates the rotation angle of the optical axis of the eyeball 14 with respect to the optical axis of the light receiving lens 16. The X coordinate of the midpoint between the corneal reflection image Pd and the corneal reflection image Pe substantially matches the X coordinate of the center of curvature O of the cornea 142. Therefore, if the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Oc, then the rotation angle θx of the eyeball 14 in the ZX plane (plane perpendicular to the Y-axis) is as follows: It can be calculated using equation 1. The rotation angle θy of the eyeball 14 within the ZY plane (a plane perpendicular to the X-axis) can also be calculated in the same manner as the rotation angle θx.

β×Oc×SINθx≒{(Xd+Xe)/2}−Xc...(Formula 1)

In step S106, the CPU 2 estimates the user's gaze point on the lens 10 using the rotation angles θx and θy calculated in step S105. Assuming that the coordinates (Hx, Hy) of the gaze point are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the gaze point can be calculated using equations 2 and 3 below. The gaze point can also be understood as the position where the line of sight is focused, the position where the user is looking, the gaze position, the gaze position, etc.

Hx=m×(Ax×θx+Bx) (Formula 2)
Hy=m×(Ay×θy+By) (Formula 3)

The parameters m in Equations 2 and 3 are conversion coefficients for converting the rotation angles θx and θy into coordinates corresponding to the pupil center c on the lens 10. It is assumed that the parameter m is determined in advance and stored in the memory unit 3. The parameters Ax, Bx, Ay, and By are line-of-sight correction parameters for correcting individual differences in line-of-sight, and are determined (obtained) by performing a calibration operation that will be described later. It is assumed that the line-of-sight correction parameters Ax, Bx, Ay, and By are stored in the memory unit 3 before the line-of-sight detection operation starts.

In step S107, the CPU 2 stores the coordinates (Hx, Hy) of the gaze point in the memory unit 3, and ends the line of sight detection operation.

Note that the line of sight detection method is not limited to the above method, and may be any method as long as it acquires line of sight information from an eye image, for example. As the final line of sight information, information indicating the direction of the line of sight may be obtained instead of information indicating the gaze point. For example, processing may be performed to obtain the rotation angle (Ax×θx+Bx or Ay×θy+By) without obtaining the coordinates (Hx, Hy) of the gaze point.

<Explanation of the necessity of calibration work>
As described above, the point of gaze can be estimated by acquiring the rotation angles θx and θy of the eyeball 14 from the eye image in the line-of-sight detection operation and converting the coordinates of the position of the pupil center c to the position on the lens 10. FIG. 6A is a diagram showing the user's field of view (the range that the user can see through the lens 10, the imaging range of the external world imaging unit 20), and shows a state in which the display device 11 is in operation. As shown in FIG. 6(A), the display device 11 displays a frame or the like at the current gaze point A (estimated position). The display device 100 controls the display device 11 so as to display information related to the real object that the user is gazing at as a UI (User Interface) according to the results of line-of-sight detection (line-of-sight information, gaze point information). May be controlled. A real object is an object that exists in the external world (real space).

However, due to factors such as individual differences in the shape of human eyeballs, it may not be possible to estimate the gaze point with high accuracy. Specifically, unless the gaze correction parameters Ax, Ay, Bx, and By are adjusted to values suitable for the user, the actual gaze point B and the estimated gaze point C will be different from each other, as shown in FIG. 6(B). A misalignment will occur. In FIG. 6B, the user is gazing at a person, but the display device 100 incorrectly estimates that the user is gazing at the background.

Therefore, before normal use of the display device 100, it is necessary to calibrate the line-of-sight detection function, determine the line-of-sight correction parameters Ax, Ay, Bx, By suitable for the user, and store them in the display device 100.

Conventionally, calibration work has been performed by displaying a plurality of indicators at different positions as shown in FIG. 6(C) before normal use of the display device 100, and having the user look at the indicators. . Then, a line-of-sight detection operation is performed when gazing at each index, and line-of-sight correction parameters Ax, Ay, Bx, By suitable for the user are determined from the calculated multiple gaze points (estimated positions) and the coordinates of each index. The technique is known as a known technique.

However, the condition of the user's eyeballs changes from moment to moment. Even if conventional calibration work is performed, when the state of the eyeball changes, the appropriate line of sight correction parameters Ax, Ay, Bx, By will change, resulting in a deviation between the actual gaze point and the estimated gaze point. te (increase). In other words, as the state of the eyeball changes, the accuracy of line-of-sight detection decreases.

For example, when the environment in which the display device 100 is used changes, such as when the user goes from indoors to outdoors, and the brightness of the outside world changes, the size of the pupil (pupil size, pupil diameter, pupil diameter and radius) changes. . As a result, the appropriate line-of-sight correction parameters Ax, Ay, Bx, By change, and the accuracy of line-of-sight detection decreases. FIG. 7 shows the correspondence between pupil size and appropriate line of sight correction parameter Ax.

Although it is possible to improve the accuracy of line-of-sight detection by performing the calibration work again, it is troublesome to perform the calibration work every time the state of the eyeball changes. Therefore, in the first embodiment, it is possible to detect the line of sight with high accuracy regardless of the state of the user's eyeballs without repeating the calibration work. Note that in FIG. 7, the appropriate line of sight correction parameter Ax increases linearly as the pupil size increases, but this tendency differs from person to person. Therefore, it is difficult to estimate (changes in) the appropriate line-of-sight correction parameters Ax, Bx, Ay, By from (changes in) pupil size, and the line-of-sight correction parameters Ax, Bx, Ay, By are changed during calibration work. Need to decide.

<Explanation of calibration operation>
In the first embodiment, during the calibration work (during calibration), the amount of light that enters the user's eyes (incident light amount) is controlled to each of a plurality of light amounts. For example, during the calibration work, the transmittance of the display device 100 (the transmittance of the light control device 19) is controlled to each of a plurality of transmittances. As shown in FIG. 8, the light control device 19 is provided to cover the front surface of the lens 10. Therefore, by controlling the transmittance of the light control device 19, the amount of light incident on the eye is controlled. Note that the light control device 19 may be provided so as to cover the back surface of the lens 10. Furthermore, the method of controlling the amount of light incident on the eye is not limited to the above method. For example, in the case of a video see-through type display device, display brightness may be controlled.

Then, line-of-sight correction parameters Ax, Bx, Ay, By (parameters used for line-of-sight detection (line-of-sight detection operation, line-of-sight detection function)) are determined for each amount of light incident on the eye. This process can also be regarded as a process of determining the line of sight correction parameters Ax, Bx, Ay, By for each pupil size.

In this way, a correspondence relationship suitable for the user can be determined as a correspondence relationship between the pupil size (the amount of light incident on the eye) and the line of sight correction parameters Ax, Bx, Ay, By. Furthermore, when the environment in which the display device 100 is used changes and the brightness of the outside world changes, the line-of-sight correction parameters Ax, Bx, Ay corresponding to the pupil size (the amount of light incident on the eyes) can be adjusted without performing the calibration work again. , By, line of sight detection can be performed with high accuracy.

A specific example of controlling the transmittance of the light control device 19 during the calibration work will be described using FIGS. 9(A) to 9(C).

First, the display device 100 controls the transmittance of the light control device 19 to the maximum, as shown in FIG. 9(A), and determines the line-of-sight correction parameters Ax, Ay, Bx, and By. Since the transmittance of the light control device 19 is the maximum, the line-of-sight correction parameters Ax, Ay, Bx, By are the line-of-sight correction parameters when the amount of incident light to the eye is the maximum (when the pupil size is the minimum) during the calibration work. It can also be interpreted as

Next, the display device 100 controls the transmittance of the light control device 19 to the minimum, as shown in FIG. 9(B), and determines the line-of-sight correction parameters Ax, Ay, Bx, and By. Since the transmittance of the light control device 19 is the minimum, the line-of-sight correction parameters Ax, Ay, Bx, By are the line-of-sight correction parameters when the amount of light incident on the eye is the minimum (when the pupil size is the largest) during the calibration work. It can also be interpreted as

Furthermore, the display device 100 controls the transmittance of the light control device 19 to 40%, as shown in FIG. 9(C), and determines the line-of-sight correction parameters Ax, Ay, Bx, and By. The line of sight correction parameters Ax, Ay, Bx, By obtained by this are the line of sight correction parameters when the amount of incident light to the eye is between the maximum and minimum (when the pupil size is between the maximum and minimum) during the calibration work. You can also capture it.

The above transmittance of 40% means that the amount of incident light to the eye is between the amount of incident light in the state of FIG. 9(A) and the amount of incident light in the state of FIG. 9(B) (the amount of incident light in the state of FIG. 9(A) The transmittance is such that the range from the light amount to the incident light amount in the state shown in FIG. 9(B) is divided into two (incident light amount). The light amount information indicating the amount of light incident on the eye can be acquired based on the brightness of the eye image, for example. The invention is not limited to this, and the detection result of a sensor that detects the brightness inside the display device 100 (on the user side) may be acquired as the light amount information, or another information based on the detection result may be acquired as the light amount information. Good too. The light amount information may be acquired based on the detection result of a sensor that detects the brightness outside the display device 100 and the transmittance of the light control device 19. From the light amount information obtained in the state of FIG. 9(A) (the state with the maximum transmittance) and the light amount information obtained in the state of FIG. 9(B) (the state with the minimum transmittance), the two light amounts are calculated. It is possible to easily determine (calculate) the transmittance for obtaining an intermediate amount of incident light between the two amounts of incident light indicated by the information.

Note that the maximum, minimum, and 40% transmittances are just examples, and the transmittance of the light control device 19 is not particularly limited. The transmittance of the light control device 19 may be controlled in a number of stages greater than three. The number of stages for controlling the transmittance of the dimming device 19 may be determined by, for example, comparing the amount of light incident on the eye when the transmittance of the dimming device 19 is maximum with one or more threshold values. For example, if the amount of incident light when the light control device 19 has the maximum transmittance is sufficiently large, the number of stages may be increased. By doing so, the pupil size (the amount of incident light on the eye) and the line of sight can be determined over a wide range from the amount of incident light when the light control device 19 has the minimum transmittance to the amount of incident light when the light control device 19 has the maximum transmittance. Information indicating the correspondence with the correction parameters in detail can be obtained. However, increasing the number of stages increases the time required for calibration work. Therefore, if the amount of incident light when the light control device 19 has the maximum transmittance is not so large, the number of stages may be reduced. By doing so, the time required for the calibration work can be shortened without obtaining more detailed information than necessary as information indicating the correspondence between the pupil size (the amount of light incident on the eye) and the line of sight correction parameter.

Furthermore, during the calibration work, the display device 100 may acquire pupil information indicating pupil size. For example, the display device 100 may calculate the distance from the X coordinate Xa of the pupil edge image a' to the X coordinate Xb of the pupil edge image b' as the pupil size.

FIG. 10 is a flowchart of the calibration operation. For example, when a user wears the display device 100 and instructs to start calibration from a menu screen displayed by the display device 100, the calibration operation shown in FIG. 10 starts. At the start of the calibration operation in FIG. 10, for example,

In step S201, the CPU 2 uses the dimming control circuit 18 to control the transmittance of the dimming device 19. For example, the CPU 2 controls the transmittance of the light control device 19 to the maximum.

In step S202, the CPU 2 controls the display device 11 to display a calibration screen including calibration indicators for both eyes of the user. For example, as shown in FIG. 6(C), a plurality of indicators are displayed.

In step S203, the CPU 2 notifies the user of the index to be watched. For example, the CPU 2 changes the shape, color, brightness, etc. of the indicator to be watched among the plurality of indicators to emphasize the indicator. Note that the method of notification of indicators to be watched is not limited to this. For example, only indicators to be watched may be displayed.

In step S204, the CPU 2 determines whether the user has performed a determination operation. The determination operation is performed to notify the display device 100 that the user is gazing at the indicator. Therefore,
The decision operation is performed while the user is looking at the indicator to be focused on. The determining operation is, for example, a button operation on a controller connected to the display device 100 by wire or wirelessly. Note that the determining operation is not particularly limited, and may be an operation of continuing to gaze at the indicator for a longer period of time than a predetermined time, or may be a voice operation. The CPU 2 waits for a decision operation, and when the decision operation is performed, the process proceeds to step S205.

In step S205, the CPU 2 performs the line of sight detection operation shown in FIG. The line of sight detection operation is performed for each of the right eye and left eye. As a result, right line-of-sight information (result of line-of-sight detection operation for the right eye) and left line-of-sight information (result of line-of-sight detection operation for the left eye) are acquired (line-of-sight acquisition).

In step S206, the CPU 2 determines whether the line of sight detection operation has been performed for all indicators. The CPU 2 advances the process to step S207 if there remain indicators for which the line-of-sight detection operation has not been performed, and proceeds to step S208 if the line-of-sight detection operation has been performed for all indicators.

In step S207, the CPU 2 notifies the user of a change in the index to be watched. For example, the CPU 2 switches the indicator to be emphasized among a plurality of indicators, advances the process to step S204, and waits for a determination operation.

In step S208, the CPU 2 determines (calculates) the above-mentioned line-of-sight correction parameters Ax, Bx, Ay, and By from the results of the line-of-sight detection operation for all indicators.

In step S209, the CPU 2 determines whether the line-of-sight correction parameters Ax, Bx, Ay, and By have been determined for all transmittances (transmittances of the light control device 19). If there remains a transmittance for which the line-of-sight correction parameters Ax, Bx, Ay, By have not been determined, the CPU 2 advances the process to step S210, and sets the line-of-sight correction parameters Ax, Bx, Ay, By for all the transmittances. If determined, the calibration operation ends.

In step S210, the CPU 2 uses the dimming control circuit 18 to control (change) the transmittance of the dimming device 19. Then, the CPU 2 advances the process to step S202.

A method of calculating the line of sight correction parameter Bx will be explained. The parameter line-of-sight correction By can be calculated using the same method as the calculation method of the line-of-sight correction parameter Bx.

FIG. 11 shows the inside of the eyeball 14 when the rotation angle θx shown in FIG. 3 is 0 degrees. In FIG. 11, photoreceptor cells, which have the role of sensing light incident on the eyeball 14 and sending signals to the brain, are offset from the optical axis of the eyeball 14 (the optical axis of the cornea 142 and pupil 141) indicated by a dashed line. Therefore, when the user looks at the center of the display surface of the display device 11, the rotation angle θx is an offset angle corresponding to the amount of deviation of the photoreceptor from the optical axis of the eyeball 14, and deviates from 0 degrees. This offset angle (deviation amount of photoreceptor cells) corresponds to the line of sight correction parameter Bx (Bx∝offset angle).

For example, the CPU 2 displays a plurality of indicators on the display screen of the display device 11 as shown in FIG. Make people pay close attention to it. The CPU 2 calculates the rotation angle θx as an offset angle by performing the line of sight detection operation shown in FIG. 5 using the eye image obtained when the user is gazing at the center of the display screen. Then, the CPU 2 calculates the line of sight correction parameter Bx according to the offset angle.

A method for calculating the line-of-sight correction parameter Ax will be explained. The line-of-sight correction parameter Ay can be calculated using the same method as the calculation method of the line-of-sight correction parameter Ax.

In Equation 1, the standard distance Oc (constant) from the corneal curvature center O to the pupil center c is used to calculate the rotation angle θx. However, the actual distance Oc' (variable) from the corneal curvature center O to the pupil center c is not necessarily the same as the distance Oc. The difference between the distance Oc' and the distance Oc becomes an error in the rotation angle θx calculated using Equation 1. The line of sight correction parameter Ax is a parameter for reducing such an error, and is inversely proportional to the actual distance Oc' (Ax∝1/Oc'). The value obtained by dividing the standard distance Oc by the line-of-sight correction parameter Ax becomes the actual distance Oc'.

For example, the CPU 2 calculates the line of sight correction parameter Ax based on a plurality of eye images obtained by capturing the eyeball 14 a plurality of times while the user sequentially gazes at a plurality of positions on the display surface of the display device 11. Specifically, the CPU 2 causes the user to sequentially gaze at two or more indicators at different horizontal positions among the plurality of indicators shown in FIG. 6(C). The CPU 2 calculates the rotation angle θx for each of the two or more eye images corresponding to the two or more indices. The eye image corresponding to the index is an eye image obtained while the user is gazing at the index. Then, the CPU 2 calculates the line of sight correction parameter Ax based on the two or more calculated rotation angles θx. For example, the CPU 2 calculates the line-of-sight correction parameter Ax so that the sum of errors (such as the sum of squares of residuals of the least squares method) of two or more calculated rotation angles θx is minimized. The line of sight correction parameter Ax is calculated so that the errors of each of the two or more calculated rotation angles θx (the difference between the target rotation angle according to the position of the index and the calculated rotation angle θx) are approximately the same. It's okay.

<Explanation of how to determine the line of sight correction parameters to be used>
According to the calibration operation shown in FIG. 10, it is possible to obtain information indicating a correspondence relationship suitable for the user as a correspondence relationship between the pupil size (the amount of light incident on the eye) and the line-of-sight correction parameters Ax, Bx, Ay, and By. Furthermore, the line of sight detection can be performed with high precision using the line of sight correction parameters Ax, Bx, Ay, By corresponding to the pupil size (the amount of light incident on the eye) without performing the calibration work again.

According to the calibration operation shown in FIG. 10, a plurality of combinations of the pupil size (the amount of light incident on the eye) and the line of sight correction parameters Ax, Bx, Ay, By can be obtained while changing the pupil size (the amount of light incident on the eye). The information on the plurality of combinations can also be regarded as a table showing the correspondence between the pupil size (the amount of light incident on the eye) and the line-of-sight correction parameters Ax, Bx, Ay, and By. After the calibration operation, if the pupil size (the amount of incident light on the eye) is the same as during the calibration operation, the line of sight correction parameters Ax, Bx, Ay, By corresponding to the pupil size (the amount of incident light on the eye) are determined from the table above. Obtainable.

However, after the calibration operation, the pupil size (the amount of light incident on the eye) is not necessarily the same as during the calibration operation. After the calibration operation, if the pupil size (the amount of incident light on the eye) is different from the one during the calibration operation, the line of sight correction parameters Ax, Bx, Ay, By corresponding to the pupil size (the amount of incident light on the eye) are determined from the table above. You can't get it. Considering such a case, a function indicating the correspondence between the pupil size (the amount of light incident on the eye) and the line-of-sight correction parameters Ax, Bx, Ay, and By may be estimated based on the plurality of combinations described above. By doing so, no matter what the pupil size (amount of incident light to the eye), the corresponding gaze correction parameters Ax, Bx, Ay, By can be obtained from the function and used to detect the gaze with high accuracy. It can be carried out.

The function indicating the correspondence between the pupil size (the amount of light incident on the eye) and the line-of-sight correction parameters Ax, Bx, Ay, By can be determined, for example, by the least squares method using Equation 4 below. In Equation 4, s is the error and Ri is the i-th pupil size. Ax (Ri) is the line-of-sight correction parameter Ax corresponding to the pupil size Ri, and is the line-of-sight correction parameter Ax determined by the calibration operation. Ax' (Ri) is the line-of-sight correction parameter Ax corresponding to the pupil size Ri, and is the line-of-sight correction parameter Ax determined by a function (candidate) indicating the correspondence between the pupil size and the line-of-sight correction parameter Ax. A function indicating the correspondence between the pupil size and the line-of-sight correction parameter Ax is determined so that the error s (sum of squared residuals) is minimized. Similarly, a function indicating the correspondence between the pupil size and the line-of-sight correction parameters Bx, Ay, and By is also determined.

s=Σ _i=1~n {Ax(Ri)−Ax'(Ri)} ² ...(Formula 4)

Note that the method for determining the function indicating the correspondence between the pupil size (the amount of light incident on the eye) and the line-of-sight correction parameters Ax, Bx, Ay, By is not limited to the least squares method. For example, linear interpolation may be performed to connect the coordinates of multiple combinations (multiple combinations of pupil size (amount of incident light to the eye) and line of sight correction parameters Ax, Bx, Ay, By) obtained through the calibration operation with a straight line. .

<<Example 2>>
Example 2 of the present invention will be described. In Example 1, light amount information indicating the amount of light incident on the user's eyes is acquired, and based on the light amount information, the amount of light incident on the user's eyes is controlled to each of a plurality of light amounts. An example was explained. In the second embodiment, an example will be described in which pupil information indicating the pupil size of the user is acquired and the amount of light incident on the user's eyes is controlled to each of a plurality of light amounts based on the pupil information.

In the second embodiment, during the calibration work, the display device 100 acquires pupil information indicating the pupil size. For example, the display device 100 calculates the distance from the X coordinate Xa of the pupil edge image a' to the X coordinate Xb of the pupil edge image b' as the pupil size.

First, the display device 100 controls the transmittance of the light control device 19 to the maximum and determines the line-of-sight correction parameters Ax, Ay, Bx, and By. Since the transmittance of the light control device 19 is maximum, the line of sight correction parameters Ax, Ay, Bx, By are the line of sight correction parameters when the pupil size is the minimum (when the amount of incident light to the eye is the maximum) during the calibration work. It can also be interpreted as

Next, the display device 100 controls the transmittance of the light control device 19 to the minimum and determines the line-of-sight correction parameters Ax, Ay, Bx, and By. Since the transmittance of the light control device 19 is the minimum, the line-of-sight correction parameters Ax, Ay, Bx, By are the line-of-sight correction parameters when the pupil size is the largest (when the amount of light incident on the eye is the smallest) during the calibration work. It can also be interpreted as

Furthermore, the display device 100 determines that the user's pupil size is based on the pupil information when the transmittance of the dimming device 19 is controlled to the maximum and the pupil information when the transmittance is controlled to the minimum. A transmittance that is intermediate between the two pupil sizes indicated by is determined (calculated). Then, the display device 100 controls the transmittance of the light control device 19 to the determined transmittance, and determines the line-of-sight correction parameters Ax, Ay, Bx, and By.

Note that the three types (three levels) of transmittance described above are just an example, and the transmittance of the light control device 19 is not particularly limited. The transmittance of the light control device 19 may be controlled in a number of stages greater than three. The number of stages for controlling the transmittance of the dimming device 19 may be determined by, for example, comparing the pupil size when the transmittance of the dimming device 19 is maximum with one or more threshold values. For example, if the pupil size when the light control device 19 has the maximum transmittance is sufficiently small, the number of stages may be increased. By doing so, the pupil size (the amount of light incident on the eye) and the line of sight can be determined for a wide range from the pupil size when the light control device 19 has the maximum transmittance to the pupil size when the light control device 19 has the minimum transmittance. Information indicating the correspondence with the correction parameters in detail can be obtained. However, increasing the number of stages increases the time required for calibration work. Therefore, if the pupil size when the light control device 19 has the maximum transmittance is not so small, the number of stages may be reduced. By doing so, the time required for the calibration work can be shortened without obtaining more detailed information than necessary as information indicating the correspondence between the pupil size (the amount of light incident on the eye) and the line of sight correction parameter.

<<Summary>>
As described above, according to Examples 1 and 2, during calibration, the amount of light incident on the user's eyes is controlled to each of a plurality of light amounts. By doing so, it becomes possible to detect the line of sight with high accuracy regardless of the state of the user's eyeballs. Specifically, a correspondence relationship suitable for the user can be determined as a correspondence relationship between the pupil size (the amount of light incident on the eye) and the line-of-sight correction parameters Ax, Bx, Ay, and By. Furthermore, even if the pupil size (the amount of incident light to the eye) changes, the line of sight correction parameters Ax, Bx, Ay, By corresponding to the pupil size (the amount of incident light to the eye) can be used without performing the calibration work again. Line of sight detection can be performed with high accuracy.

Note that Examples 1 and 2 are merely examples, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of Examples 1 and 2 within the scope of the gist of the present invention. A configuration obtained by appropriately combining the configurations of Examples 1 and 2 is also included in the present invention.

<<Other Examples>>
The present invention provides a system or device with a program that implements one or more of the functions of the above-described embodiments via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The disclosure of this embodiment includes the following configuration, method, program, and medium.
(Configuration 1)
comprising a control means for controlling the amount of light incident on the user's eyes;
The electronic device is characterized in that the control means controls the amount of light incident on the user's eyes to each of a plurality of light amounts during calibration of a line-of-sight detection function that obtains line-of-sight information regarding the user's line of sight.
(Configuration 2)
The electronic device according to configuration 1, further comprising a calibration means for performing the calibration.
(Configuration 3)
The line of sight information is information regarding the line of sight of a user who wears an optical see-through display device on his head,
3. The electronic device according to configuration 1 or 2, wherein the control means controls the transmittance of the display device to each of a plurality of transmittances during the calibration.
(Configuration 4)
The electronic device according to configuration 3, wherein the electronic device is the display device.
(Configuration 5)
further comprising an acquisition means for acquiring light amount information indicating the amount of light incident on the user's eyes,
The electronic device according to any one of configurations 1 to 4, wherein the control means controls the amount of light incident on the user's eyes to each of a plurality of light amounts based on the light amount information. .
(Configuration 6)
The line of sight information is information regarding the line of sight of a user who wears an optical see-through display device on his head,
The control means, during the calibration,
(1) controlling the transmittance of the display device to the maximum;
(2) controlling the transmittance of the display device to a minimum;
(3) First light amount information acquired by the acquisition means in a first state where the transmittance of the display device is controlled to a maximum, and a second state where the transmittance of the display device is controlled to a minimum. and the second light amount information acquired by the acquisition means, the amount of light incident on the user's eyes is determined to be the same as the amount of light indicated by the first light amount information and the amount of light indicated by the second light amount information. 5. The electronic device according to configuration 5, wherein the transmittance of the display device is controlled to a transmittance that is intermediate between .
(Configuration 7)
Further comprising an acquisition means for acquiring pupil information indicating the size of the user's pupil,
The electronic device according to any one of configurations 1 to 4, wherein the control means controls the amount of light incident on the user's eyes to each of a plurality of light amounts based on the pupil information. .
(Configuration 8)
The line of sight information is information regarding the line of sight of a user who wears an optical see-through display device on his head,
The control means, during the calibration,
(1) controlling the transmittance of the display device to the maximum;
(2) controlling the transmittance of the display device to a minimum;
(3) First pupil information acquired by the acquisition means in a first state where the transmittance of the display device is controlled to a maximum, and a second state where the transmittance of the display device is controlled to a minimum. and the second pupil information acquired by the acquisition means, so that the size of the pupil is intermediate between the size indicated by the first pupil information and the size indicated by the second pupil information. 8. The electronic device according to configuration 7, wherein the transmittance of the display device is controlled to a transmittance of .
(Configuration 9)
9. The electronic device according to any one of configurations 1 to 8, wherein in the calibration, parameters used for the line of sight detection function are determined for each amount of light incident on the user's eyes.
(Configuration 10)
controlling the amount of light incident on the user's eyes;
In the step, the amount of light incident on the user's eyes is controlled to each of a plurality of light amounts during calibration of a line-of-sight detection function that obtains line-of-sight information regarding the user's line of sight. Method.
(Program: A program for causing a computer to function as each means of the electronic device according to any one of Configurations 1 to 9.
(media)
A computer-readable storage medium storing a program for causing a computer to function as each means of the electronic device according to any one of Configurations 1 to 9.

100: Display device 2: CPU

Claims (12)

comprising a control means for controlling the amount of light incident on the user's eyes;
The electronic device is characterized in that the control means controls the amount of light incident on the user's eyes to each of a plurality of light amounts during calibration of a line-of-sight detection function that obtains line-of-sight information regarding the user's line of sight. The electronic device according to claim 1, further comprising a calibration means for performing the calibration. The line of sight information is information regarding the line of sight of a user who wears an optical see-through display device on his head,
3. The electronic device according to claim 1, wherein the control means controls the transmittance of the display device to each of a plurality of transmittances during the calibration. The electronic device according to claim 3, wherein the electronic device is the display device. further comprising an acquisition means for acquiring light amount information indicating the amount of light incident on the user's eyes,
3. The electronic device according to claim 1, wherein the control means controls the amount of light incident on the user's eyes to each of a plurality of light amounts based on the light amount information. The line of sight information is information regarding the line of sight of a user who wears an optical see-through display device on his head,
The control means, during the calibration,
(1) controlling the transmittance of the display device to the maximum;
(2) controlling the transmittance of the display device to a minimum;
(3) First light amount information acquired by the acquisition means in a first state where the transmittance of the display device is controlled to a maximum, and a second state where the transmittance of the display device is controlled to a minimum. and the second light amount information acquired by the acquisition means, the amount of light incident on the user's eyes is determined to be the same as the amount of light indicated by the first light amount information and the amount of light indicated by the second light amount information. 6. The electronic device according to claim 5, wherein the transmittance of the display device is controlled to a transmittance that is intermediate between . Further comprising an acquisition means for acquiring pupil information indicating the size of the user's pupil,
3. The electronic device according to claim 1, wherein the control means controls the amount of light incident on the user's eyes to each of a plurality of light amounts based on the pupil information. The line of sight information is information regarding the line of sight of a user who wears an optical see-through display device on his head,
The control means, during the calibration,
(1) controlling the transmittance of the display device to the maximum;
(2) controlling the transmittance of the display device to a minimum;
(3) First pupil information acquired by the acquisition means in a first state where the transmittance of the display device is controlled to a maximum, and a second state where the transmittance of the display device is controlled to a minimum. and the second pupil information acquired by the acquisition means, so that the size of the pupil is intermediate between the size indicated by the first pupil information and the size indicated by the second pupil information. 8. The electronic device according to claim 7, wherein the transmittance of the display device is controlled to a transmittance that is equal to the transmittance of the display device. 3. The electronic device according to claim 1, wherein in the calibration, parameters used for the line of sight detection function are determined for each amount of light incident on the user's eyes. controlling the amount of light incident on the user's eyes;
In the step, the amount of light incident on the user's eyes is controlled to each of a plurality of light amounts during calibration of a line-of-sight detection function that obtains line-of-sight information regarding the user's line of sight. Method. A program for causing a computer to execute steps for controlling the amount of light incident on a user's eyes, the program comprising:
The program is characterized in that, in the step, the amount of light incident on the user's eyes is controlled to each of a plurality of light amounts during calibration of a line-of-sight detection function that obtains line-of-sight information regarding the user's line of sight. A computer-readable storage medium storing a program for causing a computer to execute steps for controlling the amount of light incident on a user's eyes, the computer-readable storage medium comprising:
In the storage medium, in the step, the amount of light incident on the user's eyes is controlled to each of a plurality of light amounts during calibration of a line-of-sight detection function that obtains line-of-sight information regarding the user's line of sight.

Images