JP7722016B2 - Gaze detection device, gaze detection method, and gaze detection program - Google Patents

Description

The present invention relates to a gaze detection device, a gaze detection method, and a gaze detection program.

A gaze detection device is known that emits detection light from a light source and irradiates it onto the subject's eyeball, acquires an image of the eyeball illuminated by the detection light, calculates the pupil center and corneal curvature center based on the image of the pupil in the acquired image and the reflected image of the detection light, and detects the vector from the corneal curvature center to the pupil center as the subject's gaze direction (see, for example, Patent Document 1).

Patent No. 4649319

In gaze detection devices like those described above, the reflected image of the detection light irradiated onto the subject's eyeball may exist on the boundary between the cornea and sclera or on the sclera. Because the cornea and sclera have different radii of curvature, in such cases gaze detection accuracy may be reduced.

The present invention has been made in consideration of the above, and aims to provide a gaze detection device, gaze detection method, and gaze detection program that can suppress a decline in detection accuracy.

The gaze detection device of the present invention comprises a display unit that displays an image, a plurality of light sources that emit detection light and irradiate it onto at least one eyeball of the subject, an imaging unit that captures an image of the eyeball irradiated with the detection light, a position detection unit that detects from the captured image the position of the pupil center indicating the center of the pupil of the eyeball irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, a gaze point detection unit that calculates the position of the subject's gaze point based on the positions of the pupil center and the corneal reflex center, and a light source control unit that switches the light source that emits the detection light from among the plurality of light sources based on the target distance between the pupil center and the corneal reflex center.

The gaze detection method according to the present invention includes displaying an image on a display unit, emitting detection light from a plurality of light sources and irradiating it onto at least one eyeball of a subject, capturing an image of the eyeball irradiated with the detection light, detecting from the captured image the position of the pupil center indicating the center of the pupil of the eyeball irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, calculating the position of the subject's point of gaze based on the position of the pupil center and the position of the center of corneal curvature, and switching the light source that emits the detection light from among the plurality of light sources based on the target distance between the pupil center and the corneal reflex center.

The gaze detection program of the present invention causes a computer to execute the following processes: displaying an image on a display unit; emitting detection light from multiple light sources and irradiating it onto at least one eyeball of a subject; capturing an image of the eyeball irradiated with the detection light; detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex; calculating the position of the subject's point of gaze based on the positions of the pupil center and the center of corneal curvature; and switching the light source that emits the detection light from among the multiple light sources based on the target distance between the pupil center and the corneal reflex center.

The present invention provides a gaze detection device, gaze detection method, and gaze detection program that can suppress a decrease in detection accuracy.

FIG. 1 is a perspective view schematically illustrating an example of a gaze detection device according to this embodiment. FIG. 2 is a diagram illustrating an example of the hardware configuration of the gaze detection device according to this embodiment. FIG. 3 is a functional block diagram showing an example of the gaze detection device according to this embodiment. FIG. 4 is a diagram showing an example in which the eyeball is illuminated by the first light source and the second light source. FIG. 5 is a schematic diagram for explaining the principle of the calibration process according to this embodiment. FIG. 6 is a schematic diagram for explaining the principle of the gaze detection process according to this embodiment. FIG. 7 is a diagram showing an example of an eyeball on which a reflected image of detection light is formed. FIG. 8 is a diagram showing an example of an eyeball on which a reflected image of detection light is formed. FIG. 9 is a diagram illustrating an example of the operation of the lighting device in the gaze detection process. FIG. 10 is a diagram illustrating an example of the operation of the lighting device in the gaze detection process. FIG. 11 is a diagram illustrating an example of the operation of the lighting device in the gaze detection process. FIG. 12 is a diagram illustrating an example of the operation of the lighting device in the gaze detection process. FIG. 13 is a diagram illustrating an example of the operation of the lighting device in the gaze detection process. FIG. 14 is a flowchart showing an example of the gaze detection process in the gaze detection method according to this embodiment. FIG. 15 is a diagram showing another example of an eyeball on which a reflected image of detection light is formed. FIG. 16 is a diagram showing another example of the operation of the lighting device in the gaze detection process. FIG. 17 is a diagram showing another example of the operation of the lighting device in the gaze detection process. FIG. 18 is a diagram showing another example of the operation of the lighting device in the gaze detection process. FIG. 19 is a flowchart showing another example of the gaze detection process in the gaze detection method according to this embodiment. FIG. 20 is a flowchart showing another example of the gaze detection process in the gaze detection method according to this embodiment.

Embodiments of the gaze detection device, gaze detection method, and gaze detection program according to the present invention will be described below with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially identical.

In the following explanation, a three-dimensional global coordinate system is set up to explain the positional relationships of each part. The direction parallel to the first axis of the specified plane is the X-axis direction, the direction parallel to the second axis of the specified plane that is perpendicular to the first axis is the Y-axis direction, and the direction parallel to the third axis that is perpendicular to both the first and second axes is the Z-axis direction. The specified plane includes the XY plane.

(Gaze detection device)
1 is a perspective view schematically illustrating an example of a gaze detection device 100 according to this embodiment. As shown in FIG. 1, the gaze detection device 100 includes a display unit 101, a stereo camera device 102, and an illumination device 103.

The display unit 101 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OELD). In this embodiment, the display unit 101 displays an image. In this embodiment, the display unit 101 displays, for example, an index for evaluating the visual function of a subject. The display unit 101 is substantially parallel to the XY plane. The X-axis direction is the left-right direction of the display unit 101, the Y-axis direction is the up-down direction of the display unit 101, and the Z-axis direction is the depth direction perpendicular to the display unit 101.

The stereo camera device 102 has a first camera 102A and a second camera 102B. The stereo camera device 102 is positioned below the display unit 101. The first camera 102A and the second camera 102B are positioned in the X-axis direction. The first camera 102A is positioned in the -X direction relative to the second camera 102B. The first camera 102A and the second camera 102B each include an infrared camera and have an optical system that can transmit near-infrared light with a wavelength of, for example, 850 nm, and an imaging element that can receive that near-infrared light.

The lighting device (light source) 103 has a first light source (lower first light source) 103A, a second light source (lower second light source) 103B, a third light source (upper light source) 103C, and a fourth light source (upper light source) 103D. The first light source 103A and the second light source 103B are arranged below the display unit 101. The third light source 103C and the fourth light source 103D (upper light source) are arranged above the display unit 101.

The first light source 103A and the second light source 103B are arranged in the X-axis direction. The first light source 103A is arranged in the -X direction from the first camera 102A. The second light source 103B is arranged in the +X direction from the second camera 102B. The third light source 103C and the fourth light source 103D are arranged in the X-axis direction. The third light source 103C is arranged in the -X direction from the first camera 102A. The fourth light source 103D is arranged in the +X direction from the second camera 102B.

The first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D each include an LED (light emitting diode) light source and are capable of emitting near-infrared light with a wavelength of, for example, 850 nm. The first light source 103A and the second light source 103B may be disposed between the first camera 102A and the second camera 102B. Similarly, the third light source 103C and the fourth light source 103D may be disposed between the first camera 102A and the second camera 102B in the X direction. The stereo camera device 102 may also be disposed above the display unit 101.

The lighting device 103 emits near-infrared light, which is detection light, to illuminate the subject's eyeball 111. The stereo camera device 102 captures an image of a part of the eyeball 111 (hereinafter collectively referred to as the "eyeball") with the second camera 102B when the eyeball 111 is irradiated with detection light emitted by the first light source 103A or the third light source 103C, and captures an image of the eyeball 111 with the first camera 102A when the eyeball 111 is irradiated with detection light emitted by the second light source 103B or the fourth light source 103D.

A frame synchronization signal is output from at least one of the first camera 102A and the second camera 102B. The first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D emit detection light based on the frame synchronization signal. The first camera 102A captures image data of the eyeball 111 when the detection light emitted from the second light source 103B or the fourth light source 103D is irradiated onto the eyeball 111. The second camera 102B captures image data of the eyeball 111 when the detection light emitted from the first light source 103A or the third light source 103C is irradiated onto the eyeball 111.

When detection light is irradiated onto the eyeball 111, part of the detection light is reflected by the pupil 112, and light from the pupil 112 enters the stereo camera device 102. Furthermore, when detection light is irradiated onto the eyeball 111, a corneal reflection image 113, which is a virtual image of the cornea, is formed on the eyeball 111, and light from the corneal reflection image 113 enters the stereo camera device 102.

By appropriately setting the relative positions of the first camera 102A and the second camera 102B and the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D, the intensity of light incident on the stereo camera device 102 from the pupil 112 is reduced, and the intensity of light incident on the stereo camera device 102 from the corneal reflection image 113 is increased. In other words, the image of the pupil 112 captured by the stereo camera device 102 has low brightness, and the image of the corneal reflection image 113 has high brightness. The gaze detection device 100 can detect the position of the pupil 112 and the position of the corneal reflection image 113 based on the brightness of the captured image.

Figure 2 is a diagram showing an example of the hardware configuration of the gaze detection device 100 according to this embodiment. As shown in Figure 2, the gaze detection device 100 includes a display unit 101, a stereo camera device 102, an illumination device 103, a computer system (control unit) 20, an input/output interface device 30, a drive circuit 40, an output device 50, and an input device 60.

The computer system 20, drive circuit 40, output device 50, and input device 60 communicate data via an input/output interface device 30. The computer system 20 includes an arithmetic processing unit 20A and a memory device 20B. The arithmetic processing unit 20A includes a microprocessor such as a CPU (central processing unit). The memory device 20B includes memory or storage such as ROM (read only memory) and RAM (random access memory). The arithmetic processing unit 20A performs arithmetic processing in accordance with a computer program 20C stored in the memory device 20B.

The drive circuit 40 generates drive signals and outputs them to the display unit 101, the stereo camera device 102, and the lighting device 103. The drive circuit 40 also supplies image data of the eyeball 111 captured by the stereo camera device 102 to the computer system 20 via the input/output interface device 30.

The output device 50 includes a display unit such as a flat panel display. The output device 50 may also include a printing device. The input device 60 generates input data when operated. The input device 60 includes a keyboard or mouse for a computer system. The input device 60 may also include a touch sensor provided on the display screen of the output device 50, which is the display unit.

In this embodiment, the display unit 101 and the computer system 20 are separate devices. However, the display unit 101 and the computer system 20 may be integrated. For example, if the gaze detection device 100 includes a tablet-type personal computer, the computer system 20, input/output interface device 30, drive circuit 40, and display unit 101 may be mounted on the tablet-type personal computer.

Figure 3 is a functional block diagram showing an example of the gaze detection device 100 according to this embodiment. As shown in Figure 3, the input/output interface device 30 has an input/output unit 302. The drive circuit 40 has a display device drive unit 402 that generates a drive signal for driving the display unit 101 and outputs it to the display unit 101, a first camera input/output unit 404A that generates a drive signal for driving the first camera 102A and outputs it to the first camera 102A, a second camera input/output unit 404B that generates a drive signal for driving the second camera 102B and outputs it to the second camera 102B, and a light source drive unit 406 that generates drive signals for driving the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D and outputs them to the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D. In addition, the first camera input/output unit 404A supplies image data of the eyeball 111 captured by the first camera 102A to the computer system 20 via the input/output unit 302. The second camera input/output unit 404B supplies image data of the eyeball 111 captured by the second camera 102B to the computer system 20 via the input/output unit 302.

The computer system 20 controls the gaze detection device 100. The computer system 20 has a display control unit 21, a light source control unit 22, an image data acquisition unit 23, a position detection unit 24, a center of curvature calculation unit 25, a gaze point detection unit 26, a memory unit 27, and an input/output control unit 28. The functions of the computer system 20 are performed by an arithmetic processing unit 20A and a memory device 20B.

The display control unit 21 displays an image to be shown to the subject on the display unit 101. The display control unit 21 can, for example, display target images in the calibration process at multiple positions (target positions) on the display unit 101. The display control unit 21 may sequentially switch the target images one by one to multiple target positions, or may display the target images so that they move sequentially to multiple target positions within the display unit 101. The number of target positions at which the target images are displayed can be set, for example, by the operator inputting the number using the input device 60, etc.

The light source control unit 22 controls the light source drive unit 406 to control the emission and non-emission of light from the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D. The light source control unit 22 switches between the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D to emit detection light based on the target distance between the pupil center and the corneal reflex center. Examples of the target distance include the vertical distance between the pupil center and the corneal reflex center, the horizontal distance between the pupil center and the corneal reflex center, and the shortest distance between the pupil center and the corneal reflex center. When the target distance between the pupil center and the corneal reflex center is equal to or greater than a predetermined threshold while the detection light is being emitted from the first light source 103A and the second light source 103B, the light source control unit 22 can switch to emitting detection light from the third light source 103C and the fourth light source 103D. Furthermore, when the target distance between the pupil center and the corneal reflex center is equal to or greater than a predetermined threshold while detection light is being emitted from the third light source 103C and the fourth light source 103D, the light source control unit 22 can switch to emitting detection light from the first light source 103A and the second light source 103B.

The light source control unit 22 switches between the first light source 103A, the second light source 103B, the third light source 103C, and the fourth light source 103D that emits the detection light, based on whether the shape of the reflected image of the detection light in the image of the eyeball 111 is included in the reference shape. In this embodiment, the reference shape may be a single ellipse. When the light source control unit 22 determines that the shape of the reflected image of the detection light in the image of the eyeball 111 is not included in the reference shape while the detection light is being emitted from the first light source 103A and the second light source 103B, it can switch to emitting the detection light from the third light source 103C and the fourth light source 103D. When the light source control unit 22 determines that the shape of the reflected image of the detection light in the image of the eyeball 111 is not included in the reference shape while the detection light is being emitted from the third light source 103C and the fourth light source 103D, it can switch to emitting the detection light from the first light source 103A and the second light source 103B.

The image data acquisition unit 23 acquires image data of the subject's eyeball 111 captured by the stereo camera device 102, which includes the first camera 102A and the second camera 102B, from the stereo camera device 102 via the input/output unit 302.

The position detection unit 24 detects position data of the pupil center based on the image data of the eyeball 111 acquired by the image data acquisition unit 23. The position detection unit 24 also detects position data of the corneal reflex center based on the image data of the eyeball 111 acquired by the image data acquisition unit 23. The pupil center is the center of the pupil 112. The corneal reflex center is the center of the corneal reflex image 113. The position detection unit 24 detects position data of the pupil center and position data of the corneal reflex center for each of the subject's left and right eyeballs 111.

The curvature center calculation unit 25 calculates position data of the corneal curvature center of the eyeball 111 based on the image data of the eyeball 111 acquired by the image data acquisition unit 23.

The gaze point detection unit 26 detects position data of the subject's gaze point based on image data of the eyeballs 111 acquired by the image data acquisition unit 23. In this embodiment, the gaze point position data refers to position data of the intersection between the subject's gaze vector, defined in a three-dimensional global coordinate system, and the display unit 101. The gaze point detection unit 26 detects the gaze vectors of each of the subject's left and right eyeballs 111 based on position data of the pupil center and position data of the corneal curvature center acquired from the image data of the eyeballs 111. After the gaze vectors are detected, the gaze point detection unit 26 detects position data of the gaze points indicating the intersection between the gaze vector and the display unit 101.

The storage unit 27 stores various data and programs related to the above-mentioned gaze detection. For example, the storage unit 27 can store data about images to be displayed on the display unit 101 for each color and brightness of the background image. The storage unit 27 also stores position data of the gaze point calculated in each calibration process.

The memory unit 27 also stores a gaze detection program that causes the computer to execute the following processes: displaying an image on the display unit 101; emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject; capturing an image of the eyeball 111 irradiated with the detection light; detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex; calculating the position of the subject's gaze point based on the position of the pupil center and the position of the center of corneal curvature; and switching between the light sources that emit detection light from among the multiple light sources based on the target distance between the pupil center and the corneal reflex center.

The memory unit 27 also stores a gaze detection program that causes the computer to execute the following processes: displaying an image on the display unit 101; emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject; capturing an image of the eyeball 111 irradiated with the detection light; detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex; calculating the position of the subject's gaze point based on the position of the pupil center and the position of the center of corneal curvature; and switching the light source that emits the detection light from among the multiple light sources based on whether the shape of the corneal reflex image 113 of the detection light in the image of the eyeball 111 is included in a reference shape.

The input/output control unit 28 outputs data to at least one of the display unit 101 and the output device 50.

Next, an overview of the processing of the center of curvature calculation unit 25 according to this embodiment will be described. In this embodiment, a case will be described in which the eyeball 111 is illuminated by the first light source 103A and the second light source 103B, and the eyeball 111 is photographed by two cameras, the first camera 102A and the second camera 102B. Note that this is not limited to the case where there are two light sources and two cameras, and a similar explanation can be given for the case where there is only one light source and one camera. Below, the principle of the gaze detection method according to this embodiment will be described.

Figure 4 is a diagram showing an example in which the eyeball 111 is illuminated by the first light source 103A and the second light source 103B. As shown in Figure 4, in this embodiment, the first camera 102A and the second light source 103B, and the second camera 102B and the first light source 103A are positioned symmetrically with respect to a line passing through the midpoint between the first camera 102A and the second camera 102B. A virtual light source (reference position of the light source) 103V can be considered to exist at the midpoint between the first camera 102A and the second camera 102B.

The corneal reflection center 121 indicates the corneal reflection center in the image of the eyeball 111 captured by the second camera 102B. The corneal reflection center 122 indicates the corneal reflection center in the image of the eyeball 111 captured by the first camera 102A. The corneal reflection center 124 indicates the corneal reflection center corresponding to the virtual light source 103V.

The position data of the corneal reflection center 124 is calculated based on the position data of the corneal reflection center 121 and the position data of the corneal reflection center 122 photographed by the stereo camera device 102. The stereo camera device 102 detects the position data of the corneal reflection center 121 and the position data of the corneal reflection center 122 in a three-dimensional local coordinate system defined for the stereo camera device 102. Camera calibration is performed in advance for the stereo camera device 102 using a stereo calibration method, and transformation parameters for converting the three-dimensional local coordinate system of the stereo camera device 102 into a three-dimensional global coordinate system are calculated. The transformation parameters are stored in the memory unit 27. The curvature center calculation unit 25 uses the transformation parameters to convert the position data of the corneal reflection center 121 and the position data of the corneal reflection center 122 photographed by the stereo camera device 102 into position data in the three-dimensional global coordinate system. The curvature center calculation unit 25 calculates the position data of the corneal reflection center 124 in the three-dimensional global coordinate system based on the position data of the corneal reflection center 121 and the position data of the corneal reflection center 122 defined in the three-dimensional global coordinate system.

The corneal curvature center 110 exists on a straight line 123 connecting the virtual light source 103V and the corneal reflection center 124. The curvature center calculation unit 25 calculates the position on the straight line 123 where the distance from the corneal reflection center 124 is a predetermined value as the position of the corneal curvature center 110. The corneal curvature radius 109 is used as this predetermined value. The corneal curvature radius 109 is the distance between the corneal surface and the corneal curvature center 110. The value of the corneal curvature radius 109 can be a predetermined value, for example, based on a general corneal curvature radius value.

When using the third light source 103C and the fourth light source 103D, the third light source 103C and the fourth light source 103D are arranged above the display unit 101 so that their X coordinates are the same as those of the first light source 103A and the second light source 103B, and the corneal curvature center 110 is calculated in the same manner as above by assuming that a virtual light source 103V2 (not shown) exists midway between the third light source 103C and the fourth light source 103D.

[Gaze detection method]
In the gaze detection method according to this embodiment, a calibration process is performed, followed by a gaze point detection process. First, the principle of the calibration process will be described. FIG. 5 is a schematic diagram for explaining the principle of the calibration process according to this embodiment. In the calibration process, a target position 130 is set so that the subject will gaze at it. The target position 130 is defined in a three-dimensional global coordinate system. The display control unit 21 displays a target image at the set target position 130.

The first light source 103A and the second light source 103B illuminate the eyeball 111. The first camera 102A and the second camera 102B photograph the eyeball 111. For example, when detection light is emitted from the first light source 103A, the second camera 102B photographs the eyeball 111. When detection light is emitted from the second light source 103B, the first camera 102A photographs the eyeball 111. When the third light source 103C and the fourth light source 103D are used instead of the first light source 103A and the second light source 103B, detection light is emitted from the third light source 103C and the fourth light source 103D, and the eyeball 111 is photographed at different times by the first camera 102A and the second camera 102B, for example. The position detection unit 24 detects position data of the pupil center 112C and the corneal reflex center 113C based on the image data of the eyeball 111 acquired by the image data acquisition unit 23. The position detection unit 24 converts each detected position data into a global coordinate system.

When detection light is emitted from the first light source 103A and the second light source 103B, the curvature center calculation unit 25 calculates the position data of the corneal center of curvature 110 based on the position data of the virtual light source 103V, the position data of the target position 130, the position data of the pupil center 112C, and the position data of the corneal reflection center 113C. Specifically, the curvature center calculation unit 25 determines a first straight line 141 connecting the virtual light source 103V and the corneal reflection center 113C. Furthermore, when detection light is emitted from the third light source 103C and the fourth light source 103D, the curvature center calculation unit 25 calculates the position data of the corneal center of curvature 110 based on the position data of the virtual light source 103V2, the position data of the target position 130, the position data of the pupil center 112C, and the position data of the corneal reflection center 113C.

The center of curvature calculation unit 25 also calculates a second line 142 connecting the target position 130 and the pupil center 112C. The center of curvature calculation unit 25 calculates the intersection of the first line 141 and the second line 142 as position data for the corneal curvature center 110. The center of curvature calculation unit 25 then calculates the distance 127 between the corneal curvature center 110 and the pupil center 112C and stores it in the memory unit 27 as calibration data. In this embodiment, the distance (Ra) between the corneal curvature center 110 and the pupil center 112C when detection light is emitted from the first light source 103A and the second light source 103B, and the distance (Rb) between the corneal curvature center 110 and the pupil center 112C when detection light is emitted from the third light source 103C and the fourth light source 103D are calculated.

Next, the principle of gaze detection processing will be explained. Figure 6 is a schematic diagram for explaining the principle of gaze detection processing according to this embodiment. In gaze detection processing, as in the calibration processing, the eyeball 111 is illuminated using the first light source 103A and the second light source 103B, or the third light source 103C and the fourth light source 103D. The first camera 102A and the second camera 102B capture images of the eyeball 111. The position detection unit 24 detects position data of the pupil center 112C and the corneal reflex center 113C based on the image data of the eyeball 111 acquired by the image data acquisition unit 23.

When detection light is irradiated using the first light source 103A and the second light source 103B, the curvature center calculation unit 25 calculates the position data of the corneal curvature center 110 based on the position data of the virtual light source 103V, the position data of the pupil center 112C, the position data of the corneal reflection center 113C, and the distance 127 between the corneal curvature center 110 and the pupil center 112C calculated in the calibration process. Specifically, the curvature center calculation unit 25 determines the straight line 173 connecting the virtual light source 103V and the corneal reflection center 113C. Similarly, when detection light is emitted using the third light source 103C and the fourth light source 103D, the curvature center calculation unit 25 calculates the position data of the corneal curvature center 110 based on the position data corresponding to the virtual light source 103V2, the position data of the pupil center 112C, the position data of the corneal reflection center 113C, and the distance 127 between the corneal curvature center 110 and the pupil center 112C calculated in the calibration process.

The curvature center calculation unit 25 also determines the position of the corneal curvature center 110, which is a distance away from the pupil center 112C to the inside of the eyeball 111 that corresponds to the distance 127. The gaze point detection unit 26 determines a straight line 178 connecting the pupil center 112C and the corneal curvature center 110, and calculates the position data of the intersection 166 between the straight line 178 and the display unit 101 as the gaze point position data.

Figure 7 shows an example of an eyeball on which a reflection image of the detection light is formed. As shown in the upper part of Figure 7, when the subject looks at a position not far from the virtual light source 103V, for example, and the corneal reflection image 113 of the detection light irradiated onto the subject's eyeball 111 is present on the cornea 111a, the shape of the corneal reflection image 113 will be, for example, a single ellipse with a small oblateness, and the position data of the corneal reflection center can be detected with high accuracy.

As shown in the middle and bottom rows of Figure 7, when the subject looks at a position significantly deviated from the virtual light source 103V, the corneal reflection image 113 of the detection light irradiated onto the subject's eyeball 111 may be located at the boundary between the cornea 111a and the sclera 111b. In this case, due to differences in the radius of curvature and reflectivity, the shape of the corneal reflection image 113 of the detection light may be distorted, for example, as shown in the middle row of Figure 7, becoming an ellipse or oval with a high flattening ratio, or multiple corneal reflection images 113 may be formed, as shown in the bottom row of Figure 7. This may reduce the accuracy of detecting the position data of the corneal reflection center, and may reduce the accuracy of detecting the gaze.

In contrast, in this embodiment, as an example, based on the target distance between the center of the subject's pupil and the center of the corneal reflection, control is performed to switch the light source that emits the detection light so that the reflected image of the detection light is formed at a position that does not extend beyond the subject's cornea 111a. The position at which the reflected image of the detection light is formed is determined, for example, by the position of the subject's eyeball and the positional relationship between the camera and the light source.

Figure 8 is a diagram showing an example of an eyeball on which a reflected image of the detection light is formed. As shown in Figure 8, the light source control unit 22 calculates the target distance between the pupil center 112C and the corneal reflex center 113C, which is calculated based on image data of the eyeball 111. The light source control unit 22 can calculate the target distance, for example, in the vertical direction (Y direction) and the horizontal direction (X direction) between the pupil center 112C and the corneal reflex center 113C. Note that the light source control unit 22 may also calculate the shortest distance between the pupil center 112C and the corneal reflex center 113C as the target distance. Hereinafter, the vertical distance between the pupil center 112C and the corneal reflex center 113C will be referred to as the target distance DY, and the horizontal distance will be referred to as the target distance DX. In the example shown in Figure 8, the light source control unit 22 calculates the target distance DY and the target distance DX. The light source control unit 22 can calculate, for example, the absolute value |Y2-Y1| of the difference between the Y coordinate (Y2) of the pupil center 112C and the Y coordinate (Y1) of the corneal reflection center 113C as the target distance DY. The light source control unit 22 can also calculate, for example, the absolute value |X2-X1| of the difference between the X coordinate (X2) of the pupil center 112C and the X coordinate (X1) of the corneal reflection center 113C as the target distance DX.

In this embodiment, when the subject moves the eyeball 111, the positions of the pupil center 112C, cornea 111a, and sclera 111b change. This changes the relative positional relationship between the corneal reflection image 113 and the pupil center 112C, cornea 111a, and sclera 111b, and the object distances DY and DX change. For example, as shown in FIG. 8, if the corneal reflection image 113 is located to the lower right of the pupil center 112C and the subject moves the eyeball 111 opposite the corneal reflection image 113 (upper left: dashed line in the figure), the relative positional relationship between the corneal reflection image 113 and the pupil center 112C, cornea 111a, and sclera 111b changes, and the object distances DY and DX increase.

In this embodiment, thresholds for the object distance DY and the object distance DX are set in advance so that the corneal reflection image 113 fits within a position on the cornea 111a. The light source control unit 22 may also set the thresholds for the object distance DY and the object distance DX during the calibration process. In this case, for example, the values of the object distance DY and the object distance DX when the subject gazes at the center position of the display screen 101S of the display unit 101 can be used as the thresholds. The light source control unit 22 sets thresholds for when the detection light is emitted from the first light source 103A and the second light source 103B, and thresholds for when the detection light is emitted from the third light source 103C and the fourth light source 103D, respectively. Therefore, if the object distance DY exceeds the threshold when the detection light is emitted from the first light source 103A and the second light source 103B, it can be determined that the subject is gazing above the center position of the display screen 101S. Furthermore, when the target distance DY exceeds a threshold value when the detection light is emitted from the third light source 103C and the fourth light source 103D, it can be determined that the subject is gazing below the center position of the display screen 101S. The light source control unit 22 switches the illumination device 103 that emits the detection light so that the calculated target distance DY and target distance DX are less than a preset threshold value. The light source control unit 22 may control the switching of the illumination device 103 using only one of the target distance DY and the target distance DX. Below, an example will be described in which a threshold value is set for the target distance DY and control is performed so that the target distance DY is less than the threshold value. The threshold value for the target distance DY when the detection light is emitted from the first light source 103A and the second light source 103B is α, and the threshold value for the target distance DY when the detection light is emitted from the third light source 103C and the fourth light source 103D is β. When controlling the switching of the lighting device 103 using both the target distance DY and the target distance DX, control can be performed so that the value of at least one of the target distance DY and the target distance DX is less than a threshold value.

Figures 9 to 13 are diagrams showing an example of the operation of the lighting device 103 in the gaze detection process. When detecting the initial gaze point in the gaze detection process, the light source control unit 22 controls, for example, the first light source 103A and the second light source 103B to emit detection light. As shown in Figure 9, for example, when the target distance DY in the Y direction between the pupil center 112C and the corneal reflex center 113C in the image data of the eyeball 111 is less than the threshold value α, the light source control unit 22 controls the first light source 103A and the second light source 103B, which are bottom light sources, to emit detection light.

On the other hand, as shown in FIG. 10, for example, when the target distance DY in the Y direction between the pupil center 112C and the corneal reflex center 113C in the image data of the eyeball 111 is equal to or greater than the threshold value α, the light source control unit 22 switches to emitting detection light from the third light source 103C and fourth light source 103D, which are upper light sources, as shown in FIG. 11.

The light source control unit 22 can also perform similar control when detection light is emitted from the third light source 103C and fourth light source 103D, which are upper light sources. As shown in FIG. 11, for example, when the target distance DY in the Y direction between the pupil center 112C and the corneal reflex center 113C in the image data of the eyeball 111 is less than the threshold value β, the light source control unit 22 controls so that detection light is emitted from the third light source 103C and fourth light source 103D, which are upper light sources.

On the other hand, as shown in FIG. 12, for example, when the target distance DY in the Y direction between the pupil center 112C and the corneal reflex center 113C in the image data of the eyeball 111 is equal to or greater than the threshold value β, the light source control unit 22 switches to emitting detection light from the first light source 103A and the second light source 103B, which are lower light sources, as shown in FIG. 13.

The light source control unit 22 causes one of the lower light sources, the first light source 103A or the second light source 103B, to emit light to irradiate the eyeball 111 with detection light, and then photographs the subject's eyeball 111 using either the first camera 102A or the second camera 102B, whichever is farthest from the emitting light source. Then, the other of the first light source 103A or the second light source 103B emits light to irradiate the eyeball 111 with detection light, and photographs the subject's eyeball 111 using either the first camera 102A or the second camera 102B, whichever is farthest from the emitting light source. For example, when detection light is emitted from the first light source 103A, the eyeball 111 is photographed by the second camera 102B. Furthermore, when detection light is emitted from the second light source 103B, the eyeball 111 is photographed by the first camera 102A.

Similarly, the light source control unit 22 causes one of the third light source 103C and the fourth light source 103D, which are upper light sources, to emit light to irradiate the eyeball 111 with detection light, and then photographs the subject's eyeball 111 using either the first camera 102A or the second camera 102B, whichever is farthest from the emitting light source. Then, the other of the third light source 103C and the fourth light source 103D emits light to irradiate the eyeball 111 with detection light, and photographs the subject's eyeball 111 using either the first camera 102A or the second camera 102B, whichever is farthest from the emitting light source. For example, when detection light is emitted from the third light source 103C, the eyeball 111 is photographed by the second camera 102B. Furthermore, when detection light is emitted from the fourth light source 103D, the eyeball 111 is photographed by the first camera 102A.

The image data acquisition unit 23 acquires image data. The position detection unit 24 detects position data of the pupil center and corneal reflex center based on the acquired image data. The position detection unit 24 determines whether the position data of the corneal reflex center has been detected correctly. If the reflected image of the detection light is located on the subject's cornea, there is a high possibility that a normal value will be detected. On the other hand, if the reflected image of the detection light is not located on the subject's cornea and, for example, is distorted by extending onto the sclera, there is a low possibility that a normal value will be detected. If a normal value is detected, the curvature center calculation unit 25 and the gaze point detection unit 26 perform their respective processes to acquire gaze point position data. Note that if a normal value is not detected, the gaze point detection unit 26 may, for example, determine that the gaze detection process is an error.

When a normal value for the corneal reflex center is detected, the curvature center calculation unit 25 calculates the corneal center of curvature based on the detected value. At this time, if the light source emitting the detection light in the gaze detection process is a lower light source (first light source 103A and second light source 103B), the curvature center calculation unit 25 calculates the corneal center of curvature using the value of the distance Ra calculated when the light source emitting the detection light is a lower light source, out of the two distances Ra and Rb calculated in the calibration process (the distance between the corneal center of curvature and the pupil center). Furthermore, if the light source emitting the detection light in the gaze detection process is an upper light source (third light source 103C and fourth light source 103D), the curvature center calculation unit 25 calculates the corneal center of curvature using the value of the distance Rb calculated when the light source emitting the detection light is an upper light source.

Next, an example of a gaze detection method according to this embodiment will be described with reference to FIG. 14. FIG. 14 is a flowchart showing an example of gaze detection processing in the gaze detection method according to this embodiment. As shown in FIG. 14, in the gaze detection processing, the light source control unit 22 causes one of the lower light sources (first light source 103A and second light source 103B) and the upper light source (third light source 103C and fourth light source 103D) to emit detection light (step S101). In this case, the light source control unit 22 controls one of the first light source 103A and the second light source 103B to emit light to irradiate the eyeball 111 with the detection light, and then photographs the subject's eyeball 111 using the camera farthest from the emitting light source, either the first camera 102A or the second camera 102B (step S102). Further, under the control of light source control unit 22, the other of first light source 103A and second light source 103B is caused to emit light to irradiate eyeball 111 with detection light, and the subject's eyeball 111 is photographed by either first camera 102A or second camera 102B, whichever is farthest from the emitting light source (step S103). For example, when detection light is emitted from first light source 103A, eyeball 111 is photographed by second camera 102B. When detection light is emitted from second light source 103B, eyeball 111 is photographed by first camera 102A.

The image data acquisition unit 23 acquires image data. The position detection unit 24 detects position data of the pupil center and the corneal reflex center based on the acquired image data. Then, the center of curvature calculation unit 25 and the point of gaze detection unit 26 perform their respective processes to acquire position data of the point of gaze (step S104). In step S104, if the light source that emits the detection light in the gaze detection process is the lower light source (first light source 103A and second light source 103B), the center of curvature calculation unit 25 calculates the corneal center of curvature using the value of the distance Ra calculated when the light source that emits the detection light in the calibration process is the lower light source. Furthermore, if the light source that emits the detection light in the gaze detection process is the upper light source (third light source 103C), the center of curvature calculation unit 25 calculates the corneal center of curvature using the value of the distance Rb calculated when the light source that emits the detection light in the calibration process is the upper light source. Note that if a normal value is not detected in step S104, an error may be determined and the process may be terminated.

After step S104, the gaze point detection unit 26 determines whether or not to terminate gaze point detection (step S105). If the result of the determination in step S105 is that gaze point detection is to be terminated (Yes in step S105), the processing is terminated. On the other hand, if gaze point detection is not to be terminated (No in step S105), the light source control unit 22 determines whether the object distance between the pupil center 112C and the corneal reflex center 113C in the image data of the eyeball 111 is less than a threshold value (step S106). If it is determined that the object distance between the pupil center 112C and the corneal reflex center 113C is less than the threshold value (Yes in step S106), the light source control unit 22 does not switch the light source that emits detection light between the lower light source and the upper light source, and repeats the processing from step S102 onwards (step S107). On the other hand, if it is determined that the target distance between the pupil center 112C and the corneal reflex center 113C is not less than the threshold, the light source control unit 22 switches the light source that emits the detection light between the lower light source and the upper light source, and repeats the processing from step S102 onwards (step S108).

Next, another example of control for switching the light source that emits detection light in this embodiment will be described. Below, we will explain an example of control for switching the light source that emits detection light based on whether the shape of the corneal reflection image 113 of the detection light in the image data of the eyeball 111 is included in the reference shape.

Figure 15 is a diagram showing another example of an eyeball on which a reflection image of the detection light is formed. As shown in the upper part of Figure 15, when the corneal reflection image 113 of the detection light irradiated onto the subject's eyeball 111 is present on the cornea 111a, the shape of the corneal reflection image 113 is, for example, a single ellipse with a small flattening ratio. Therefore, in this embodiment, a single ellipse with a flattening ratio less than a threshold value can be used as the reference shape. For example, the predetermined value can be calculated in advance for when the corneal reflection image 113 is present on the cornea 111a, and the average or minimum value of the calculation results can be used, but this is not limited to this embodiment. When the corneal reflection image 113 of the detection light irradiated onto the subject's eyeball 111 is present on the cornea 111a, the shape of the corneal reflection image 113 will be a shape included in the reference shape. The reference shape can be stored, for example, in the memory unit 27.

On the other hand, if the corneal reflection image 113 of the detection light irradiated onto the subject's eyeball 111 is located at the boundary between the cornea 111a and the sclera 111b, the shape of the corneal reflection image 113 may be, for example, an ellipse or oval with a large flattening ratio, as shown in the middle of Figure 15, or multiple corneal reflection images 113 may be formed, as shown in the bottom of Figure 15. In other words, if the corneal reflection image 113 of the detection light irradiated onto the subject's eyeball 111 is located at the boundary between the cornea 111a and the sclera 111b, the shape of the corneal reflection image 113 will be a shape that is not included in the reference shape.

Based on this, the light source control unit 22 performs image processing on the image data of the eyeball 111 to determine the shape of the corneal reflection image 113 of the detection light. For example, the light source control unit 22 calculates areas in the image data where the brightness exceeds a predetermined value as the areas of the corneal reflection image 113. The light source control unit 22 calculates the number of calculated areas of the corneal reflection image 113. If the calculation result is 2 or more, the light source control unit 22 can determine that the shape of the corneal reflection image 113 is not included in the reference shape.

Also, if the calculation result is 1, the light source control unit 22 determines the shape of the calculated corneal reflection image 113. For example, the light source control unit 22 compares the calculated corneal reflection image 113 with the reference shape Q, and if the rate of match is a predetermined value or more, it can determine that the shape of the corneal reflection image 113 is included in the reference shape. On the other hand, if the light source control unit 22 compares the calculated corneal reflection image 113 with the reference shape Q and the rate of match is less than a predetermined value, it can determine that the shape of the corneal reflection image 113 is not included in the reference shape.

If the light source control unit 22 determines that the shape of the corneal reflection image 113 is not included in the reference shape, it switches the light source that emits the detection light between the lower light source (first light source 103A, second light source 103B) and the upper light source (third light source 103C, fourth light source 103D). On the other hand, if the light source control unit 22 determines that the shape of the corneal reflection image 113 is included in the reference shape, it does not switch the light source that emits the detection light.

Figures 16 to 18 are diagrams showing other examples of the operation of the lighting device 103 in the gaze detection process. The light source control unit 22 controls, for example, the first light source 103A and the second light source 103B to emit detection light. For example, it determines whether the shape of the corneal reflection image 113 of the detection light in the image data of the eyeball 111 is included in the reference shape.

When the light source control unit 22 determines that the shape of the corneal reflection image 113 is included in the reference shape, it does not switch the light source that emits the detection light, as shown in FIG. 16. On the other hand, when it determines that the shape of the corneal reflection image 113 is not included in the reference shape, as shown in FIG. 17, the light source control unit 22 switches so that the detection light is emitted from the third light source 103C and the fourth light source 103D, which are upper light sources, as shown in FIG. 18.

Figure 19 is a flowchart showing another example of the gaze detection process in the gaze detection method according to this embodiment. As shown in Figure 19, steps S201 to S205 are the same as steps S101 to S105 (see Figure 14) in the case of control for switching the light source that emits detection light based on the target distance between the subject's pupil center and the corneal reflex center.

If the result of the determination in step S205 is that detection of the gaze point is to be terminated (Yes in step S205), the processing is terminated. If detection of the gaze point is not to be terminated (No in step S205), the light source control unit 22 determines whether the shape of the corneal reflection image 113 in the image data of the eyeball 111 is included in the reference shape (step S206). If it is determined that the shape of the corneal reflection image 113 is included in the reference shape (Yes in step S206), the light source control unit 22 does not switch the light source that emits detection light between the lower light source and the upper light source, and repeats the processing from step S202 onwards (step S207). On the other hand, if it is determined that the shape of the corneal reflection image 113 is not included in the reference shape, the light source control unit 22 switches the light source that emits detection light between the lower light source and the upper light source, and repeats the processing from step S102 onwards (step S208).

Figure 20 is a flowchart showing another example of the gaze detection process in the gaze detection method according to this embodiment. The example shown in Figure 20 shows a combination of control for switching the light source that emits detection light based on the target distance between the subject's pupil center and the corneal reflex center, and control for switching the light source that emits detection light based on whether the shape of the corneal reflex image 113 is included in the reference shape.

As shown in Figure 20, steps S301 to S304 are similar to steps S101 (S201) to S104 (S204) in the angle control described above (see Figures 14 and 19).

After step S304, the gaze point detection unit 26 determines whether the gaze point has been detected normally (step S305). If it is determined in step S305 that the gaze point has been detected normally (Yes in step S305), the gaze point detection unit 26 and light source control unit 22 perform the processes of steps S306 to S309. Steps S306 to S309 are similar to the processes of steps S105 to S108 in controlling the switching of the light source that emits detection light based on the target distance between the subject's pupil center and the corneal reflex center (see FIG. 14).

On the other hand, if it is determined in step S305 that the gaze point has not been detected normally (No in step S305), the light source control unit 22 performs the processes of steps S310 to S312. Steps S310 to S312 are similar to the processes of steps S206 to S208 in controlling the switching of the light source that emits detection light based on whether the shape of the corneal reflection image 113 is included in the reference shape (see FIG. 19).

Control for switching the light source that emits detection light based on the target distance between the subject's pupil center and corneal reflex center can be performed based on the normal values of the pupil center position and corneal reflex center position when normal values are calculated for the pupil center position and corneal reflex center position, i.e., when the gaze point position is detected normally. On the other hand, control for switching the light source that emits detection light based on whether the shape of the corneal reflex image 113 is included in the reference shape can be performed based on the shape of the corneal reflex image 113 without calculating the pupil center position and corneal reflex center position. Therefore, even when normal values cannot be obtained for the pupil center position and corneal reflex center position, control for switching the light source that emits detection light can be performed appropriately.

As described above, the gaze detection device 100 according to this embodiment includes a display unit 101 that displays an image, multiple light sources (first light source 103A, second light source 103B, third light source 103C, fourth light source 103D) that emit detection light and illuminate at least one eyeball 111 of the subject, a stereo camera device 102 that captures an image of the eyeball 111 illuminated with the detection light, a position detection unit 24 that detects, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball 111 illuminated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, a gaze point detection unit 26 that calculates the position of the gaze point of the subject based on the positions of the pupil center and the corneal reflex center, and a light source control unit 22 that switches between the multiple light sources that emit detection light based on the target distance between the pupil center and the corneal reflex center.

The gaze detection method according to this embodiment also includes displaying an image on the display unit 101, emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject, capturing an image of the eyeball 111 irradiated with the detection light, detecting from the captured image the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, calculating the position of the subject's gaze point based on the position of the pupil center and the position of the center of corneal curvature, and switching the light source that emits the detection light from among the multiple light sources based on the target distance between the pupil center and the corneal reflex center.

In addition, the gaze detection program according to this embodiment causes a computer to execute the following processes: displaying an image on the display unit 101; emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject; capturing an image of the eyeball 111 irradiated with the detection light; detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex; calculating the position of the subject's gaze point based on the position of the pupil center and the position of the center of corneal curvature; and switching between the light sources emitting detection light from among the multiple light sources based on the target distance between the pupil center and the corneal reflex center.

With the configuration of this embodiment, the light source that emits the detection light can be switched from among multiple light sources based on the target distance between the pupil center and the corneal reflection center, preventing the reflected image of the detection light irradiated onto the subject's eyeball from extending outside the cornea. This prevents a decrease in gaze detection accuracy.

In the gaze detection device 100 according to this embodiment, the multiple light sources include a first light source 103A and a second light source 103B installed below the display unit 101, and a third light source 103C and a fourth light source 103D installed above the display unit 101, the distance being the target distance DY in the vertical direction, and the light source control unit 22 switches the light source that emits the detection light between the lower light source and the upper light source. With this configuration, the light source that emits the detection light is switched between the lower light source and the upper light source based on the target distance DY in the vertical direction, so that the reflected image of the detection light can be reliably positioned within the subject's cornea.

In the gaze detection device 100 according to this embodiment, the light source control unit 22 switches the light source that emits the detection light based on whether the shape of the corneal reflection image 113 of the detection light in the image of the eyeball 111 is included in the reference shape. This allows the light source that emits the detection light to be switched appropriately even when the positions of the pupil center and the corneal reflection center cannot be obtained.

The gaze detection device 100 according to this embodiment also includes a display unit 101 that displays an image, multiple light sources (first light source 103A, second light source 103B, third light source 103C, fourth light source 103D) that emit detection light and irradiate it onto at least one eyeball 111 of the subject, a stereo camera device 102 that captures an image of the eyeball 111 irradiated with the detection light, a position detection unit 24 that detects, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, a gaze point detection unit 26 that calculates the position of the gaze point of the subject based on the position of the pupil center and the position of the corneal reflex center, and a light source control unit 22 that switches the light source that emits the detection light from among the multiple light sources based on whether the shape of the corneal reflex image 113 of the detection light in the image of the eyeball 111 is included in a reference shape.

The gaze detection method according to this embodiment also includes displaying an image on the display unit 101, emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject, capturing an image of the eyeball 111 irradiated with the detection light, detecting from the captured image the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex, calculating the position of the subject's point of gaze based on the position of the pupil center and the position of the center of corneal curvature, and switching the light source that emits the detection light from among the multiple light sources based on whether the shape of the corneal reflex image 113 of the detection light in the image of the eyeball 111 is included in a reference shape.

In addition, the gaze detection program according to this embodiment causes the computer to execute the following processes: displaying an image on the display unit 101; emitting detection light from multiple light sources and irradiating it onto at least one eyeball 111 of the subject; capturing an image of the eyeball 111 irradiated with the detection light; detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball 111 irradiated with the detection light and the position of the corneal reflex center indicating the center of the corneal reflex; calculating the position of the subject's gaze point based on the position of the pupil center and the position of the center of corneal curvature; and switching the light source that emits the detection light from among the multiple light sources based on whether the shape of the corneal reflex image 113 of the detection light in the image of the eyeball 111 is included in the reference shape.

With the configuration of this embodiment, the light source that emits the detection light can be switched from among multiple light sources based on whether the shape of the corneal reflection image 113 of the detection light in the image of the eyeball 111 is included in the reference shape, thereby preventing the reflection image of the detection light irradiated onto the subject's eyeball from extending outside the cornea. This prevents a decrease in the accuracy of gaze detection.

In the gaze detection device 100 according to this embodiment, the reference shape is a single ellipse with a flattening ratio less than a threshold value, and the light source control unit 22 switches the light source that emits the detection light when it determines that the shape of the corneal reflection image 113 is not included in the reference shape. With this configuration, the light source that emits the detection light is switched when it is determined that the shape of the corneal reflection image 113 is not included in the range of a single ellipse with a flattening ratio less than the threshold value, so that the corneal reflection image 113 of the detection light can be positioned within the cornea of the subject.

In the gaze detection device 100 according to this embodiment, the multiple light sources include a first light source 103A and a second light source 103B installed below the display unit 101, and a third light source 103C and a fourth light source 103D installed above the display unit 101, and the light source control unit 22 switches the light source that emits the detection light between the lower light source and the upper light source. This configuration ensures that the reflected image of the detection light is positioned within the subject's cornea.

Although the embodiments of the present invention have been described above, the embodiments are not limited to the content of these embodiments. Furthermore, the components described above include those that would be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of what is known as equivalents. Furthermore, the components described above can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the embodiments described above.

For example, in the above embodiment, a configuration was described in which the multiple light sources include lower light sources (first light source 103A, second light source 103B) arranged below the display unit 101 and upper light sources (third light source 103C, fourth light source 103D) arranged above the display unit 101, but this is not limiting. For example, in addition to the lower light sources and upper light sources, or instead of at least one of these light sources, light sources may be arranged on the left and right sides of the display unit 101.

In addition, in the above embodiment, a configuration in which two light sources (third light source 103C and fourth light source 103D) are provided as upper light sources has been described as an example, but this is not limited to this. For example, a configuration in which one light source is provided as the upper light source may also be used.

In addition, in the above embodiment, a configuration in which the first camera 102A and the second camera 102B are arranged below the display unit 101 has been described as an example, but this is not limiting. The first camera 102A and the second camera 102B may also be arranged above or to the side of the display unit 101.

In the above embodiment, the light source control unit 22 emits detection light from either the lower light source (first light source 103A, second light source 103B) or the upper light source (third light source 103C, fourth light source 103D). In addition, the light source control unit 22 may emit detection light from both the lower light source and the upper light source in sequence, and calculate the distance between the pupil center and the corneal reflex center in the image data of the eyeball 111. In this case, the gaze point detection unit 26 may detect the gaze point using image data when detection light is emitted from either the lower light source or the upper light source, whichever light source produces the shorter calculated distance. Furthermore, the light source control unit 22 may emit detection light from both the lower light source and the upper light source in sequence, and calculate the shape of the corneal reflex image in the image data of the eyeball 111. In this case, the gaze point detection unit 26 may detect the gaze point using image data when detection light is emitted from either the lower light source or the upper light source, whichever light source produces the calculated corneal reflex image shape closest to the reference shape.

This disclosure includes matters that contribute to the realization of the SDG's goal of "Good health and well-being for all" and contribute to value creation through healthcare products and services.

DX, DY...object distance, Q...reference shape, Ra, Rb, 127...distance, 20...computer system, 20A...arithmetic processing device, 20B...storage device, 20C...computer program, 21...display control unit, 22...light source control unit, 23...image data acquisition unit, 24...position detection unit, 25...center of curvature calculation unit, 26...gazing point detection unit, 27...storage unit, 28...input/output control unit, 30...input/output interface device, 40...drive circuit, 50...output device, 60...input device, 100...gaze detection device, 101...display unit, 101S...display screen, 102...stereo camera device, 102A...first camera, 102B...second camera, 10 2C, 103C...third light source, 103...illumination device, 103A...first light source, 103B...second light source, 103D...fourth light source, 103V...virtual light source, 109...corneal curvature radius, 110...corneal curvature center, 111...eyeball, 111a...cornea, 111b...sclera, 112...pupil, 112C...pupil center, 113...corneal reflection image, 113C, 121, 122, 124...corneal reflection center, 123, 173, 178...straight line, 130...target position, 141...first straight line, 142...second straight line, 166...intersection, 302...input/output unit, 402...display device driver, 404A...first camera input/output unit, 404B...second camera input/output unit, 406...light source driver

Claims (5)

a display unit for displaying an image;
a plurality of light sources that emit detection light and irradiate it onto at least one eyeball of the subject;
an imaging unit that captures an image of the eyeball irradiated with the detection light;
a position detection unit that detects, from the captured image, the position of a pupil center indicating the center of the pupil of the eyeball irradiated with the detection light and the position of a corneal reflection center indicating the center of the corneal reflection;
a gaze point detection unit that calculates the position of the gaze point of the subject based on the position of the pupil center and the position of the corneal curvature center of the eyeball irradiated with the detection light ;
a light source control unit that switches the light source that emits the detection light among the plurality of light sources based on a target distance between the pupil center and the corneal reflex center. the plurality of light sources include a lower light source disposed below the display unit and an upper light source disposed above the display unit;
the target distance is a distance in the vertical direction,
The gaze detection device according to claim 1 , wherein the light source control unit switches the light source that emits the detection light between the lower light source and the upper light source. The gaze detection device according to claim 1 , wherein the light source control unit switches the light source that emits the detection light based on whether a shape of a reflection image of the detection light in the image of the eyeball is included in a reference shape. Displaying an image on a display unit;
emitting detection light from a plurality of light sources and irradiating the light onto at least one eyeball of the subject;
capturing an image of the eyeball illuminated with the detection light;
Detecting, from the captured image, a position of a pupil center indicating the center of a pupil of the eyeball irradiated with the detection light and a position of a corneal reflection center indicating the center of a corneal reflection;
calculating a position of the gaze point of the subject based on the position of the pupil center and the position of the corneal curvature center of the eyeball irradiated with the detection light ;
and switching the light source that emits the detection light from among the plurality of light sources based on an object distance between the pupil center and the corneal reflex center. A process of displaying an image on a display unit;
a process of emitting detection light from a plurality of light sources and irradiating it onto at least one eyeball of the subject;
capturing an image of the eyeball illuminated with the detection light;
A process of detecting, from the captured image, the position of the pupil center indicating the center of the pupil of the eyeball irradiated with the detection light and the position of the corneal reflection center indicating the center of the corneal reflection;
a process of calculating the position of the gaze point of the subject based on the position of the pupil center and the position of the corneal curvature center of the eyeball irradiated with the detection light ;
and a process of switching the light source that emits the detection light among the plurality of light sources based on a target distance between the pupil center and the corneal reflex center.