JP2012187178A - Visual line detection device and visual line detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a visual line detection device that correctly detects a position on which a user fixes his/her eyes even when a distance between the user and the device varies during calibration and during practice of visual line detection processing.SOLUTION: The visual line detection device 1 includes: an imaging part 12 that generates an image obtained by photographing at least a partial area of the user's face, which is the area including the eyes, during lighting of a light source 11; a distance characteristic quantity extraction part 26 that obtains distance characteristic quantities to be changed according to a distance from a display part 2 to the user's face from the image; a distance estimation part 27 that estimates the distance from the display part 2 to the user's face, on the basis of the distance characteristic quantities; a Purukinje image detection part 24 that detects the cornea reflection image of the light source and the pupil gravity of the user from the area of the eye included in the image; a visual line detection part 25 that detects the user's visual direction on the basis of a positional relationship between the pupil gravity and the cornea reflection image; and an eye-fixing position detection part 28 that obtains the eye-fixing position of the user on the display part 2, on the basis of the direction of the visual line and the distance from the display part to the user's face.

Description

  The present invention relates to a gaze detection apparatus and a gaze detection method that detect a gaze direction by detecting, for example, a cornea reflection image.

2. Description of the Related Art Conventionally, a line-of-sight detection device that detects a user's line-of-sight direction by detecting a cornea reflection image (Purkinje image) of a light source by analyzing an image obtained by photographing a user's eyes has been developed.
In such a line-of-sight detection device, a so-called calibration process is performed before the user uses a device on which the line-of-sight detection device is mounted in order to accurately detect the position at which the user gazes. In the calibration process, for example, the line-of-sight detection device measures the difference between the position of the Purkinje image on the image and the position of the pupil when the user gazes in a plurality of directions specified in advance, and the difference and gaze are measured in the reference table. Store the location in association with each other. When the line-of-sight detection process is performed, the line-of-sight detection apparatus determines a gaze position corresponding to the difference between the measured Purkinje image position and the pupil position by referring to the reference table. Then, according to the gaze position obtained by the line-of-sight detection device, the device on which the line-of-sight detection device is mounted executes a control operation such as screen scrolling, thereby reducing the user's operation.

  However, the positional relationship between the device on which the visual line detection device is mounted and the user when executing the visual line detection processing is different from the positional relationship between the device on which the visual line detection device is mounted and the user when executing the calibration processing. Sometimes. In particular, when the gaze detection device is mounted on a device that the user holds with his / her hand, such as a mobile phone or a portable information terminal, the distance between the device and the user at the time of execution of the gaze detection process is always kept constant. It is difficult. And if the distance between a user and an apparatus changes, even if a gaze direction is the same, the position where a user gazes will change. For this reason, there is a possibility that the line-of-sight detection device cannot accurately detect the position at which the user gazes even if the reference table created at the time of calibration is used. Therefore, a technique has been proposed in which the distance from the line-of-sight detection device to the user is measured and the position at which the user gazes is detected based on the distance and the line-of-sight direction (see, for example, Patent Documents 1 and 2).

For example, the apparatus according to an example of such a known technique measures the distance from the apparatus to the operator from an image obtained by photographing the operator using an autofocus function. And the apparatus calculates | requires the position which an operator gazes on the screen of a display part based on the difference of the position on the image between the distance and an iris and a Purkinje image.
An apparatus according to an example of another known technique extracts a plurality of feature points and contours of a face from an image obtained by photographing a user's face, obtains a face orientation based on a positional relationship between the plurality of feature points, The distance from the device to the user's face is obtained based on the size of the device. The apparatus executes control such as scrolling, enlarging, and reducing the display screen of the display unit according to the orientation of the face or the distance from the apparatus to the user's face.

Japanese Patent Laid-Open No. 6-187091 JP 2009-31368 A

However, in order to measure the distance from the device to the operator using the autofocus function, the device changes the subject distance from the optical system having a shallow depth of field to the object point focused on the image sensor. Is provided with a mechanism for moving the optical system. This complicates the structure of the entire device.
In addition, the device that controls the screen display based on the facial feature points or contours extracted from the image detects that the position where the user is gazing has moved when the user moves only the line of sight without moving the face. Because it was not possible, there was a risk that the screen display could not be properly controlled.

  Therefore, the present specification provides a line-of-sight detection device that can accurately detect the position where the user is gazing even when the distance between the user and the device is different between calibration and execution of the line-of-sight detection process. Objective.

  According to one embodiment, a line-of-sight detection device is provided. The line-of-sight detection device includes: a light source that irradiates a user's eyes; and an imaging unit that generates an image obtained by capturing at least a part of the user's face, which is an area including the eyes while the light source is on. A storage unit for storing distance information representing a relationship between a distance from the display unit to the user's face and a distance feature amount on the image that changes according to the distance; and a distance feature amount extraction for obtaining a distance feature amount from the image A distance estimation unit that estimates the distance from the display unit corresponding to the distance feature amount to the user's face by referring to the distance information, and the cornea reflection image of the light source and the user's Purkinje image detection unit that detects the center of gravity of the pupil, gaze detection unit that detects the user's gaze direction according to the positional relationship between the center of gravity of the pupil and the cornea reflection image, and the gaze direction and the distance from the display unit to the user's face Based on the user on the display And a gaze position detector for determining the gaze position.

The objects and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the appended claims.
It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention as claimed.

  The line-of-sight detection device disclosed in this specification can accurately detect the position where the user is gazing even when the distance between the user and the device is different between calibration and execution of the line-of-sight detection process.

It is a hardware block diagram of the portable terminal carrying the gaze detection apparatus by 1st Embodiment. It is a schematic front view of the portable terminal carrying the gaze detection apparatus by 1st Embodiment. It is a functional block diagram of the control part of the gaze detection apparatus by a 1st embodiment. It is a figure which shows an example of a reference table. (A) is a mimetic diagram showing a user's face on an image in case a subject distance is the 1st distance comparatively near. (B) is a schematic diagram showing the face of the user on the image when the subject distance is a second distance that is farther than the first distance. It is a graph showing the experimental result of the relationship between the space | interval between the gravity centers of the right and left nostrils on an image, and object distance. It is a figure which shows an example of a gaze position table. It is an operation | movement flowchart of a calibration process. It is an operation | movement flowchart of a gaze detection process. (A) represents an example of an image showing a Purkinje image, and (b) represents the relationship between the number of pixels with saturated luminance values and the subject distance among the pixels along the horizontal line of (a). It is a graph showing the experimental result of. It is a schematic front view of the portable terminal carrying the gaze detection apparatus by 3rd Embodiment. It is a graph showing the experimental result of the relationship between the space | interval between Purkinje images of two light sources, and object distance. It is a functional block diagram of the control part of the gaze detection apparatus by a 4th embodiment. It is an operation | movement flowchart of an inclination detection process.

Hereinafter, the gaze detection apparatus according to the first embodiment will be described with reference to the drawings.
This line-of-sight detection device analyzes an image obtained by capturing the Purkinje image of the light source, and detects the user's line-of-sight direction based on the distance between the Purkinje image and the center of gravity of the pupil. Furthermore, the line-of-sight detection device is a distance feature quantity that changes according to the distance from the display unit of the device on which the line-of-sight detection device is mounted to the user's face on the image. An interval is obtained, and a distance from the display unit to the user's face is obtained according to the interval. And this gaze detection apparatus pinpoints the user's gaze position on the screen of a display part based on the distance from the display part of the apparatus by which a gaze detection apparatus is mounted to a user's face, and a user's gaze direction. In this embodiment, the difference between the position of the front principal point of the optical system of the camera of the device equipped with the line-of-sight detection device and the display unit of the device is ignored with respect to the distance from the user's face to the display unit. As small as possible. Therefore, hereinafter, for the sake of convenience, the distance from the display unit of the device equipped with the line-of-sight detection device to the user's face is referred to as a subject distance.

  FIG. 1 is a hardware configuration diagram of a portable terminal equipped with the visual line detection device according to the first embodiment. The mobile terminal 1 includes a user interface 2, a memory 3, a control unit 4, a housing 5, and a line-of-sight detection device 10. The mobile terminal 1 is, for example, a mobile phone or a mobile information terminal. Furthermore, the mobile terminal 1 may have a communication interface circuit (not shown) for connecting the mobile terminal 1 to another device. Note that FIG. 1 is a diagram for explaining the constituent elements of the mobile terminal 1 and is not a diagram showing the actual arrangement of the constituent elements of the mobile terminal 1.

The user interface 2 is an example of a display unit and includes, for example, a touch panel display. The user interface 2 is arranged on the front surface of the housing 5 such that the display screen of the user interface 2 faces the user facing the front surface of the housing 5. The user interface 2 displays various icons or operation buttons, for example, in response to a control signal from the control unit 4. In addition, when the user touches the position of the displayed icon, the user interface 2 generates an operation signal corresponding to the position and outputs the operation signal to the control unit 4.
The user interface 2 may have a display device such as a liquid crystal display and a plurality of operation switches such as a keypad.

  The memory 3 includes, for example, a readable / writable nonvolatile semiconductor memory. The memory 3 stores various application programs executed on the control unit 4 and various data. Further, the memory 3 stores information representing an icon or an operation button area being displayed on the display screen of the user interface 2.

Furthermore, the memory 3 also functions as a storage unit of the line-of-sight detection device 10. And the memory 3 memorize | stores the various data utilized in order to detect the position where a user gazes. For example, the memory 3 stores a reference table representing the relationship between the relative position of the pupil center of gravity with respect to the center of gravity of the Purkinje image and the user's line-of-sight direction. Further, the memory 3 stores a distance reference table that represents the relationship between the distance feature amount extracted from the image and the subject distance. Further, the memory 3 stores a gaze position table that represents the relationship between the subject distance and the line-of-sight direction and the position where the user gazes. Furthermore, the memory 3 stores a program for executing the line-of-sight detection process.
The memory circuit that functions as the storage unit of the line-of-sight detection device 10 may be mounted on the mobile terminal 1 separately from the memory 3.

  The line-of-sight detection device 10 obtains the position where the user is gazing on the display screen of the user interface 2. Then, the line-of-sight detection device 10 passes information indicating the position where the user is gazing to the application program being executed by the control unit 4. Details of the line-of-sight detection device 10 will be described later.

The control unit 4 includes one or a plurality of processors and their peripheral circuits. And the control part 4 is connected with each part of the portable terminal 1 via a signal line, and controls the portable terminal 1 whole. For example, the control unit 4 executes processing according to the operation signal received from the user interface 2 and the application program being executed.
The control unit 4 also functions as a control unit of the line-of-sight detection device 10. The function and operation as the control unit of the line-of-sight detection device 10 will be described later.
Further, the control unit 4 is an example of an application execution unit. For example, each position on the display screen of the user interface 2 stored in the memory 3 and the position at which the user is gazing, which is obtained by the line-of-sight detection device 10. Compare the icon display area. When the position where the user is gazing and the display area of any icon or operation button overlap, the control unit 4 executes processing corresponding to the icon or operation button.

FIG. 2 is a schematic front view of the mobile terminal 1. On the display screen 2a of the user interface 2 provided in the approximate center of the portable terminal 1, a map 201 and scroll buttons 202 to 205 for scrolling the screen around the map 201 are displayed.
In this case, if the position at which the user gazes overlaps the display area of any of the scroll buttons 202 to 205, the control unit 4 scrolls the screen in a preset direction with respect to the scroll button. Further, when the user's gaze position is within the area where the map 201 is displayed and the user brings his / her face close to the display screen 2a, the control unit 4 may enlarge the map on the user interface 2.

The line-of-sight detection device 10 includes a light source 11, a camera 12, a memory 3, and a control unit 4 in order to detect a user's line-of-sight direction and a gaze position on the display screen 2a.
The light source 11 illuminates the face of the user who operates the mobile terminal 1, particularly the eyes and surroundings thereof. Therefore, the light source 11 includes, for example, at least one infrared light emitting diode and a drive circuit that supplies power from a power source (not shown) to the infrared light emitting diode in accordance with a control signal from the control unit 4. The light source 11 emits illumination light while receiving a control signal for turning on the light source from the control unit 4.
Note that the number of the light sources 11 included in the line-of-sight detection device 10 is not limited to one, and the line-of-sight detection device 10 may include a plurality of light sources arranged at different positions.

  The camera 12 is an example of an imaging unit, and generates an image in which at least a part of the user's face including the eyes is captured. For this purpose, the camera 12 includes an image sensor having a solid-state imaging device arranged in a two-dimensional manner having sensitivity to light from the light source 11, and an imaging optical system that forms an image of a user's eye on the image sensor. . The camera 12 further includes a visible light cut filter between the image sensor and the imaging optical system in order to suppress detection of an image reflected by the iris and a corneal reflection image of light from a light source other than the light source 11. May be.

Further, the number of pixels of the image sensor of the camera 12 can be set to, for example, 1 million, 2 million, or 4 million so that the image has a resolution that can identify the Purkinje image of the light source 11. An image generated by the image sensor is a color image, and each pixel has three color components of red (R), green (G), and blue (B). Each color component takes any value from 0 to 255, for example. The larger the color component value, the higher the luminance of the color. Alternatively, the image generated by the image sensor may be a monochrome image in which the luminance of each pixel increases as the light intensity increases.
The camera 12 captures the user's face illuminated by the lit light source by photographing the user's face at a predetermined cycle while the light source 11 is lit while the line-of-sight detection process is being performed. Generated images. The camera 12 outputs the generated image to the control unit 4 every time an image is generated.

Referring again to FIG. 2, the camera 12 is disposed in the vicinity of the upper end of the display screen 2 a of the user interface 2 of the mobile terminal 1. The camera 12 is directed so that the user's face facing the display screen 2a can be photographed.
The light source 11 is arranged on the outer periphery of the user interface 2 so as to illuminate the face of the user facing the display screen 2a. In this example, the light source 11 is disposed close to the upper end of the user interface 2 and aligned with the camera 12.

As described above, the control unit 4 is also a control unit of the mobile terminal 1 and is connected to each unit of the visual line detection device 10 through a signal line. And the control part 4 controls each part of the visual line detection apparatus 10 by transmitting a control signal to each part of the visual line detection apparatus 10 through a signal line.
Further, the control unit 4 executes a gaze direction detection process for obtaining a position where the user is gazing while an application using information on the position where the user is gazing is executed on the mobile terminal 1. The control unit 4 also determines the relative positional relationship between the Purkinje image of the light source and the pupil center of gravity, the user's line-of-sight direction, and the like at a specific timing such as when the mobile terminal 1 is started or when the above-described application is started. A calibration process for obtaining the relationship is executed.

FIG. 3 is a functional block diagram related to the line-of-sight detection process of the control unit 4. The control unit 4 includes a light source control unit 21, a camera control unit 22, a face detection unit 23, a Purkinje image detection unit 24, a line-of-sight detection unit 25, a distance feature quantity extraction unit 26, and a distance estimation unit 27. A gaze position detection unit 28 and a calibration unit 29 are provided.
Each of these units included in the control unit 4 is a functional module realized by a computer program executed on a processor included in the control unit 4. Each of these units included in the control unit 4 may be formed as a separate circuit from the processor included in the control unit 4. Alternatively, these units included in the control unit 4 may be mounted on the mobile terminal 1 as a single integrated circuit in which circuits corresponding to the units are integrated, separately from the processor included in the control unit 4. In this case, the integrated circuit may include a storage circuit that stores a reference table or the like separately from the memory 3 and functions as a storage unit included in the visual line detection device 10.

  The light source control unit 21 turns on or off the light source 11. For example, the light source control unit 21 turns on the light source 11 during the line-of-sight detection process. Further, when a plurality of light sources 11 are included, the light source control unit 21 may transmit a control signal to each light source so that two or more light sources are turned on simultaneously. When a plurality of light sources are turned on at the same time, the Purkinje image also includes a plurality of light source images that are turned on. Therefore, the line-of-sight detection device 10 uses a light source in which the arrangement pattern of bright spots on the image is actually turned on. It can be checked whether or not it matches the arrangement pattern. Therefore, the line-of-sight detection device 10 can easily distinguish between an image of another light source such as an indoor lamp or the sun and an image of the light source included in the line-of-sight detection device 10.

  The camera control unit 22 controls the timing at which the camera 12 captures an image during the execution of the line-of-sight detection process or the calibration process. When notified from the light source control unit 21 that the light source 11 is turned on, the camera control unit 22 transmits a shooting instruction signal that causes the camera 12 to perform shooting at a predetermined cycle. The camera control unit 22 temporarily stores the image in the memory 3 every time an image is received from the camera 12.

The face detection unit 23 performs an operation on the image received from the camera 12 on the image received from the camera 12 in order to identify a region where the user's eyes may be captured on the image during the line-of-sight detection process or the calibration process. Detect the area where the face is reflected.
In the present embodiment, at the time of shooting, the user's face is illuminated with infrared light from the light source 11, and skin reflectance with respect to infrared light is relatively high (for example, skin reflectance is near infrared. (Several tens of percent in the wavelength range), the brightness of the pixels in which the skin of the face is reflected on the image is high. On the other hand, since the area of the hair or the user's back area on the image has a low reflectance with respect to infrared light or is far from the light source, the luminance of the pixel in which the hair or the area behind the user is reflected is relatively low.
Therefore, when the value of each pixel of the image is a color image represented by the RGB color system, the face detection unit 23 converts the value of each pixel into a value represented by the YUV color system. Then, the face detection unit 23 extracts pixels whose luminance component (Y component) value of each pixel is equal to or greater than a predetermined threshold as face area candidate pixels that may have a face. When the value of each pixel of the image is a monochrome image representing luminance, the face detection unit 23 compares the value of each pixel with a predetermined threshold value. For example, the predetermined threshold is set to a value obtained by multiplying the maximum value of the luminance component on the image by 0.8.

When the operation of the mobile terminal 1 is executed according to the user's line of sight, the user's face is facing the front of the mobile terminal 1 and at a position approximately several tens of centimeters away from the front of the mobile terminal 1. It is assumed that there is. Therefore, the area occupied by the user's face on the image is relatively large, and the size of the area occupied by the face on the image is estimated to some extent.
Therefore, the face detection unit 23 performs a labeling process on the face area candidate pixels, and sets a set of face area candidate pixels adjacent to each other as a face candidate area. Then, the face detection unit 23 determines whether the size of the face candidate area is included in a reference range corresponding to the size of the user's face. If the size of the face candidate area is included in the reference range corresponding to the size of the user's face, the face detection unit 23 determines that the face candidate area is a face area in which the user's face is reflected.
Note that the size of the face candidate area is represented by, for example, the maximum number of pixels in the horizontal direction of the face candidate area. In this case, the reference range is set to, for example, from 1/4 to 2/3 of the number of pixels in the horizontal direction of the image. Alternatively, the size of the face candidate area may be represented by the number of pixels included in the face candidate area, for example. In this case, the reference range is set to, for example, 1/16 or more to 4/9 or less of the number of pixels of the entire image.

  The face detection unit 23 may add not only the size of the face candidate area but also the shape of the face candidate area to the determination condition for determining the face candidate area as the face area. A human face generally has a substantially elliptical shape. Therefore, the face detection unit 23, for example, when the size of the face candidate area is included in the reference range and the circularity of the face candidate area is equal to or greater than a predetermined threshold corresponding to a general face contour. Alternatively, the face candidate area may be a face area. The face detection unit 23 obtains the total number of pixels located on the contour of the face candidate area as the perimeter of the face candidate area, and multiplies the total number of pixels in the face candidate area by 4π as the square of the perimeter. The circularity can be calculated by dividing.

  Alternatively, the face detection unit 23 may approximate the face candidate area to an ellipse by applying the least square method by applying the coordinates of each pixel on the contour of the face candidate area to an elliptic equation. The face detection unit 23 may use the face candidate area as a face area when the ratio of the major axis to the minor axis of the ellipse is included in the range of the ratio of the major axis to the minor axis of the face. Note that, when the shape of the face candidate region is evaluated by ellipse approximation, the face detection unit 23 may detect an edge pixel corresponding to an edge by performing an inter-pixel calculation on the luminance component of each pixel of the image. . In this case, the face detection unit 23 connects the edge pixels using, for example, a labeling process, and uses the edge pixels connected to a certain length or more as the contour of the face candidate region.

  The face detection unit 23 may detect the face area according to any of various other methods for detecting the face area shown on the image. For example, the face detection unit 23 performs template matching between the face candidate area and a template corresponding to a general face shape, calculates the degree of coincidence between the face candidate area and the template, and the degree of coincidence is predetermined. If the value is greater than or equal to the value, the face candidate area may be determined as a face area.

When the face area can be extracted, the face detection unit 23 generates information representing the face area. For example, the information representing the face area may be a binary image having the same size as the image and having different pixel values for the pixels in the face area and the pixels outside the face area.
The face detection unit 23 passes information representing the face area to the Purkinje image detection unit 24.

The Purkinje image detection unit 24 detects a region in which the user's eyes are reflected in the face region during execution of the line-of-sight detection processing or calibration processing, and detects a Purkinje image in the region in which the eyes are reflected.
The luminance of the pixel corresponding to the eye is significantly different from the luminance of the pixel corresponding to the periphery of the eye. Therefore, the Purkinje image detection unit 24 detects, for each pixel in the face region, an edge pixel whose luminance changes in the vertical direction by performing a difference calculation between adjacent pixels in the vertical direction using, for example, a Sobel filter. The Purkinje image detection unit 24 sets, for example, a region surrounded by two edge lines in which a predetermined number or more of edge pixels are connected in a substantially horizontal direction corresponding to the size of the eye.
Alternatively, the Purkinje image detection unit 24 detects a region that most closely matches the template in the face region by template matching between the template representing the image of the eye on the image and the face region, and uses the detected region as the eye region. It is good.

  Further, the Purkinje image detection unit 24 detects an area where the pupil is reflected in the eye area. In the present embodiment, the Purkinje image detection unit 24 performs template matching between a template corresponding to the pupil and the eye region, and detects a region having the highest degree of matching with the template in the eye region. Then, the Purkinje image detection unit 24 determines that the pupil is reflected in the detected area when the maximum value of the coincidence is higher than a predetermined coincidence threshold. A plurality of templates may be prepared according to the size of the pupil. In this case, the Purkinje image detection unit 24 executes template matching between each template and the eye region, and obtains the highest value of the matching degree. If the highest coincidence value is higher than the coincidence degree threshold, the Purkinje image detection unit 24 determines that the pupil appears in an area overlapping the template corresponding to the highest coincidence value. Note that the degree of coincidence is calculated as, for example, a normalized cross-correlation value between a template and a region overlapping the template. The coincidence threshold is set to 0.7 or 0.8, for example.

The luminance of the region where the pupil is reflected is lower than the luminance of the surrounding region, and the pupil is substantially circular. Therefore, the Purkinje image detection unit 24 sets two concentric rings with different radii in the eye region. When the difference value obtained by subtracting the average luminance value of the inner pixels from the average luminance value of the pixels corresponding to the outer ring is larger than a predetermined threshold value, the Purkinje image detection unit 24 is surrounded by the inner ring. The region may be the pupil region. The Purkinje image detection unit 24 may add to the condition for detecting the pupil region that the average luminance value of the region surrounded by the inner ring is equal to or less than a predetermined threshold value. In this case, for example, the predetermined threshold is set to a value obtained by adding 10% to 20% of the difference between the maximum luminance value and the minimum luminance value in the eye region to the minimum luminance value.
When the Purkinje image detection unit 24 succeeds in detecting the pupil region, the Purkinje image detection unit 24 calculates the average value of the horizontal coordinate values and the average value of the vertical coordinate values of each pixel included in the pupil region as the coordinates of the center of gravity of the pupil region. . On the other hand, if the Purkinje image detection unit 24 fails to detect the pupil region, the Purkinje image detection unit 24 returns a signal indicating that to the control unit 4.

Further, the Purkinje image detection unit 24 detects the Purkinje image of the light source 11 in the eye region. The luminance of the region where the Purkinje image of the light source 11 is reflected is higher than the luminance of the surrounding region, and the luminance value is substantially saturated (that is, the luminance value is approximately the luminance value that the pixel value can take). It is the highest value). In addition, the shape of the region where the Purkinje image of the light source 11 is reflected substantially matches the shape of the light emitting surface of each light source. Therefore, the Purkinje image detection unit 24 sets two rings that have a shape that substantially matches the contour shape of the light emitting surface of the light source in the eye region, and that have different sizes and centers. Then, the Purkinje image detection unit 24 obtains a difference value obtained by subtracting the average value of the luminance of the outer pixels from the average value of the luminance of the pixels corresponding to the inner ring. When the difference value is larger than the predetermined difference threshold value and the inner luminance average value is higher than the predetermined luminance threshold value, the Purkinje image detection unit 24 defines the area surrounded by the inner ring as the Purkinje image of the light source 11. To do. Note that the difference threshold value can be, for example, an average value of difference values between neighboring pixels in the eye region. The predetermined luminance threshold can be set to 80% of the maximum luminance value in the eye region, for example.
Note that the Purkinje image detection unit 24 may detect the region in which the pupil is shown using any of various other methods for detecting the region in which the pupil is shown on the image. Similarly, the Purkinje image detection unit 24 detects the region where the Purkinje image of the light source is captured using any of various other methods for detecting the region where the Purkinje image of the light source is captured on the image. Also good.

When the Purkinje image detection unit 24 succeeds in detecting the Purkinje image of the light source 11, the average value of the horizontal coordinate values and the average value of the vertical coordinate values of each pixel included in the light source are used as the coordinates of the center of gravity of the Purkinje image. calculate. On the other hand, if the Purkinje image detection unit 24 fails to detect the Purkinje image of the light source, the Purkinje image detection unit 24 returns a signal indicating that to the control unit 4.
The Purkinje image detection unit 24 notifies the calibration unit 29 of the center of gravity of the Purkinje image and the center of the pupil when performing the calibration process. On the other hand, the Purkinje image detection unit 24 notifies the gaze detection unit 25 of the center of gravity of the Purkinje image and the center of the pupil when performing the gaze detection process.

  The line-of-sight detection unit 25 detects the user's line-of-sight direction based on the center of gravity of the Purkinje image and the center of the pupil during the line-of-sight detection process.

Since the surface of the cornea is substantially spherical, the position of the Purkinje image of the light source is substantially constant regardless of the line-of-sight direction. On the other hand, the center of gravity of the pupil moves according to the user's line-of-sight direction. Therefore, the line-of-sight detection unit 25 can detect the user's line-of-sight direction by obtaining the relative position of the pupil center of gravity with respect to the center of gravity of the Purkinje image.
In the present embodiment, the line-of-sight detection unit 25 determines the relative position of the pupil centroid relative to the centroid of the Purkinje image of the light source, for example, the horizontal direction of the centroid of the Purkinje image from the horizontal coordinate and the vertical coordinate of the pupil centroid. It is obtained by subtracting the coordinate and the vertical coordinate. The line-of-sight detection unit 25 determines the user's gaze position by referring to a reference table that represents the relationship between the relative position of the pupil center of gravity and the user's line-of-sight direction.
The line-of-sight detection unit 25 notifies the gaze position detection unit 28 of the user's line-of-sight direction.

  FIG. 4 is a diagram illustrating an example of a reference table. In each column of the left column of the reference table 400, the coordinates of the relative position of the pupil centroid relative to the centroid of the Purkinje image of the light source are represented. In each column in the right column of the reference table 400, the line-of-sight direction of the user corresponding to the coordinates of the relative position of the pupil centroid in the same row is represented. In this example, the line-of-sight direction is represented by the difference in angle between the horizontal direction and the vertical direction from the reference direction, with the line-of-sight direction when the user is gazing at the reference point on the display screen 2a as the reference direction. The Note that the coordinates of the relative position of the center of gravity of the pupil are expressed in units of pixels on the image.

  The distance feature amount extraction unit 26 extracts a distance feature amount that changes according to the subject distance from the image. Here, the size of the user's face on the image obtained by the camera 12 decreases as the subject distance increases. Therefore, the distance between characteristic parts of the face on the image also changes according to the subject distance.

  This will be described with reference to FIGS. 5 (a) and 5 (b). FIG. 5A is a schematic diagram showing the user's face on the image when the subject distance is a relatively short first distance. FIG. 5B is a schematic diagram showing the user's face on the image when the subject distance is the second distance that is longer than the first distance. As shown in FIGS. 5A and 5B, the images 500 and 510 include the left and right nostrils 501 of the user's face, respectively. Since the subject distance at the time of shooting the image 500 is shorter than the subject distance at the time of shooting the image 510, the area of the user's face shown in the image 500 is larger than the area of the user's face shown in the image 510. Is also big. Therefore, the interval Δn between the left and right nostrils 501 shown in the image 500 is wider than the interval Δf between the left and right nostrils 501 shown in the image 510. Thus, it can be seen that the greater the subject distance, the shorter the distance between the left and right nostrils on the image. Similarly, as the subject distance is shorter, the distance between the centers of the eyes on the image, the distance between the mouth and the nose, the distance between the center of any eye and the mouth, or the distance between the left and right mouth corner points, etc. The distance on the image between characteristic parts becomes large.

FIG. 6 is a graph showing the experimental results of the relationship between the distance between the center of gravity of the right and left nostrils on the image and the subject distance. In FIG. 6, the horizontal axis represents the subject distance, and the vertical axis represents the interval (pixel unit) between the center of gravity of the right and left nostrils. Each point represents the distance between the center of gravity of the right and left nostrils at a subject distance of 300 mm to 600 mm. In this experiment, we used an image of a face photographed with a 1 / 3.2-inch (horizontal 4.73 mm, vertical 3.52 mm) 2-megapixel CMOS sensor and a camera with an imaging optical system with a diagonal field angle of 58 °. It was. As indicated by each point, the distance between the center of gravity of the right and left nostrils decreases as the subject distance increases. Further, the distance between the center of gravity of the right and left nostrils when the subject distance is maximum (600 mm) shown by the point 601 is shown, and the right and left nostrils when the subject distance is minimum (300 mm) shown by the point 602 is also shown. The distance between the center of gravity is about 4 times. Thus, it can be seen that the distance between the center of gravity of the right and left nostrils on the image varies greatly depending on the subject distance. Therefore, the distance between the center of gravity of the right and left nostrils on the image can be used as the distance feature amount. Similarly, the distance between the centers of the left and right eyes, the distance between the nose tip and the center of the mouth, and the like can also be used as the distance feature amount.
Therefore, in the present embodiment, the distance feature quantity extraction unit 26 detects at least two characteristic parts such as eyes, nostrils, and corner points of the mouth in the face region, and determines the distance between the characteristic parts as a distance feature. Amount.

The distance feature amount extraction unit 26 performs template matching while changing the relative position between a face region and a template corresponding to a general shape of a characteristic part such as an eye, a nostril, a mouth, and the like. The degree of coincidence between the area and the template is calculated. For example, the nostril template is set so as to have a substantially circular area with low luminance and a high luminance area around it. The distance feature amount extraction unit 26 determines that a part corresponding to the template is captured at the specific position when the degree of coincidence is a predetermined value or more at the specific position in the face region. The distance feature quantity extraction unit 26 obtains positions on the image of at least two parts, and uses the distance between the parts as a distance feature quantity.
The distance feature amount extraction unit 26 passes the distance feature amount to the distance estimation unit 27 when the line-of-sight detection process is executed. On the other hand, the distance feature amount extraction unit 26 notifies the calibration unit 29 of the distance feature amount when the calibration process is executed.

The distance estimation unit 27 estimates the subject distance based on the distance feature amount. In the present embodiment, the distance estimation unit 27 estimates the subject distance corresponding to the distance feature amount by referring to a distance reference table that is stored in advance in the memory 3 and represents the relationship between the distance feature amount and the subject distance. To do.
Alternatively, the distance estimation unit 27 may obtain the subject distance by inputting the distance feature amount obtained by the distance feature amount extraction unit 26 into the relational expression between the distance feature amount and the subject distance. The distance reference table and the relational expression between the distance feature amount and the subject distance are examples of distance information.
The distance estimation unit 27 notifies the gaze position detection unit 28 of the subject distance when the line-of-sight detection process is executed. On the other hand, the distance estimation unit 27 notifies the calibration unit 29 of the distance when the calibration process is executed.

The gaze position detection unit 28 detects a position where the user is gazing on the display screen 2a based on the user's line-of-sight direction and subject distance. Hereinafter, for the sake of convenience, a position where the user is gazing on the display screen 2a is simply referred to as a gazing position.
In the present embodiment, the gaze position detection unit 28 determines the user's gaze position by referring to a gaze position table that represents the relationship between the user's gaze direction and subject distance and the user's gaze position on the display screen 2a. To do.

  FIG. 7 is a diagram illustrating an example of a gaze position table. In the left column of the gaze position table 700, the subject distance is represented. In addition, the line at the upper end of the gaze position table 700 represents the user's line-of-sight direction. In each column of the gaze position table 700, the coordinates of the gaze position on the display screen 2a corresponding to the subject distance in the same row and the line-of-sight direction in the same column are expressed in units of pixels on the display screen 2a. For example, the column 701 of the gaze position table 700 indicates that the gaze position is (cx, cy + 40) when the distance is 200 mm and the line-of-sight direction is 0 ° in the horizontal direction and 1 ° in the vertical direction. Note that cx and cy are the gaze position when the line-of-sight direction is (0, 0), that is, the coordinates of the reference gaze position, for example, the horizontal and vertical coordinates of the center of the display screen 2a.

Alternatively, the gaze position detection unit 28 obtains the gaze position with respect to the line-of-sight direction as a reference gaze position with reference to the gaze position table when the subject distance is a predetermined reference distance, and determines the reference gaze position according to the subject distance. It may be corrected. The reference distance is, for example, a subject distance designated at the time of executing the calibration process, and is set to 300 mm. In this case, the horizontal coordinate vx and the vertical coordinate vy of the gaze position calculated by the correction are calculated by the following equations.
vx = R (vrx-cx) + cx
vy = R (vry-cy) + cy
Here, vrx and vry are the horizontal coordinate and vertical coordinate of the reference gaze position on the display screen, respectively, and cx and cy are the coordinates of the gaze position when the line-of-sight direction is (0, 0). R is the ratio of the distance estimated by the distance estimation unit 27 to the reference distance.
The gaze position detection unit 28 notifies the application program being executed by the control unit 4 of the user's gaze position.

The calibration unit 29 updates the reference table and the distance reference table by executing a calibration process.
For example, when the calibration unit 29 receives a signal indicating that the mobile terminal 1 is activated from the control unit 4 or receives a signal indicating that a specific application is activated, the calibration unit 29 performs the calibration process. Execute.

When the calibration process is started, the calibration unit 29 displays a calibration mark at a predetermined reference position on the display screen 2 a of the user interface 2. The reference position can be, for example, either the center of the display screen 2a or the four corners of the display screen 2a.
Further, the calibration unit 29 determines a specified distance from the display screen 2a to the user's face in order to instruct how far away from the display screen 2a of the portable terminal 1 the face is to be arranged during calibration. It is displayed on the display screen 2a.
During the execution of calibration, it is assumed that the user gazes at the calibration mark displayed on the display screen 2a.

  The calibration unit 29 receives, from the Purkinje image detection unit 24, the coordinates of the center of gravity of the Purkinje image and the center of the pupil detected based on the image taken when the mark is displayed at the center of the display screen 2a. Then, the calibration unit 29 updates the reference table so that the relative position of the pupil center of gravity relative to the center of the Purkinje image corresponds to the line-of-sight direction when the user gazes at the center of the display screen 2a. Note that the coordinates of the gaze position and the line-of-sight direction correspond one-to-one. Therefore, for example, the calibration unit 29 sets the viewing direction when the user gazes at the center of the display screen 2a to 0 ° in both the horizontal direction and the vertical direction.

Further, the calibration unit 29 obtains the coordinates of the center of the Purkinje image and the center of the pupil detected based on the image taken when the mark is displayed in the upper left corner of the display screen 2a from the Purkinje image detection unit 24. receive. The calibration unit 29 then refers to the reference table so that the relative position of the pupil center of gravity based on the center of the Purkinje image at this time corresponds to the line of sight when the user gazes at the upper left corner of the display screen 2a. Update. Similarly, the calibration unit 29 determines the relative position of the pupil center of gravity with respect to the center of the Purkinje image on the captured image when the mark is displayed at another corner of the display screen 2a, and the user determines the display position of the mark. The reference table is updated so as to be associated with the line-of-sight direction at the time of gaze.
Further, the calibration unit 29 calculates the relative position of the pupil centroid relative to the centroid of the Purkinje image with respect to other line-of-sight directions as the relative position of the pupil centroid corresponding to the positions of the five marks already obtained. For example, by linear interpolation. Then, the calibration unit 29 updates the reference table so as to associate the relative positions of the pupil centroids with respect to various gaze directions.

  Further, the calibration unit 29 receives the distance feature amount at the specified distance from the distance feature amount extraction unit 26. The calibration unit 29 then updates the distance reference table so that the distance feature amount is associated with the specified distance. Further, the calibration unit 29 may cause the camera 12 to create an image of the user's face by changing the designated distance. In this case, the calibration unit 29 receives from the distance feature amount extraction unit 26 distance feature amounts extracted from images taken at the respective specified distances. Then, the calibration unit 29 updates the distance reference table so that each specified distance is associated with the distance feature amount at the specified distance.

If the distance feature amount is a distance on the image between two specific parts of the face as in the present embodiment, the subject distance and the distance feature amount are inversely proportional. Therefore, in this case, the calibration unit 29 multiplies the distance feature amount obtained for the specified distance d0 by the ratio (d0 / d) of the specified distance d0 to the subject distance d. You may ask for. Then, the calibration unit 29 may update the distance reference table based on the distance feature amount calculated by the calculation and the corresponding subject distance. In this case, the line-of-sight detection device 10 only needs to photograph the user's face for one specified distance, so that the labor of calibration processing can be reduced.
The calibration unit 29 stores the updated reference table and distance reference table in the memory 3.

  In addition, when the individual difference regarding the distance feature amount is so small as to be negligible, a distance reference table in advance based on a distance feature amount extracted from an image obtained by photographing a general face model in advance or a distance feature amount calculated by simulation. May be created. In this case, the calibration unit 29 does not have to update the distance reference table.

FIG. 8 shows an operation flowchart of the calibration process executed by the control unit 4.
The control unit 4 displays a predetermined mark at the reference position on the display screen 2a of the user interface 2 (step S101). Next, the control part 4 acquires the image which image | photographed the user's face in the state which the light source 11 turned on from the camera 12 (step S102). The face detection unit 23 of the control unit 4 extracts a face region where a face is shown on the image (step S103). Then, the face detection unit 23 passes information representing the face area to the Purkinje image detection unit 24 and the distance feature quantity extraction unit 26 of the control unit 4.

  The Purkinje image detection unit 24 detects an eye region within the face region (step S104). Then, the Purkinje image detector 24 detects the center of gravity of the pupil in the eye region (step S105). The Purkinje image detection unit 24 detects the Purkinje image of the light source that is lit in the eye region (step S106).

The Purkinje image detection unit 24 determines whether or not the pupil center of gravity and the Purkinje image of the light source have been successfully detected (step S107). If the Purkinje image detection unit 24 fails to detect the center of the pupil or the Purkinje image of the light source (No at Step S107), the control unit 4 instructs the camera 12 to perform re-imaging (Step S108). And the control part 4 re-executes the process after step S102.
On the other hand, when the Purkinje image detection unit 24 has successfully detected the pupil center of gravity and the Purkinje image of the light source (Yes in step S107), the Purkinje image detection unit 24 calibrates the center of gravity of the Purkinje image of the light source and the pupil center of gravity of the control unit 4. Notification to the application unit 29.
The calibration unit 29 obtains the relative position of the pupil centroid relative to the centroid of the Purkinje image. Then, the calibration unit 29 updates the reference table based on the relative position and the line-of-sight direction determined by the position of the mark displayed on the display screen 2a (step S109).

The distance feature amount extraction unit 26 extracts a distance feature amount at a specified distance from the image (step S110). The distance feature quantity extraction unit 26 notifies the calibration unit 29 of the distance feature quantity. Then, the calibration unit 29 updates the distance reference table so as to associate the designated distance with the distance feature amount (step S111).
After step S111, the control unit 4 ends the calibration process. Note that the Purkinje image detection unit 24 may change the order of the processing in step S105 and the processing in step S106. Further, the Purkinje image detection unit 24 may search for the Purkinje image of the user's pupil and light source in the entire face area. In this case, the process of step S104 is omitted. Furthermore, the control unit 4 may exchange the order of the processes of steps S104 to S109 and the processes of steps S110 and S111.

FIG. 9 shows an operation flowchart of the line-of-sight detection process executed by the control unit 4.
The control unit 4 acquires an image obtained by photographing the user's face with the light source 11 turned on from the camera 12 (step S201). Then, the face detection unit 23 of the control unit 4 extracts a face area where the face is shown on the image (step S202). The face detection unit 23 determines whether the face area has been successfully extracted (step S203). When face area extraction has failed (step S203—No), the user is not facing the mobile terminal 1. Therefore, the control unit 4 ends the line-of-sight detection process. Thereafter, the control unit 4 may execute a calibration process. Or the control part 4 may perform the process after step S201 again. In this case, the camera control unit 22 of the control unit 4 may transmit a control signal to the camera 12 so that the user's face is photographed under an exposure condition different from the exposure condition at the previous photographing.

On the other hand, when the face detection unit 23 has successfully extracted the face area (step S203—Yes), the face detection unit 23 uses the Purkinje image detection unit 24 and the distance feature quantity extraction unit 26 of the control unit 4 to obtain information representing the face area. To pass.
The Purkinje image detection unit 24 detects an eye region within the face region (step S204). Then, the Purkinje image detection unit 24 detects the center of the pupil in the eye region (step S205). Further, the Purkinje image detection unit 24 detects the Purkinje image of the light source 11 in the eye region (step S206). Then, the Purkinje image detection unit 24 determines whether or not the pupil center of gravity and the Purkinje image of the light source have been successfully detected (step S207).

When the Purkinje image detection unit 24 fails to detect the pupil center of gravity and the Purkinje image of the light source (No in step S207), the control unit 4 executes the processes in and after step S201 again. At that time, the camera control unit 22 may transmit a control signal to the camera 12 so as to photograph the user's face under an exposure condition different from the exposure condition at the previous photographing.
On the other hand, when the Purkinje image detection unit 24 has successfully detected the pupil center of gravity and the Purkinje image of the light source (step S207-Yes), the Purkinje image detection unit 24 uses the line of sight of the Purkinje image of the light source and the pupil center of gravity of the control unit 4. Notify the detection unit 25.
The line-of-sight detection unit 25 refers to the reference table and detects the line-of-sight direction corresponding to the position of the pupil center of gravity based on the center of gravity of the Purkinje image (step S208). The line-of-sight detection unit 25 passes information indicating the line-of-sight direction to the gaze position detection unit 28 of the control unit 4.

  The distance feature amount extraction unit 26 extracts a distance feature amount from the image (step S209). The distance feature amount extraction unit 26 passes the distance feature amount to the distance estimation unit 27 of the control unit 4. Then, the distance estimation unit 27 estimates the subject distance corresponding to the distance feature amount with reference to the distance reference table (step S210). Then, the distance estimation unit 27 passes the estimated value of the subject distance to the gaze position detection unit 28.

The gaze position detection unit 28 refers to the gaze position table and obtains a gaze position corresponding to the estimated values of the gaze direction and the subject distance (step S211). Then, the gaze position detection unit 28 passes information indicating the gaze position to the application program being executed by the control unit 4.
Thereafter, the control unit 4 ends the line-of-sight detection process. Note that the Purkinje image detection unit 24 may reverse the order of the processing in step S205 and the processing in step S206. Further, the Purkinje image detection unit 24 may search for the Purkinje image of the user's pupil and light source in the entire face area. In this case, the process of step S204 is omitted. Alternatively, when the Purkinje image detection unit 24 fails to detect either the pupil center of gravity or the Purkinje image of the light source on the image of interest, the control unit 4 performs the processing after step S201 without performing the other detection processing. May be re-executed.

  As described above, the line-of-sight detection device according to the first embodiment obtains the subject distance when performing the line-of-sight detection process, and determines the gaze position of the user from the distance and the line-of-sight direction obtained based on the Purkinje image. Ask. Therefore, this gaze detection apparatus can accurately detect the gaze position even when the subject distance is different between the execution of calibration and the execution of the gaze detection process. Further, this line-of-sight detection device can extract a feature amount representing the subject distance from the same image as that used for detecting the Purkinje image, and can estimate the subject distance based on the feature amount. For this reason, this gaze detection apparatus does not need to have a mechanism separate from the gaze direction detection mechanism for estimating the subject distance, so that the apparatus configuration can be simplified.

  Next, a gaze detection apparatus according to the second embodiment will be described. The line-of-sight detection apparatus according to the second embodiment obtains the size of the Purkinje image on the image as a distance feature amount.

  The line-of-sight detection apparatus according to the second embodiment differs from the line-of-sight detection apparatus according to the first embodiment in the processing of the distance feature amount extraction unit of the control unit. Therefore, hereinafter, the distance feature quantity extraction unit and its related parts will be described. Also in this embodiment, the line-of-sight detection device is mounted on a portable terminal. For the other parts of the line-of-sight detection device and the portable terminal on which the line-of-sight detection device is mounted, refer to the description of the related parts relating to the first embodiment.

Since the light source 11 is brighter than the surrounding area, the luminance value of the pixel corresponding to the Purkinje image on the image is also higher than the luminance value of the surrounding pixels. Here, if the exposure condition of the camera 12 is appropriately set, the luminance value of the pixel corresponding to the Purkinje image is close to the maximum value of the luminance value that can be taken on the image. That is, the luminance value of the pixel corresponding to the Purkinje image on the image is saturated.
Further, the closer the subject distance, the larger the size of the Purkinje image on the image. For this reason, the closer the subject distance is, the larger the number of pixels having the maximum luminance value corresponding to the size of the Purkinje image.

FIG. 10A shows an example of an image in which a Purkinje image is shown. FIG. 10B shows a pixel with a saturated luminance value among the pixels along the horizontal line 1000 in FIG. Hereinafter, it is a graph showing the experimental result of the relationship between the number of objects and the subject distance for convenience). In FIG. 10B, the horizontal axis represents the subject distance, and the vertical axis represents the luminance value. Points 1001 to 1004 represent the saturated pixel numbers at subject distances of 200 mm, 300 mm, 400 mm, and 500 mm, respectively. In this experiment, we used an image of the eye taken by a camera with a 1 / 3.2 inch (horizontal direction 4.73mm, vertical direction 3.52mm) 2 million pixel CMOS sensor and an imaging optical system with a diagonal field angle of 58 °. It was. And the luminance value of each pixel took the value of 0-255, and the pixel with the luminance value 240 or more was made into the saturation pixel.
As indicated by points 1001 to 1004, it can be seen that the number of saturated pixels decreases as the subject distance increases. The number of saturated pixels when the subject distance is minimum (200 mm) is five times that of the saturated pixel number when the subject distance is maximum (500 mm). Thus, the number of saturated pixels corresponding to the size of the Purkinje image on the image can be used as the distance feature amount.

Therefore, the distance feature amount extraction unit 26 according to this embodiment acquires information representing the area of the Purkinje image on the image from the Purkinje image detection unit 24. Then, the distance feature quantity extraction unit 26 obtains the center of gravity of the Purkinje image region, and sets a scanning line that is a straight line having a width of one pixel that passes through the center of gravity. Note that the direction of the scanning line is arbitrary, and is set in the horizontal direction or the vertical direction, for example. The distance feature amount extraction unit 26 counts the number of saturated pixels on the scanning line and in the Purkinje image region. For example, the distance feature amount calculation unit 26 sets a pixel whose luminance value is equal to or greater than a threshold as a saturated pixel. The threshold value is set to the maximum value of the luminance value or a value slightly smaller than the maximum value, for example, any one of 240 to 255.
The distance feature amount extraction unit 26 passes the number of saturated pixels as the distance feature amount to the distance estimation unit 27 when the line-of-sight detection process is executed. On the other hand, the distance feature amount extraction unit 26 notifies the calibration unit 29 of the number of saturated pixels as the distance feature amount when the calibration process is executed.
According to the modification, the distance feature amount extraction unit 26 may calculate the total number of saturated pixels included in the Purkinje image region as the distance feature amount.

The distance estimation unit 27 refers to the distance reference table stored in the memory 3 to obtain a subject distance corresponding to the number of saturated pixels that is the distance feature amount, and notifies the gaze position detection unit 28 of the subject distance. The gaze position detection unit 28 obtains a gaze position on the display screen based on the subject distance and the gaze direction obtained by the gaze detection unit 25. Thus, in this embodiment, the distance reference table represents the correspondence between the number of saturated pixels and the subject distance.
Further, the calibration unit 29 receives the number of saturated pixels at various designated distances from the distance feature amount extraction unit 26. Then, the calibration unit 29 updates the distance reference table so that each designated distance is associated with the number of saturated pixels at the designated distance.

  As described above, the line-of-sight detection device according to the second embodiment has a saturated pixel number that has a relatively large variation range due to a change in the subject distance within the Purkinje image region on the image that can be detected with high accuracy. As a distance feature quantity. Therefore, the visual line detection device according to the second embodiment can accurately estimate the subject distance. Therefore, the visual axis detection device according to the second embodiment can accurately determine the gaze position.

  Next, a gaze detection apparatus according to the third embodiment will be described. The line-of-sight detection device according to the third embodiment has at least two light sources arranged at a predetermined distance, and obtains the distance between Purkinje images corresponding to each light source on the image as a distance feature amount.

  The line-of-sight detection apparatus according to the third embodiment differs from the line-of-sight detection apparatus according to the first embodiment in that the number of light sources is plural and the processing of the distance feature quantity extraction unit of the control unit is different. Therefore, in the following, the light source, the distance feature amount extraction unit, and the related portions will be described. Also in this embodiment, the line-of-sight detection device is mounted on a portable terminal. For the other parts of the line-of-sight detection device and the portable terminal on which the line-of-sight detection device is mounted, refer to the description of the related parts relating to the first embodiment.

  FIG. 11 is a schematic front view of the mobile terminal 1 on which the line-of-sight detection device according to the third embodiment is mounted. As shown in FIG. 11, in this embodiment, the first light source 11-1 is provided near the upper end of the display screen 2a of the user interface 2, and the second light source 11-2 is provided near the lower end of the display screen 2a. . The arrangement of the light sources is not limited to this example. For example, one light source may be arranged near the left end of the display screen 2a and the other one light source may be arranged near the right end of the display screen 2a. Alternatively, three or more light sources may be arranged around the display screen 2a. However, it is preferable that the light sources are arranged so that the distances between the light sources are as far as possible so that the distance between Purkinje images on the image is increased.

  Since the human cornea is formed in a substantially spherical shape, it functions as a convex mirror. Therefore, as the subject distance increases, the distance between Purkinje images corresponding to each of the plurality of light sources on the image decreases.

  FIG. 12 is a graph showing the experimental results of the relationship between the distance between the Purkinje images of two light sources arranged side by side in the vertical direction across the display screen 2a on the image and the subject distance. In FIG. 12, the horizontal axis represents the subject distance, and the vertical axis represents the interval (pixel unit) between Purkinje images. Each point represents an interval between Purkinje images at a subject distance of 200 mm to 650 mm. In this experiment, light sources were placed near the top and bottom edges of a display screen with a length of 77 mm in the vertical direction. Then, an image obtained by photographing an eye with a camera having a 1 / 3.2 inch (horizontal direction 4.73 mm, vertical direction 3.52 mm) 2 million pixel CMOS sensor and an imaging optical system having a diagonal field angle of 58 ° was used. As indicated by each point, the interval between Purkinje images decreases as the subject distance increases. In addition, the interval between Purkinje images when the subject distance is maximum (650 mm) indicated by the point 1201 and the interval between Purkinje images when the subject distance is minimum (200 mm) shown by the point 1202 is 15 More than double. Thus, it can be seen that the interval between Purkinje images varies greatly depending on the subject distance. Therefore, the distance between Purkinje images on the image can be used as a distance feature amount.

Therefore, the distance feature quantity extraction unit 26 according to this embodiment acquires information representing the center of gravity of the Purkinje image of each light source on the image from the Purkinje image detection unit 24. Then, the distance feature amount extraction unit 26 obtains the distance between the centers of gravity of the Purkinje images.
The distance feature amount extraction unit 26 passes the distance between the centers of gravity of the Purkinje images to the distance estimation unit 27 as the distance feature amount when the line-of-sight detection process is executed.

The distance estimation unit 27 refers to the distance reference table stored in the memory 3 to obtain a subject distance corresponding to the distance between the centroids of the Purkinje image, which is a distance feature quantity, and sends the subject distance to the gaze position detection unit 28. Notice. In addition, the Purkinje image detection unit 24 notifies the line-of-sight detection unit 25 of the center of gravity of the Purkinje image of any one of the light sources, for example, the light source 11-1 disposed on the upper side. The line-of-sight detection unit 25 obtains the user's line-of-sight direction based on the position of the center of gravity of the Purkinje image of the light source 11-1.
The gaze position detection unit 28 obtains a gaze position on the display screen based on the subject distance and the gaze direction obtained by the gaze detection unit 25. Thus, in this embodiment, the distance reference table represents the correspondence between the distance between the centers of gravity of the Purkinje images and the subject distance.

In this embodiment, the calibration unit 29 does not have to update the distance reference table in the calibration process. The interval between Purkinje images on the image is determined by the distance between the light sources, the angle of view of the camera, the number of pixels of the image sensor, and the radius of curvature of the cornea in addition to the subject distance. Other than the subject distance and the radius of curvature of the cornea are known, and for an adult, it can be considered that the radius of curvature of the cornea is substantially constant. Therefore, the correspondence between the distance between Purkinje images and the subject distance is obtained in advance based on a general corneal model.
Note that the calibration unit 29 may update the distance reference table as in the other embodiments. In this case, the calibration unit 29 receives distances between Purkinje images on the image at various designated distances from the distance feature amount extraction unit 26. And the calibration part 29 updates a distance reference table so that each designated distance and the distance between Purkinje images in the designated distance may be matched.

  As described above, the line-of-sight detection device according to the third embodiment includes a plurality of light sources that have a relatively large fluctuation range due to a change in the subject distance within the Purkinje image region on the image that can be detected with high accuracy. The distance between Purkinje images corresponding to each of these is obtained as a distance feature amount. Therefore, this line-of-sight detection device can accurately estimate the subject distance. Therefore, this line-of-sight detection device can accurately determine the gaze position. Furthermore, since this line-of-sight detection apparatus does not need to update the distance reference table, it is possible to reduce the user's trouble at the time of executing the calibration process.

  According to the modification, instead of obtaining the gaze position itself, the gaze detection device calculates the movement amount of the gaze position based on the amount of change in the gaze direction obtained from each of the plurality of images acquired at different times and the subject distance. You may ask for it. In this case, each time the gaze detection unit of the control unit obtains the gaze direction from the image, the acquisition time and the gaze direction of the image are stored in the memory. The memory stores the line-of-sight direction and the image acquisition time for a certain period (for example, 10 seconds). The gaze position detection unit of the control unit calculates the angle difference between the gaze direction obtained from the image acquired at the current time and the gaze direction obtained from the image acquired one to several frames before the image. As the amount of change. The gaze position detection unit obtains the amount of movement of the gaze position by multiplying the tangent of the angle difference in the line-of-sight direction by the subject distance obtained for the current image. The gaze position detection unit notifies the movement amount of the gaze position to the application program being executed by the control unit.

The control unit scrolls the screen according to the amount of movement of the gaze position, for example, according to the application program. Even if the amount of change in the line of sight from the acquisition of the image one to several frames before the acquisition of the current image is the same, if the subject distance is different, the amount of movement of the gaze position is also different, so the control unit scrolls the screen. It is also preferred to change the amount or scroll speed. For example, even if the amount of change in the line-of-sight direction is the same, if the subject distance is doubled, the amount of movement of the gaze position is also doubled, so the control unit also doubles the scroll amount or scroll speed.
In this way, this line-of-sight detection device can cause the control unit of the mobile terminal to perform screen control according to the amount of movement of the gaze position regardless of the subject distance.

  Next, a gaze detection apparatus according to the fourth embodiment will be described. The line-of-sight detection device according to the fourth embodiment includes at least two light sources that are spaced apart in the vertical direction, and is perpendicular to the line-of-sight detection device depending on the number of saturated pixels of the Purkinje image corresponding to each light source on the image. The inclination of the face along is calculated.

  The line-of-sight detection device according to the fourth embodiment differs from the line-of-sight detection device according to the third embodiment in that the control unit detects the inclination of the face with respect to the line-of-sight detection device. Therefore, hereinafter, detection of the tilt of the face of the control unit and related parts will be described. Also in this embodiment, it is assumed that the line-of-sight detection device is mounted on a portable terminal, and one light source is disposed near the upper end and the lower end of the display screen as shown in FIG. For the other parts of the line-of-sight detection device and the portable terminal on which the line-of-sight detection device is mounted, refer to the description of the related parts in the above embodiments.

  FIG. 13 is a functional block diagram of a control unit of the visual line detection device according to the fourth embodiment. The control unit 4 includes a light source control unit 21, a camera control unit 22, a face detection unit 23, a Purkinje image detection unit 24, a line-of-sight detection unit 25, a distance feature quantity extraction unit 26, and a distance estimation unit 27. A gaze position detection unit 28, a calibration unit 29, and an inclination detection unit 30 are included.

  In the present embodiment, the distance feature amount extraction unit 26 extracts the distance between a plurality of parts of the face on the image as the distance feature amount as in the first embodiment. Or the distance feature-value extraction part 26 may extract the distance between the Purkinje images corresponding to each of the some light source on an image as a distance feature-value like 3rd Embodiment.

  The inclination detection unit 30 detects the relative inclination of the user's face with respect to the mobile terminal 1. For this purpose, the inclination detection unit 30 receives information representing the area of the Purkinje image of each light source on the image from the Purkinje image detection unit 24. And the inclination detection part 30 counts the saturation pixel number in the area | region of a Purkinje image for every light source. The method for counting the number of saturated pixels can be the same as the method for counting the number of saturated pixels by the distance feature amount extraction unit according to the second embodiment.

  As described with respect to the second embodiment, the saturation pixel number of the Purkinje image decreases as the subject distance increases. Therefore, if the number of saturated pixels in the Purkinje image of the upper light source 11-1 is smaller than the number of saturated pixels in the Purkinje image of the lower light source 11-2, the distance from the upper light source 11-1 to the user's eye is small. The distance from the lower light source 11-2 to the user's eyes is larger. That is, the user's face is inclined with respect to the mobile terminal 1 so that the upper end of the display screen 2a is farther from the user's face than the lower end of the display screen 2a of the mobile terminal 1. This is called for convenience that the user's face is tilted upward.

Conversely, if the number of saturated pixels in the Purkinje image of the upper light source 11-1 is larger than the number of saturated pixels in the Purkinje image of the lower light source 11-2, the distance from the upper light source 11-1 to the user's eye. Is smaller than the distance from the lower light source 11-2 to the user's eyes. That is, the user's face is inclined with respect to the mobile terminal 1 so that the lower end of the display screen 2a is farther from the user's face than the upper end of the display screen 2a of the mobile terminal 1. This is called for convenience that the user's face is tilted downward.
If the number of saturated pixels in the Purkinje image of the upper light source 11-1 is substantially equal to the number of saturated pixels in the Purkinje image of the lower light source 11-2, the display screen 2a and the user's face are substantially parallel.

FIG. 14 is an operation flowchart of the inclination detection processing by the inclination detection unit 30. The control unit 4 executes the tilt detection process by the tilt detection unit 30 separately from the line-of-sight detection process. Or the control part 4 may perform an inclination detection process following step S211 of the gaze detection process shown in FIG.
The inclination detection unit 30 calculates the number of saturated pixels of the Purkinje image of the upper light source 11-1 and the number of saturated pixels of the Purkinje image of the lower light source 11-2 (step S301). Then, the inclination detector 30 compares the difference between the saturation pixel number cu of the Purkinje image of the upper light source 11-1 and the saturation pixel number cl of the Purkinje image of the lower light source 11-2 with a predetermined number Th (step S302). . If the saturation pixel number cu of the Purkinje image of the upper light source 11-1 is larger than the saturation pixel number cl of the Purkinje image of the lower light source 11-2 by a predetermined number Th or more, the inclination detecting unit 30 has the user's face. It determines with inclining downward (step S303). Then, the tilt detection unit 30 notifies the application program being executed by the control unit 4 that the user's face is tilted downward. On the other hand, if the saturation pixel number cl of the Purkinje image of the lower light source 11-2 is greater than the saturation pixel number cu of the Purkinje image of the upper light source 11-1, the inclination detection unit 30 determines that the face of the user Is tilted upward (step S304). Then, the tilt detection unit 30 notifies the application program being executed by the control unit 4 that the user's face is tilted upward. If the absolute value | cu-cl | of the difference between the saturation pixel numbers of the Purkinje images of the two light sources is less than the predetermined value Th, the inclination detection unit 30 causes the user's face to be substantially parallel to the display screen of the mobile terminal 1. (Step S305). Then, the tilt detection unit 30 notifies the application program being executed by the control unit 4 that the user's face is substantially parallel to the display screen of the mobile terminal 1. After steps S303 to S305, the inclination detection unit 30 ends the inclination detection process. The predetermined number Th is obtained, for example, by experiment or simulation, and is set to “1”, for example.

  In this case, the control unit 4 scrolls the screen according to the movement amount of the user's gaze position according to the application program, for example, as in the above modification. At that time, the control unit 4 controls the user so that the scroll amount or the scroll speed is constant when the line of sight moves toward the upper side of the display screen 2a and when the line of sight moves toward the lower side. It is preferable to adjust the scroll amount or the scroll speed according to the face orientation. For example, when the user's face is facing upward, the control unit 4 decreases the screen scroll amount when the user moves the gaze position upward from the approximate center of the display screen 2a from the approximate center of the display screen 2a. Make it larger than the scroll amount of the screen when moved. When the user's face is substantially parallel to the display screen 2a, the control unit 4 moves the gaze position upward from the approximate center of the display screen 2a and downward from the approximate center of the display screen 2a. The scroll amount or scroll speed of the screen is made equal when moved.

  According to the modification, when the mobile terminal is equipped with a sensor that can detect angular velocity such as a gyro sensor, the control unit acquires information representing the angular velocity of the mobile terminal from the sensor. Then, the tilt detection unit may obtain the tilt of the user's face with respect to the display screen of the mobile terminal based on the angular velocity. For example, when it is determined that the display screen of the mobile terminal and the user's face are substantially parallel, the control unit falls from the gyro sensor to the rear side of the upper end of the mobile terminal, and the lower end of the mobile terminal is the front side, that is, It is assumed that the user is notified that the angular velocity in the direction of falling is detected. In this case, the inclination detection unit determines that the user's face direction is upward.

  Further, the tilt detection unit may determine the tilt of the user's face with respect to the display screen of the mobile terminal in the horizontal direction. In this case, two light sources are arranged at different positions in the horizontal direction on the front surface of the mobile terminal. For example, two light sources are arranged close to the right end and the left end of the display screen. Then, the inclination detector obtains the number of saturated pixels of the Purkinje image of each light source on the image, and compares the number of saturated pixels for each light source. If the number of saturated pixels of the right light source is larger than the number of saturated pixels of the left light source, the inclination detection unit causes the user's face to move away from the display screen of the mobile terminal toward the left side in the horizontal direction. It is judged that it is leaning to. Conversely, if the number of saturated pixels of the light source on the left side is larger than the number of saturated pixels on the right side of the light source, the tilt detection unit displays the display screen of the mobile terminal toward the right side of the user's face along the horizontal direction. It is determined that it is tilted away from it.

According to still another modification, a light source may be arranged in the vicinity of each of the four corners of the display screen. In this case, in order to detect the inclination of the user's face along the vertical direction, the inclination detection unit, on the image, the distance between the two Purkinje images corresponding to the upper two light sources, and the lower two The distance between two Purkinje images corresponding to the light source is obtained. If the distance between the two Purkinje images corresponding to the upper two light sources is larger than the distance between the two Purkinje images corresponding to the lower two light sources, the inclination detecting unit faces the user downward. Is determined. Conversely, if the distance between the two Purkinje images corresponding to the two lower light sources is larger than the distance between the two Purkinje images corresponding to the two upper light sources, the tilt detection unit will face the user upward. Judge that there is.
Similarly, in order to detect the inclination of the user's face along the horizontal direction, the inclination detector detects the distance between the two Purkinje images corresponding to the two light sources on the left side and the right side on the image. And the distance between two Purkinje images corresponding to the two light sources. For example, if the distance between the two Purkinje images corresponding to the two light sources on the right side is larger than the distance between the two Purkinje images corresponding to the two light sources on the left side, the inclination detecting unit Determines that the left side is tilted away from the display screen of the mobile terminal.

  The line-of-sight detection device according to each of the above-described embodiments or modifications can be incorporated into various devices that use the user's gaze position on the display unit. For example, the line-of-sight detection device may be incorporated in a car navigation system or a computer. In this case, the light source and camera included in the line-of-sight detection device and the display unit may be arranged separately.

  All examples and specific terms listed herein are intended for instructional purposes to help the reader understand the concepts contributed by the inventor to the present invention and the promotion of the technology. It should be construed that it is not limited to the construction of any example herein, such specific examples and conditions, with respect to showing the superiority and inferiority of the present invention. Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

The following supplementary notes are further disclosed regarding the embodiment described above and its modifications.
(Appendix 1)
A light source that illuminates the user's eyes;
An imaging unit that generates an image in which at least a part of the user's face is captured in an area including the eyes while the light source is lit;
A storage unit for storing distance information representing a relationship between a distance from the display unit to the face and a distance feature amount on the image that changes according to the distance;
A distance feature amount extraction unit for obtaining the distance feature amount from the image;
A distance estimation unit that estimates a distance from the display unit corresponding to the distance feature amount to the face by referring to the distance information;
Purkinje image detection unit for detecting a cornea reflection image of the light source and a pupil center of gravity of the user from a region including the eye included in the image,
A line-of-sight detection unit that detects the direction of the line of sight of the user according to the positional relationship between the pupil center of gravity and the corneal reflection image;
A gaze position detection unit for obtaining a gaze position of the user on the display unit based on the line-of-sight direction and the distance from the display unit to the face;
A line-of-sight detection apparatus comprising:
(Appendix 2)
The line-of-sight detection apparatus according to appendix 1, wherein the distance feature quantity extraction unit obtains positions of at least two parts of the face on the image and obtains an interval between the at least two parts as the distance feature quantity.
(Appendix 3)
The line-of-sight detection device according to appendix 1, wherein the distance feature quantity extraction unit obtains the size of the corneal reflection image on the image as the distance feature quantity.
(Appendix 4)
The line-of-sight detection device according to appendix 3, wherein the distance feature quantity extraction unit uses the number of pixels with saturated luminance values included in a region corresponding to the corneal reflection image on the image as the size of the corneal reflection image. .
(Appendix 5)
The light source includes a first light emitting element and a second light emitting element arranged at different positions,
The Purkinje image detection unit detects, as the cornea reflection image, a first cornea reflection image of the first light emitting element and a second cornea reflection image of the second light emitting element on the image,
The line-of-sight detection device according to appendix 1, wherein the distance feature quantity extraction unit obtains an interval between the first corneal reflection image and the second corneal reflection image on the image as the distance feature quantity.
(Appendix 6)
The first light emitting element and the second light emitting element are disposed along a predetermined direction,
The size of the first corneal reflection image and the size of the second corneal reflection image are obtained, and when the size of the second corneal reflection image is smaller than the size of the first corneal reflection image, An inclination detecting unit that determines that the face is inclined with respect to the display unit in a direction in which a distance between the light emitting element and the face is smaller than a distance between the second light emitting element and the face; The line-of-sight detection device according to appendix 5.
(Appendix 7)
The light source has at least four light emitting elements arranged at different positions along the outer periphery of the display unit, and the first light emitting element and the second light emitting element among the light emitting elements are the display The third light emitting element and the fourth light emitting element of the light emitting elements are disposed below the vertical center of the display unit,
The Purkinje image detection unit detects the cornea reflection images of the first to fourth light emitting elements on the image as the cornea reflection image,
A first interval between the cornea reflection image of the first light emitting element and the cornea reflection image of the second light emitting element on the image, and a cornea reflection image of the third light emitting element and the fourth light emitting element. A direction in which the lower end of the display unit is closer to the user's face than the upper end of the display unit when the second interval between the cornea reflection images is obtained and the first interval is smaller than the second interval. The line-of-sight detection device according to appendix 1, further comprising an inclination detection unit that determines that the face is inclined with respect to the display unit.
(Appendix 8)
A display unit;
A light source that illuminates the user's eyes;
An imaging unit that generates an image in which at least a part of the user's face is captured in an area including the eyes while the light source is lit;
A storage unit that stores distance information representing a relationship between a distance from the display unit to the face and a distance feature amount on the image that changes according to the distance;
A distance feature amount extraction unit for obtaining the distance feature amount from the image;
A distance estimation unit that estimates a distance from the display unit corresponding to the distance feature amount to the face by referring to the distance information;
Purkinje image detection unit for detecting a cornea reflection image of the light source and a pupil center of gravity of the user from a region including the eye included in the image,
A line-of-sight detection unit that detects the direction of the line of sight of the user according to the positional relationship between the pupil center of gravity and the corneal reflection image;
A gaze position detection unit for obtaining a gaze position of the user on the display unit based on the line-of-sight direction and the distance from the display unit to the face;
An application execution unit that executes processing associated with the gaze position;
A mobile terminal.
(Appendix 9)
While the light source that illuminates the user's eyes is turned on, an image including at least a part of the user's face that is an area including the eyes is generated;
From the image, a distance feature amount on the image that changes according to the distance from the display unit to the face is obtained,
Estimating a distance from the display unit corresponding to the distance feature amount to the face by referring to distance information representing a relationship between a distance from the display unit to the face and the distance feature amount;
From a region including the eye included in the image, a cornea reflection image of the light source and a pupil center of gravity of the user are detected,
Detecting the user's line-of-sight direction according to the positional relationship between the pupil center of gravity and the corneal reflection image;
Obtaining the gaze position of the user on the display unit based on the line-of-sight direction and the distance from the display unit to the face;
A gaze detection method including the above.
(Appendix 10)
While the light source that illuminates the user's eyes is turned on, an image obtained by capturing an image of at least a part of the user's face, which is an area including the eyes, is acquired from the imaging unit.
From the image, a distance feature amount on the image that changes according to the distance from the display unit to the face is obtained,
Estimating a distance from the display unit corresponding to the distance feature amount to the face by referring to distance information representing a relationship between a distance from the display unit to the face and the distance feature amount;
From a region including the eye included in the image, a cornea reflection image of the light source and a pupil center of gravity of the user are detected,
Detecting the user's line-of-sight direction according to the positional relationship between the pupil center of gravity and the corneal reflection image;
Obtaining the gaze position of the user on the display unit based on the line-of-sight direction and the distance from the display unit to the face;
A computer program for eye-gaze detection that causes a computer to execute the above.

DESCRIPTION OF SYMBOLS 1 Mobile terminal 2 User interface 2a Display screen 3 Memory 4 Control part 5 Chassis 10 Eye-gaze detection apparatus 11, 11-1 to 11-2 Light source 12 Camera 21 Light source control part 22 Camera control part 23 Face detection part 24 Purkinje image detection part 25 Gaze detection unit 26 Distance feature amount extraction unit 27 Distance estimation unit 28 Gaze position detection unit 29 Calibration unit 30 Tilt detection unit

Claims (6)

A light source that illuminates the user's eyes;
An imaging unit that generates an image in which at least a part of the user's face is captured in an area including the eyes while the light source is lit;
A storage unit for storing distance information representing a relationship between a distance from the display unit to the face and a distance feature amount on the image that changes according to the distance;
A distance feature amount extraction unit for obtaining the distance feature amount from the image;
A distance estimation unit that estimates a distance from the display unit corresponding to the distance feature amount to the face by referring to the distance information;
Purkinje image detection unit for detecting a cornea reflection image of the light source and a pupil center of gravity of the user from a region including the eye included in the image,
A line-of-sight detection unit that detects the direction of the line of sight of the user according to the positional relationship between the pupil center of gravity and the corneal reflection image;
A gaze position detection unit for obtaining a gaze position of the user on the display unit based on the line-of-sight direction and the distance from the display unit to the face;
A line-of-sight detection apparatus comprising:   The line-of-sight detection device according to claim 1, wherein the distance feature quantity extraction unit obtains positions of at least two parts of the face on the image and obtains an interval between the at least two parts as the distance feature quantity.   The line-of-sight detection device according to claim 1, wherein the distance feature amount extraction unit obtains the size of the corneal reflection image on the image as the distance feature amount. The light source includes a first light emitting element and a second light emitting element arranged at different positions,
The Purkinje image detection unit detects, as the cornea reflection image, a first cornea reflection image of the first light emitting element and a second cornea reflection image of the second light emitting element on the image,
The line-of-sight detection device according to claim 1, wherein the distance feature quantity extraction unit obtains an interval between the first corneal reflection image and the second corneal reflection image on the image as the distance feature quantity. The first light emitting element and the second light emitting element are disposed along a predetermined direction,
The size of the first corneal reflection image and the size of the second corneal reflection image are obtained, and when the size of the second corneal reflection image is smaller than the size of the first corneal reflection image, An inclination detecting unit that determines that the face is inclined with respect to the display unit in a direction in which a distance between the light emitting element and the face is smaller than a distance between the second light emitting element and the face; The line-of-sight detection device according to claim 4. While the light source that illuminates the user's eyes is turned on, an image including at least a part of the user's face that is an area including the eyes is generated;
From the image, a distance feature amount on the image that changes according to the distance from the display unit to the face is obtained,
Estimating a distance from the display unit corresponding to the distance feature amount to the face by referring to distance information representing a relationship between a distance from the display unit to the face and the distance feature amount;
From a region including the eye included in the image, a cornea reflection image of the light source and a pupil center of gravity of the user are detected,
Detecting the user's line-of-sight direction according to the positional relationship between the pupil center of gravity and the corneal reflection image;
Obtaining the gaze position of the user on the display unit based on the line-of-sight direction and the distance from the display unit to the face;
A gaze detection method including the above.

Images