JP2018197974A - Line-of-sight detection computer program, line-of-sight detection device and line-of-sight detection method - Google Patents

Abstract

To provide a line-of-site detection computer program capable of improving accuracy of detection of a line-of-site direction.SOLUTION: A line-of-sight detection computer program comprises instructions to cause a computer: to detect, from an image, a pupil region in which a pupil of an eye is photographed and a corneal reflection image of an illuminating light source which is lit among the plurality of illumination light sources, wherein the image is generated by an imaging unit photographing a user's eye; to detect a user's line-of-sight direction based on the pupil region and the cornea reflection image; to predict the user's line-of-sight direction for the next time the imaging unit photographs based on the user's line-of-sight direction detected from each of the at least two images which are respectively generated at a different timing; and to lit at least one illumination light source corresponding to a detectable range including the predicted user line-of-sight direction among the plurality of illumination light sources, the next time the imaging unit photographs.SELECTED DRAWING: Figure 10

Description

  The present invention relates to a computer program for eye gaze detection, a gaze detection device, and a gaze detection method for detecting a gaze direction of a person based on, for example, an image obtained by photographing a human eye.

  By tracking a person's line of sight or the position where the person is gazing, you can get information about what the person is interested in, or automatically perform operations related to the position you are gazing at Is being considered to do. Hereinafter, for the sake of convenience, a position where a person is gazing is simply referred to as a gazing position.

  A person's gaze position or line-of-sight direction is closely related to the direction of the person's pupil. Therefore, a technique is known in which a pupil is detected from an image obtained by photographing a human eye, and the gaze direction or gaze position of the person is specified based on the positional relationship between the pupil and a reference point. As the reference point, for example, a cornea reflection image of a light source (hereinafter referred to as a Purkinje image) is used. Since the position of the Purkinje image is almost constant regardless of the orientation of the human face and the orientation of the pupil, using the Purkinje image as a reference point makes it possible to detect the line-of-sight direction with relatively high accuracy. .

  However, in order for the Purkinje image to appear on the image, it is required that the position where the light from the light source is reflected is located on the cornea as viewed from the camera that captures the human eye. Therefore, in order for the Purkinje image to appear on the image, there are restrictions on the relative positional relationship between the light source, the camera, and the eye, and the line-of-sight direction. Therefore, a technique for improving the detection accuracy of the line-of-sight direction by controlling a light source to be lit among a plurality of light sources has been proposed (see, for example, Patent Documents 1 to 3).

JP-A-6-138367 JP 2005-296383 A JP 2014-183392 A

  In the techniques disclosed in Patent Documents 1 to 3, for example, according to the evaluation of the quality of the detection accuracy of the gaze direction when sequentially turning on the face direction, the light source, or the positional deviation between the pupil and the Purkinje image, The light source to be lit is controlled.

  However, since the face direction and the line-of-sight direction do not necessarily match, controlling a light source that is turned on based on the face direction does not necessarily turn on an appropriate light source that allows the Purkinje image to be observed. In addition, when the light sources are sequentially turned on, the time required until an appropriate light source is selected is increased, and as a result, the time required until the gaze direction can be detected is increased. And if the line-of-sight direction moves while the light sources are turned on sequentially, the optimal light source is turned on for the line-of-sight direction before the movement, and the appropriate light source is turned on for the current line-of-sight direction Not exclusively. In addition, even when the light source to be turned on is controlled according to the positional deviation between the pupil and the Purkinje image, when the line-of-sight direction moves, the optimal light source is turned on with respect to the line-of-sight direction before the movement, and the current line-of-sight direction is On the other hand, an appropriate light source is not always turned on. As a result, with the above technique, the Purkinje image may not appear on the image, and the detection accuracy in the line-of-sight direction may be insufficient.

  In one aspect, an object of the present invention is to provide a computer program for eye gaze detection that can improve the accuracy of gaze direction detection.

  According to one embodiment, a computer program for eye gaze detection is provided. The computer program for eye-gaze detection includes a pupil region in which an eye pupil is captured from an image generated by an imaging unit photographing a user's eye, and a cornea of a lighting illumination light source among a plurality of illumination light sources An image pickup unit that detects a reflection image, detects a user's line-of-sight direction based on the pupil region and the cornea reflection image, and based on the user's line-of-sight direction detected from each of at least two images generated at different timings Predicting the user's line-of-sight direction at the next shooting, and for each of the plurality of illumination light sources, a detectable range of the line-of-sight direction is defined according to the positional relationship between the illumination light source and the imaging unit. And causing the computer to turn on at least one of the illumination light sources corresponding to the detectable range including the predicted user's line-of-sight direction at the next shooting of the imaging unit. Including instructions for causing a row.

  The detection accuracy of the gaze direction can be improved.

It is a hardware block diagram of the digital signage system which is one Embodiment of a visual line detection apparatus. It is a figure which shows an example of arrangement | positioning of a some illumination light source and an infrared camera. It is a functional block diagram regarding the gaze detection process of a processor. It is a figure which shows an example of the relationship between a lighting light source and an infrared camera, and a user's eyes | visual_axis direction in case the one Purkinje image detected on an image is one. It is a figure which shows an example of a gaze position table. It is a figure which shows an example of a lighting model. It is a figure explaining the principle of the process which selects the illumination light source which turns on until the prediction result of a gaze direction becomes available. It is a figure explaining the principle of the process which selects the illumination light source to light after the prediction result of a gaze direction becomes available. It is a figure which shows the operation | movement flowchart of a gaze detection process until the prediction result of a gaze direction becomes usable for lighting control of an illumination light source. It is a figure which shows the operation | movement flowchart of a gaze detection process after the prediction result of a gaze direction becomes available for lighting control of an illumination light source.

Hereinafter, the line-of-sight detection device will be described with reference to the drawings.
This line-of-sight detection device generates an image by photographing a user's eye illuminated by any of a plurality of illumination light sources with a camera, and detects and detects an eye pupil and Purkinje image on the image. Based on the pupil and Purkinje image, the user's line-of-sight direction is detected. This line-of-sight detection device predicts the line-of-sight direction at the time of the next shooting based on the line-of-sight direction detected from the images obtained so far. The line-of-sight detection device is lit from a plurality of illumination light sources so that the predicted line-of-sight direction is included in the range of the line-of-sight direction in which the line-of-sight direction can be detected, that is, the range of the line-of-sight direction in which the Purkinje image can be observed. Select the illumination source to be activated.

  In the embodiment described below, the line-of-sight detection apparatus is mounted on a digital signage system, and the line-of-sight detection apparatus tracks the line-of-sight direction and gaze position of a person who is a user of the digital signage system. However, this line-of-sight detection device can be used in various devices that track the direction of the line of sight and use the tracking result.

  FIG. 1 is a hardware configuration diagram of a digital signage system which is an embodiment of a visual line detection device. The digital signage system 1 includes a display device 2, a plurality of illumination light sources 3-1 to 3-n (where n is an integer of 3 or more), an infrared camera 4, an input device 5, and a storage medium access device 6. A memory 7 and a processor 8. Furthermore, the digital signage system 1 may have a communication interface circuit (not shown) for connecting the digital signage system 1 to other devices.

  The display device 2 includes, for example, a liquid crystal display or an organic electroluminescence display. The display device 2 displays, for example, various texts, icons, still images, or moving images according to the video signal from the processor 8.

  Each of the plurality of illumination light sources 3-1 to 3-n is a light source that emits infrared light, for example, at least one infrared light emitting diode and a power supply to the infrared light emitting diode according to a control signal from the processor 8. And a drive circuit (not shown) for supplying power from (not shown). Each of the illumination light sources 3-1 to 3-n has a light emitting surface of each of the illumination light sources 3-1 to 3-n in the imaging direction of the infrared camera 4 so that the imaging direction of the infrared camera 4 can be illuminated. It is attached to face. Each of the illumination light sources 3-1 to 3-n emits illumination light after receiving a control signal for turning on the illumination light source from the processor 8 until receiving a control signal for turning off the illumination light source. .

  In the present embodiment, the number of illumination light sources only needs to be larger than the number of illumination light sources that are simultaneously turned on (2 in the present embodiment). For example, the number of illumination light sources can be six ( That is, n = 6). In addition, each of the illumination light sources 3-1 to 3-n is arranged at a different position so that the range of the user's gaze direction that can be detected is different from each other. In addition, the range of the user's gaze direction that can be detected is hereinafter simply referred to as a detectable range.

  The infrared camera 4 is an example of an imaging unit, and generates an image in which at least one eye of the user is captured. Therefore, the infrared camera 4 includes an image sensor having a solid-state imaging device arranged in a two-dimensional shape having sensitivity to infrared light emitted from the illumination light sources 3-1 to 3-n, and an object on the image sensor. An imaging optical system that forms an image is included. In order to suppress detection of the reflected image by the iris and the Purkinje image of light from a light source other than the illumination light sources 3-1 to 3-n, the infrared camera 4 is between the image sensor and the imaging optical system. You may further have a visible light cut filter. The imaging optical system may be a single focus optical system or a variable focus optical system. The infrared camera 4 shoots at a predetermined frame rate and generates an image during execution of the line-of-sight detection process. In addition, the infrared camera 4 has the resolution which can identify the Purkinje image and pupil of the illumination light sources 3-1 to 3-n reflected on the pupil of the user who uses the digital signage system 1 on this image. The infrared camera 4 passes the image to the processor 8 every time it generates an image. In the following description, “image” represents an image generated by the infrared camera 4 and showing the user's eyes unless otherwise specified.

  In the present embodiment, the illumination light sources 3-1 to 3-n and the infrared camera 4 are attached to the same housing 10. The illumination light sources 3-1 to 3-n and the infrared camera 4 may be provided separately.

FIG. 2 is a diagram illustrating an example of the arrangement of the illumination light sources 3-1 to 3-n and the infrared camera 4. In this example, the number of illumination light sources is six, and a housing 10 in which illumination light sources 3-1 to 3-6 and an infrared camera 4 are attached to a product shelf 11 arranged in the vicinity of the display device 2. It is attached. Then, the housing 10 is directed to the user 200 of the product shelf 11 so that the illumination light sources 3-1 to 3-6 and the infrared camera 4 are directed to the face of the user 200 who views the product placed on the product shelf 11. It is arranged on the surface on the opposite side. In addition, the illumination light sources 3-1 to 3-6 are arranged in a line in order from the left as viewed from the user 200, along the alignment direction of both eyes of the user 200, that is, along the horizontal direction. And the infrared camera 4 is arrange | positioned between the illumination light source 3-3 and the illumination light source 3-4, and image | photographs the user's 200 whole face or a part of face including at least one eye of the user 200. FIG.
The arrangement of the illumination light sources 3-1 to 3-n and the infrared camera 4 is not limited to this example. For example, the illumination light sources 3-1 to 3-n are arranged at different positions in the vertical direction. May be. Furthermore, the vertical position of the illumination light sources 3-1 to 3-n and the vertical position of the infrared camera 4 may be different from each other.

  The input device 5 includes, for example, a keyboard and a pointing device such as a mouse. An operation signal input by the user via the input device 5 is passed to the processor 8.

  The display device 2 and the input device 5 may be integrated like a touch panel display, for example. In this case, when the user touches the position of the icon displayed on the display screen of the display device 2, the input device 5 generates an operation signal corresponding to the position and outputs the operation signal to the processor 8. .

  The storage medium access device 6 is an example of a storage unit, and is a device that accesses a storage medium 9 such as a magnetic disk, a semiconductor memory card, and an optical storage medium. The storage medium access device 6 reads a computer program for line-of-sight detection processing executed on the processor 8 and stored in the storage medium 9, for example, and passes it to the processor 8.

  The memory 7 is another example of the storage unit, and includes, for example, a readable / writable nonvolatile semiconductor memory and a readable / writable volatile semiconductor memory. The memory 7 stores a computer program for line-of-sight detection processing, various application programs, and various data executed on the processor 8.

  Further, the memory 7 stores various data used for detecting the user's line-of-sight direction and gaze position. For example, the memory 7 stores the positions of the illumination light sources 3-1 to 3-n and the infrared camera 4 in real space, the focal length of the infrared camera 4, and the like. The memory 7 may store a lighting model and a detection history of the line-of-sight direction. Details of the lighting model will be described later. Furthermore, when it is assumed that the distance between the user and the infrared camera 4 is a preset distance, the memory 7 may store the preset distance.

  The processor 8 is an example of a control unit, and includes, for example, at least one Central Processing Unit (CPU) and its peripheral circuits. Further, the processor 8 may include a numerical arithmetic processor or a graphics processing unit (GPU). The processor 8 is connected to each part of the digital signage system 1 through a signal line, and controls the entire digital signage system 1. For example, the processor 8 causes the display device 2 to display a predetermined moving image or the like according to the operation signal received from the input device 5 and the application program being executed.

  Further, the processor 8 detects the user's line-of-sight direction and gaze position by executing a line-of-sight detection process each time an image is obtained. Then, the processor 8 executes processing according to the detection result. For example, the processor 8 refers to the position information of the product stored in the memory 7, identifies the product closest to the detected gaze position, reads information about the identified product from the memory 7, and displays it on the display device 2. Display.

FIG. 3 is a functional block diagram relating to the line-of-sight detection processing of the processor 8. The processor 8 includes a line-of-sight detection unit 21, a gaze position estimation unit 22, a prediction unit 23, and a light source control unit 24.
Each of these units included in the processor 8 is a functional module realized by a computer program executed on the processor 8. Further, each of these units included in the processor 8 may be mounted as a part of a circuit in the processor 8 that realizes the function of each unit.

In this embodiment, the processor 8 executes background difference processing between the image and the background image stored in the memory 7 every time an image is obtained from the infrared camera 4 until the line-of-sight detection processing is started. Then, a pixel whose absolute difference value between corresponding pixels is a predetermined value or more is extracted. Then, the processor 8 starts the line-of-sight detection process when the number of extracted pixels exceeds a predetermined number.
The digital signage system 1 may have a proximity sensor (not shown). When the proximity sensor detects an object located in front of the infrared camera 4, the processor 8 may start the line-of-sight detection process.

  In the present embodiment, the processor 8 turns on two of the illumination light sources 3-1 to 3-n while the line-of-sight detection process is being performed. Then, the processor 8 selects an illumination light source to be turned on based on the predicted line-of-sight direction for each predetermined period, for example, for each imaging period of the infrared camera 4.

  Each time an image is obtained during execution of the line-of-sight detection process, the line-of-sight detection unit 21 Purkinje of the illumination light source that is lit among the pupils of the user's eyes and the illumination light sources 3-1 to 3-n on the image. Detect the image. The line-of-sight detection unit 21 detects the user's line-of-sight direction based on the positional relationship between the pupil and the Purkinje image. In the following, for convenience of explanation, an illumination light source that is lit may be simply referred to as a lit light source.

  The line-of-sight detection unit 21 first detects, from the image, an area in which each of the user's left and right eyes is reflected (hereinafter simply referred to as an eye area). For this purpose, the line-of-sight detection unit 21 detects the eye region using, for example, a discriminator that has been learned in advance so as to detect the eye region from the image. In this case, for example, Adaboost or Real Adaboost, a support vector machine, or a deep neural network is used as the discriminator. Then, the line-of-sight detection unit 21 sets a window on the image, changes the position of the window, and inputs the value of each pixel in the window or the feature amount extracted from the window to the discriminator. It is determined whether or not the eye area. Further, for example, a Haar-like feature quantity or a Histograms of Oriented Gradients feature quantity is extracted as the feature quantity.

  Alternatively, the line-of-sight detection unit 21 may detect an area that most closely matches the template by template matching between the template representing the eye and the image, and may use the detected area as the eye area. Alternatively, the line-of-sight detection unit 21 may detect the eye area according to any of various other methods for detecting the eye area shown on the image.

  When the eye area is detected, the line-of-sight detection unit 21 detects the pupil and the Purkinje image of the lighting light source in any one of the eye areas.

  For example, the line-of-sight detection unit 21 performs template matching between the template corresponding to the pupil and the eye region in order to detect the pupil, and detects a region having the highest degree of matching with the template in the eye region. When the highest coincidence value is higher than a predetermined coincidence threshold value, the line-of-sight detection unit 21 determines the detected region as a pupil region in which the pupil is reflected. Note that the size of the pupil changes according to the brightness around the user. Moreover, although the pupil is substantially circular, when the infrared camera 4 is viewing the pupil from an oblique direction, the shape of the pupil on the image is an elliptical shape that is long in the vertical direction. Therefore, a plurality of templates may be prepared according to the size or shape of the pupil. In this case, the line-of-sight detection unit 21 performs template matching between each template and the eye region, and obtains the highest matching degree. If the highest coincidence value is higher than the coincidence degree threshold, the line-of-sight detection unit 21 determines that the region overlapping the template corresponding to the highest coincidence value is a pupil region. Note that the degree of coincidence is calculated as, for example, a normalized cross-correlation value between a template and a region overlapping the template. The coincidence threshold is set to 0.7 or 0.8, for example.

  The luminance of the region where the pupil is reflected is lower than the luminance of the surrounding region, and the pupil is substantially circular. Therefore, the line-of-sight detection unit 21 sets a double ring filter having two rings with different radii concentrically within the eye region. When the difference value obtained by subtracting the average luminance value of the inner pixels from the average luminance value of the pixels corresponding to the outer ring is larger than a predetermined threshold, the line-of-sight detection unit 21 is surrounded by the inner ring. The region may be a pupil region. The line-of-sight detection unit 21 may add to the condition for detecting the pupil region that the average luminance value of the region surrounded by the inner ring is equal to or less than a predetermined threshold value. In this case, for example, the predetermined threshold is set to a value obtained by adding 10% to 20% of the difference between the maximum luminance value and the minimum luminance value in the eye region to the minimum luminance value.

  Note that the line-of-sight detection unit 21 may detect the pupil region using any of various other methods for detecting the region in which the pupil is shown on the image.

  In addition, the luminance of the region where the Purkinje image of the lighting source is reflected is higher than the luminance of the surrounding region, and the luminance value is substantially saturated (that is, the luminance value is a luminance value that can be taken by the pixel value). It is almost the highest value of). In addition, the shape of the region where the Purkinje image of the lighting light source is reflected substantially matches the shape of the light emitting surface of the lighting light source. Therefore, the line-of-sight detection unit 21 has a shape that substantially matches the contour shape of the light emitting surface of the lighting light source in the eye region, and has a different size and a center to detect the Purkinje image of the lighting light source. Set up two rings. The line-of-sight detection unit 21 obtains a difference value obtained by subtracting the average value of the brightness of the outer pixels from the average value of the brightness of the pixels corresponding to the inner ring. When the difference value is larger than the predetermined difference threshold value and the inner luminance average value is higher than the predetermined luminance threshold value, the line-of-sight detection unit 21 sets the area surrounded by the inner ring as the Purkinje image of the lighting light source. . The difference threshold value can be, for example, an average value of difference values between neighboring pixels in the eye area. Further, the predetermined luminance threshold can be set to 80% of the maximum luminance value in the eye area, for example.

  Note that the line-of-sight detection unit 21 may detect an area in which the Purkinje image of the lighting light source is captured using any of various other methods for detecting the area in which the Purkinje image of the light source is captured on the image. Good.

  The line-of-sight detection unit 21 calculates the average value of the horizontal coordinate value and the average value of the vertical coordinate value of each pixel included in the pupil region as the position coordinate of the centroid of the pupil region (hereinafter simply referred to as the pupil centroid). . Similarly, the line-of-sight detection unit 21 calculates the average value of the horizontal coordinate values and the average value of the vertical coordinate values of each pixel included in the region where the Purkinje image is captured as the position coordinate of the center of gravity of the Purkinje image.

  In the present embodiment, the line-of-sight detection unit 21 detects the user's line-of-sight direction according to different methods depending on whether the detected Purkinje image is one or two.

  When there is one detected Purkinje image, the line-of-sight detection unit 21 identifies a lighting light source corresponding to the Purkinje image detected on the image, out of the two lighting light sources.

  FIG. 4 is a diagram illustrating an example of the relationship between the lighting light source and the infrared camera 4 and the user's line-of-sight direction when there is one Purkinje image detected on the image. In this example, it is assumed that the illumination light source 3-2 located on the left side of the infrared camera 4 and the illumination light source 3-5 located on the right side of the infrared camera 4 are lit. If the user faces the infrared camera 4, the user's cornea 401 faces the infrared camera 4. Therefore, when viewed from the infrared camera 4, the position where the light from the illumination light sources 3-2 and 3-5 is reflected is located on the user's cornea 401, and it is assumed that two Purkinje images appear on the image. . Therefore, when only one Purkinje image appears on the image, for example, it is assumed that the user's line of sight is directed away from the infrared camera 4 as indicated by an arrow 402 in FIG. The As indicated by an arrow 403, the light from the illumination light source (illumination light source 3-2 in FIG. 4) that is closer to the user's line-of-sight direction is reflected by the cornea 401. The infrared camera 4 is reached. On the other hand, as indicated by an arrow 404, the light from the illumination light source (illumination light source 3-5 in FIG. 4) far from the user's line of sight is reflected by the surface of the eyeball off the cornea 401 and becomes infrared. Since the camera 4 is reached, the Purkinje image of the illumination light source does not appear on the image. Precisely, since the surface of the eyeball other than the cornea is white, if the light from the illumination light source is reflected on the surface of the eyeball other than the cornea, the Purkinje image and the surrounding luminance difference are small on the image, and the Purkinje image is identified. Difficult to do.

  Therefore, the line-of-sight detection unit 21 determines whether the line-of-sight direction is on the right side or the left side of the infrared camera 4 based on the relative positional relationship of the pupil with respect to the head of the eye and the eye corner. judge. Then, the line-of-sight detection unit 21 determines that the Purkinje image of the illumination light source closer to the line-of-sight direction is displayed on the image among the two illumination light sources that are lit.

  For example, the line-of-sight detection unit 21 performs a corner detection filter process such as a harris filter on one of the two detected eye regions to detect a corner in the eye region. Then, the line-of-sight detection unit 21 sets a corner closer to the other eye region as the head of the two detected corners, and sets the other corner as the corner of the eye. The line-of-sight detection unit 21 determines the line-of-sight direction by calculating the midpoint of the eye and the corner of the eye and comparing the midpoint and the position of the pupil. For example, if the pupil of the two eye regions located on the right side in the image (that is, the user's left eye) is closer to the eye head side than the midpoint of the eye head and the eye corner, the line-of-sight detection unit 21 Determines that the line-of-sight direction is to the left of the infrared camera 4. The line-of-sight detection unit 21 determines that the Purkinje image on the image belongs to the illumination light source located on the left side of the infrared camera 4. On the other hand, if the pupil is closer to the outer corner of the eye than the middle point of the eye and the outer corner of the eye, the line-of-sight detection unit 21 determines that the line-of-sight direction is facing the right side of the infrared camera 4. The line-of-sight detection unit 21 determines that the Purkinje image on the image belongs to the illumination light source located on the right side of the infrared camera 4.

  Of the two eye regions, the eye region located on the left side of the image (that is, the user's right eye), if the pupil is closer to the occipital side than the midpoint of the eye and the corner of the eye, the line-of-sight detection unit 21. Determines that the line-of-sight direction is on the right side of the infrared camera 4. The line-of-sight detection unit 21 determines that the Purkinje image on the image belongs to the illumination light source located on the right side of the infrared camera 4. On the other hand, if the pupil is closer to the outer corner of the eye than the middle point of the eye and the outer corner of the eye, the line-of-sight detection unit 21 determines that the line-of-sight direction is facing the left side of the infrared camera 4. The line-of-sight detection unit 21 determines that the Purkinje image on the image belongs to the illumination light source located on the left side of the infrared camera 4.

  When the lighting light source corresponding to the Purkinje image is identified, the line-of-sight detection unit 21 determines the user based on the positional relationship between the identified lighting light source and the infrared camera 4 and the positional relationship between the Purkinje image and the pupil center of gravity. The direction of the line of sight is detected.

  Since the surface of the cornea is substantially spherical, the position of the Purkinje image of the lighting light source is almost constant regardless of the line-of-sight direction. On the other hand, the center of the pupil moves according to the line-of-sight direction. Therefore, the line-of-sight detection unit 21 can detect the line-of-sight direction by obtaining the relative position of the pupil center of gravity relative to the center of gravity of the Purkinje image.

  For example, the line-of-sight detection unit 21 calculates the relative position of the pupil center of gravity with respect to the center of gravity of the Purkinje image of the lighting light source, for example, the horizontal coordinate of the center of gravity of the Purkinje image from the horizontal coordinate and the vertical coordinate of the pupil center of gravity. Obtained by subtracting the vertical coordinate. The line-of-sight detection unit 21 determines the line-of-sight direction by referring to a reference table that represents the relationship between the relative position of the pupil center of gravity and the line-of-sight direction.

  Since the line-of-sight direction corresponding to the Purkinje image differs according to the positional relationship between the infrared camera 4 and the lighting light source, the reference table indicating the relationship between the relative position of the pupil center of gravity and the line-of-sight direction is an illumination light source 3-1. Each of 3-n is prepared in advance and stored in the memory 7. Therefore, the line-of-sight detection unit 21 may read a reference table corresponding to the specified lighting light source from the memory 7 and use it to detect the line-of-sight direction.

  In addition, when the Purkinje images of the two illuminating light sources that are lit are shown on the image, the line-of-sight detection unit 21 is, for example, a literature (Takehiko Ohno, “Gaze measurement by one-point calibration and its application”). The direction of the line of sight may be detected according to a method described in Information Processing Society of Japan, 2006). That is, the line-of-sight detection unit 21 calculates the three-dimensional position of the curvature center of the cornea based on the interval between the two Purkinje images, the assumed corneal curvature radius, and the distance from the infrared camera 4 to the cornea. In addition, the line-of-sight detection unit 21 represents the position of the center of gravity of the pupil on the image (that is, the direction from the infrared camera 4 toward the center of the pupil with respect to the optical axis of the infrared camera 4), and the infrared camera 4 Based on the distance to the cornea, the three-dimensional position of the center of gravity of the pupil is calculated. The line-of-sight detection unit 21 sets a vector obtained by subtracting the three-dimensional position of the center of curvature of the cornea from the three-dimensional position of the center of gravity of the pupil as a vector representing the line-of-sight direction of the user. The distance from the infrared camera 4 to the cornea is, for example, on the image relative to the distance on the image corresponding to the focal distance of the infrared camera 4 and the average distance between the centers of both eyes (64 mm). Is calculated based on the ratio of the center-to-center distances of both eyes. In this case, the distance between the centers of both eyes on the image may be calculated as, for example, the distance between the middle points between the corners of the left and right eyes and the eye head. Alternatively, when the digital signage system 1 has a distance sensor (not shown) such as a stereo camera or a depth camera, the distance from the infrared camera 4 to the cornea is obtained based on a measurement value by the distance sensor. Also good. Alternatively, when the position of the user is assumed in advance, the distance from the infrared camera 4 to the cornea may be stored in advance in the memory 7.

  Note that the line-of-sight detection unit 21 may detect the user's line-of-sight direction based on the positional relationship between one of the Purkinje images and the center of the pupil even when two Purkinje images are captured on the image. In this case, the line-of-sight detection unit 21 can detect the user's line-of-sight direction by the same method as when one Purkinje image is captured. Further, the line-of-sight detection unit 21 can specify the lighting light source corresponding to each Purkinje image according to the arrangement of the two Purkinje images. For example, among the two Purkinje images, the lighting light source corresponding to the Purkinje image located on the right side in the image is an illumination light source located on the right side of the two illumination light sources that are lit.

  The line-of-sight detection unit 21 passes the detected line-of-sight direction to the gaze position estimation unit 22.

The gaze position estimation unit 22 estimates the user's gaze position based on the detected gaze direction of the user.
In the present embodiment, the gaze position estimation unit 22 estimates the gaze position by referring to a gaze position table that represents the relationship between the gaze direction and the gaze position. Note that the relationship between the gaze direction and the gaze position changes according to the distance between the gaze position (for example, the display screen of the display device 2) and the user whose gaze direction is detected.

  Therefore, a plurality of gaze position tables are prepared in advance according to the assumed distance (for example, 1 m, 2 m, etc.) between the infrared camera 4 and the user using the digital signage system 1 and stored in the memory 7. The Then, the gaze position estimation unit 22 selects a gaze position table corresponding to the distance from the infrared camera 4 to the cornea among the plurality of gaze position tables. Then, the gaze position estimating unit 22 refers to the selected gaze position table to obtain a gaze position corresponding to the line-of-sight direction.

  FIG. 5 is a diagram illustrating an example of the gaze position table. The line of sight is displayed in the top row of the gaze position table 500. In each column of the gaze position table 500, the coordinates of the gaze position corresponding to the line-of-sight direction of the same column are expressed in a predetermined unit (for example, a pixel unit of the display screen of the display device 2 or a mm unit). . For example, the column 501 of the gaze position table 500 indicates that the gaze position when the line-of-sight direction is 0 ° in the horizontal direction and 1 ° in the vertical direction is (cx, cy + 40). Note that cx and cy are the gaze position when the line-of-sight direction is (0, 0), that is, the coordinates of the reference gaze position, for example, the horizontal coordinate and the vertical coordinate on the vertical plane at the attachment position of the infrared camera 4. .

  Note that the gaze position estimation unit 22 includes the position in the user's real space (for example, the position of the real space at the midpoint between the eyes of the user) and the object to be observed by the user (for example, the product shelf or the display device 2). The gaze position may be estimated based on the relative positional relationship with the display screen) and the line-of-sight direction.

  The gaze position estimation unit 22 writes the detected gaze direction and attention position in the detection history stored in the memory 7 every time the user's gaze direction and gaze position are detected.

ここで、P_tは、最新の画像に基づいて検出された視線方向であり、P_t-1は、１フレーム前の画像に基づいて検出された視線方向である。そしてP_t+1は、次フレームの赤外カメラ４による撮影時における、視線方向の予測値である。またαは補正係数であり、例えば、1に設定される。なお、補正係数αは、例えば、ユーザの眼から赤外カメラ４へ向かう方向と、視線方向とがなす角θが大きいほど、大きい値に設定される。そして補正係数αと角θとの関係を表すテーブルが予めメモリ７に保存され、予測部２３は、そのテーブルを参照して、角θに対応する補正係数αを決定すればよい。 The prediction unit 23 predicts the user's line-of-sight direction when the infrared camera 4 captures next based on the user's line-of-sight direction detected from each of at least two images generated at different timings. For example, the prediction unit 23 determines whether the infrared camera 4 is next based on the image of the current frame, that is, the user's line-of-sight direction detected from the latest image and the user's line-of-sight direction detected from the image one frame before. Predict the user's gaze direction when shooting. Note that the prediction unit 23 does not have to predict the line-of-sight direction until two line-of-sight direction detection results are obtained. The prediction unit 23 may predict the line-of-sight direction according to any of various prediction methods after obtaining the detection results of the line-of-sight direction for two times. For example, the prediction unit 23 predicts the line-of-sight direction according to the linear prediction method represented by the following expression until the obtained line-of-sight direction detection results reach a predetermined number of 2 or more.

Here, P _t is the line-of-sight direction detected based on the latest image, and P _t−1 is the line-of-sight direction detected based on the image one frame before. P _{t + 1} is a predicted value in the line-of-sight direction at the time of shooting by the infrared camera 4 in the next frame. Α is a correction coefficient, and is set to 1, for example. For example, the correction coefficient α is set to a larger value as the angle θ formed by the direction from the user's eye toward the infrared camera 4 and the line-of-sight direction is larger. A table representing the relationship between the correction coefficient α and the angle θ is stored in the memory 7 in advance, and the prediction unit 23 may determine the correction coefficient α corresponding to the angle θ by referring to the table.

Further, when the obtained detection result of the gaze direction reaches a predetermined number of 2 or more, the prediction unit 23 calculates a predicted value P _{t + 1} of the gaze direction at the time of image acquisition of the next frame using a prediction filter such as a Kalman filter. May be.

ここでf(x_i)は、視線方向の予測値を算出するための多項式であり、x_iは、i番目の画像についての視線方向の予測値を算出するために利用される、i番目の画像よりも前のn個のフレームのそれぞれの画像において検出された視線方向を表す。y_iは、i番目の画像について検出された視線方向を表す。なお、nは、視線方向の予測に用いられる、過去に検出された視線方向の数である。またσ_iは、正規化された、プルキンエ像間の距離に依存する視線方向の検出精度であり、プルキンエ像間の距離が大きいほど小さくなるように予め求められる。そしてσ_iは、i番目の画像におけるプルキンエ像間の距離に応じて決定されればよい。視線方向の検出に用いる二つのプルキンエ像間の間隔が大きいほど、計測分解能が向上するため、視線方向の検出精度が向上する。そこで、（２）式のように、視線検出精度を視線方向の予測に反映させることで、予測部２３は、視線方向の予測精度をより向上できる。 At this time, the prediction unit 23 may calculate the predicted value P _{t + 1} in the line-of-sight direction by polynomial approximation obtained in time series and using each of the detected plurality of line-of-sight directions as a variable. And when the prediction part 23 determines the polynomial, you may determine a polynomial so that the residual Q calculated according to following Formula may become the minimum.

Here, f (x _i ) is a polynomial for calculating the predicted value of the gaze direction, and x _i is used to calculate the predicted value of the gaze direction for the i-th image. It represents the line-of-sight direction detected in each image of n frames before the image. y _i represents the line-of-sight direction detected for the i-th image. Note that n is the number of gaze directions detected in the past used for gaze direction prediction. Also, σ _i is the normalized detection accuracy of the line-of-sight direction depending on the distance between Purkinje images, and is obtained in advance so as to decrease as the distance between Purkinje images increases. Σ _i may be determined according to the distance between Purkinje images in the i-th image. As the interval between the two Purkinje images used for the detection of the line-of-sight direction increases, the measurement resolution improves, so the detection accuracy of the line-of-sight direction improves. Therefore, as shown in the equation (2), the prediction unit 23 can improve the prediction accuracy of the gaze direction by reflecting the gaze detection accuracy in the prediction of the gaze direction.

The prediction unit 23 passes the predicted value P _{t + 1} in the line-of-sight direction at the time of shooting the next frame to the light source control unit 24.

Based on the predicted value P _{t + 1} of the line-of-sight direction at the time of shooting the next frame, the light source control unit 24 uses two illuminations that are turned on when the image of the next frame is acquired among the plurality of illumination light sources 3-1 to 3-n. Select a light source. Then, the light source control unit 24 generates a control signal for turning on the selected two illumination light sources and turning off the other illumination light sources among the plurality of illumination light sources 3-1 to 3-n. Output to the illumination light sources 3-1 to 3-n.

In the present embodiment, the light source control unit 24 refers to the lighting model that indicates the illumination light source in which the Purkinje image appears on the image for each line-of-sight direction, and the line-of-sight direction prediction value P _{t + 1} is detected as the line-of-sight direction. The illumination light source included in the possible range, that is, the illumination light source in which the Purkinje image appears on the image is specified.

  FIG. 6 is a diagram illustrating an example of a lighting model. The lighting model 600 shows one model for each row, and one model includes the real space position (x, y, z) of the head, the line-of-sight direction (θx, θy), and the illumination light source to be lit. Identification number. Note that x represents horizontal coordinates, y represents vertical coordinates, and z represents depth-direction coordinates along the optical axis of the infrared camera 4. Θx represents the angle in the horizontal direction, and θy represents the angle in the vertical direction. In addition, the real space position of a head can be made into the midpoint between both eyes of a user, for example.

For example, the light source control unit 24 identifies a model in which the line-of-sight direction closest to the predicted value P _{t + 1} of the line-of-sight direction is represented among the models represented in the lighting model 600. The light source control unit 24 refers to a model in which the real space position of the head closest to the position of the user's head among the identified models is represented, and the illumination corresponding to the predicted value P _{t + 1} in the line-of-sight direction What is necessary is just to specify a light source.

  7 and 8 are diagrams illustrating the principle of processing for selecting an illumination light source to be lit. 7 and 8, a detectable range 701 represents a detectable range of the user's line of sight, that is, a range of the line of sight in which the Purkinje image of the illumination light source 3-1 appears on the image. Similarly, the detectable ranges 702 to 706 represent ranges in the line-of-sight direction in which the Purkinje images of the illumination light sources 3-2 to 3-6 appear on the image, respectively.

With reference to FIG. 7, the control of the illumination light source to be lit until the prediction result of the line-of-sight direction becomes available will be described.
When the line-of-sight detection process is started, first, the light source control unit 24 has the largest detectable range of the line-of-sight direction, and a range in which the line-of-sight direction cannot be detected near the direction toward the infrared camera 4 is generated. Illumination light sources 3-2 and 3-5 are turned on so that there is not. In this case, it is assumed that the user's line-of-sight direction indicated by an arrow 711 is directed to a portion where each of the detectable ranges 701 to 703 overlaps. The light source control unit 24 refers to the lighting model and identifies the illumination light sources 3-1 to 3-3 as the illumination light sources that can detect the line-of-sight direction among the illumination light sources 3-1 to 3-6. That is, the light from the illumination light sources 3-1 to 3-3 is reflected by the points 721 to 723 on the user's cornea 700 and reaches the infrared camera 4. On the other hand, since the light from the illumination light sources 3-4 to 3-6 is reflected by the surface of the eyeball other than the cornea 700 of the user and reaches the infrared camera 4, no Purkinje image is captured. Among these three illumination light sources, the light source control unit 24 has the widest range in which the line-of-sight direction can be detected and the detection accuracy when detecting the line-of-sight direction using two Purkinje images is improved. The farthest illumination light sources 3-1 and 3-3 are selected as illumination light sources to be turned on. Then, the light source control unit 24 turns on the illumination light sources 3-1 and 3-3 and turns off the other illumination light sources when the infrared camera 4 is next photographed.

  Next, control of the illumination light source to be lit after the prediction result of the line-of-sight direction becomes available will be described with reference to FIG. An arrow 811 represents the line-of-sight direction detected based on the image of the previous frame, and an arrow 812 represents the line-of-sight direction detected based on the image of the current frame. An arrow 813 represents the line-of-sight direction predicted for the next frame.

  In the next frame, the light source control unit 24 selects an illumination light source to be lit so that the line-of-sight direction can be detected even if there is an error between the predicted line-of-sight direction and the actual line-of-sight direction. Therefore, the light source control unit 24 selects an illumination light source to be lit from illumination light sources in which both the gaze direction predicted for the next frame and the gaze direction for the current frame are included in the detectable range.

In FIG. 8, as indicated by an arrow 813, the predicted line-of-sight direction overlaps with the detectable ranges 702 to 704, so the illumination light source that can be detected for the predicted line-of-sight direction is the illumination light source 3-2. ~ 3-4. That is, it is assumed that light from the illumination light sources 3-2 to 3-4 is reflected at points 821 to 823 on the cornea 700 and reaches the infrared camera 4. On the other hand, light from the illumination light sources 3-1, 3-5, and 3-6 is assumed to be reflected by the surface of the eyeball other than the cornea 700 and reach the infrared camera 4. Further, as indicated by an arrow 812, the detected line-of-sight direction in the current frame overlaps with the detectable ranges 701 to 703, so that the illumination light source that can be detected for the line-of-sight direction in the current frame is an illumination light source. 3-1 to 3-3. That is, light from the illumination light sources 3-1 to 3-3 is reflected at points 831 to 833 on the cornea 700 and reaches the infrared camera 4. Therefore, the light source control unit 24 selects an illumination light source 3-2 and an illumination light source 3-3 that can detect the line-of-sight direction in both the current frame and the next frame.
When there are three or more illumination light sources that can detect the line-of-sight direction in both the current frame and the next frame, the light source control unit 24 selects the two most distant illumination light sources from those illumination light sources. Good.

  The light source control unit 24 generates a control signal for turning on the two selected illumination light sources among the illumination light sources 3-1 to 3-n and turning off the other illumination light sources for each frame. Are output to the illumination light sources 3-1 to 3-n. Further, the light source control unit 24 adds the identification numbers of the two selected illumination light sources to the detection history for each frame.

  FIG. 9 is an operation flowchart of the line-of-sight detection process performed by the processor 8 until the line-of-sight direction prediction result can be used for lighting control of the illumination light source. The processor 8 executes a line-of-sight detection process according to the following operation flowchart every time an image is acquired.

  The light source control unit 24 determines whether the line-of-sight direction has already been detected (step S101). If the line-of-sight direction has not been detected once, that is, immediately after the start of the line-of-sight detection process (step S101-No), the light source control unit 24 determines in advance among the illumination light sources 3-1 to 3-n. The two illumination light sources are turned on (step S102). On the other hand, if the line-of-sight direction is detected in the image of the immediately preceding frame (step S101—Yes), the light source control unit 24 turns on the two illumination light sources selected based on the detected line-of-sight direction and the lighting model. (Step S103).

  After step S102 or step S103, the infrared camera 4 captures the user and generates an image. The line-of-sight detection unit 21 detects the pupil and the Purkinje image on the generated image (step S104). The line-of-sight detection unit 21 determines whether there is one detected Purkinje image (step S105). If the detected Purkinje image is one (step S105-Yes), the line-of-sight detection unit 21 detects the line-of-sight direction based on one Purkinje image and the pupil (step S106). On the other hand, if there are two Purkinje images detected (step S105-No), the line-of-sight detection unit 21 detects the line-of-sight direction based on the two Purkinje images and the pupil (step S107).

  After step S106 or S107, the gaze position estimation unit 22 estimates the user's gaze position based on the detected gaze direction (step S108).

  Further, the prediction unit 23 determines whether or not the gaze direction is detected for two frames or more (step S109). If the line-of-sight direction is detected for two or more frames, the prediction unit 23 predicts the line-of-sight direction at the time of shooting the next frame based on the line-of-sight direction of the current frame and the line-of-sight direction of the previous frame (step S110). . After step S110 or if the number of frames in which the line-of-sight direction is detected in step S109 is less than 2, the processor 8 ends the line-of-sight detection process.

  FIG. 10 shows an operation flowchart of the gaze detection process after the gaze direction prediction result executed by the processor 8 becomes available for lighting control of the illumination light source. The processor 8 executes a line-of-sight detection process according to the following operation flowchart every time an image is acquired.

  The light source control unit 24 turns on the two illumination light sources selected based on the line-of-sight direction predicted in the previous frame and the line-of-sight direction detected in the previous frame and the lighting model (step S201).

  After the two illumination light sources are turned on, the infrared camera 4 captures the user and generates an image. The line-of-sight detection unit 21 detects the pupil and Purkinje image on the generated image (step S202). The line-of-sight detection unit 21 detects the line-of-sight direction based on the two Purkinje images and the pupil (step S203). The gaze position estimation unit 22 estimates the user's gaze position based on the detected gaze direction (step S204).

  The prediction unit 23 predicts the gaze direction at the time of shooting the next frame based on at least the gaze direction of the current frame and the gaze direction of the previous frame (step S205). After step S205, the processor 8 ends the line-of-sight detection process.

  As described above, this line-of-sight detection apparatus is configured so that the Purkinje image appears on the image based on the predicted line-of-sight direction of the illumination light source to be turned on among the plurality of illumination light sources, that is, the Purkinje image. Is selected so that the line-of-sight direction can be detected. At this time, the line-of-sight detection device includes an illumination light source that is turned on so that the Purkinje image of the illumination light source to be turned on is displayed on the image for both the line-of-sight direction predicted for the next frame and the line-of-sight direction detected for the current frame. select. As a result, this gaze detection apparatus can make the Purkinje image always appear on the image, so that the gaze direction detection accuracy can be improved. When there are three or more illumination light sources in which the Purkinje image appears on the image, the line-of-sight detection device turns on the two illumination light sources that are farthest from each other among the three or more illumination light sources. Thereby, since this gaze detection apparatus can enlarge the space | interval between Purkinje images on an image as much as possible, it can improve the detection accuracy of a gaze direction more.

  According to the modification, the line-of-sight detection device may turn on only one of the plurality of illumination light sources and turn off the others for each frame. In this case, every time an image generated by the infrared camera 4 is obtained, the line-of-sight detection unit 21 detects the user's line-of-sight according to the method of detecting the line-of-sight direction using one Purkinje image in step S203 of FIG. What is necessary is just to detect a direction.

  Further, immediately after the line-of-sight detection process is started, the light source control unit 24 turns on one preset illumination light source in step S102 of FIG. Then, the light source control unit 24 detects the detected gaze direction by referring to the lighting model based on the latest gaze direction detection result in step S103 of FIG. 9 until the gaze direction prediction result is obtained. One illumination light source included in the possible range is turned on. Then, when the prediction result of the line-of-sight direction is obtained, the light source control unit 24 refers to the lighting model in step S201 of FIG. 10, and both the predicted line-of-sight direction and the latest line-of-sight direction are within the detectable range. One illumination light source included in the light source may be turned on.

  Similarly to the line-of-sight detection apparatus according to the above-described embodiment, the line-of-sight detection apparatus according to this modification also uses an illumination light source that is turned on from among a plurality of illumination light sources so that a Purkinje image is captured on an image obtained by the infrared camera 4. Since the selection is made, the detection accuracy of the line-of-sight direction can be improved.

  Further, according to another modification, the light source control unit 24 can detect that the line-of-sight direction of the user predicted at the next shooting by the infrared camera 4 is included regardless of the line-of-sight direction detected from the latest image. The illumination light source corresponding to the range may be turned on at the next shooting. As a result, the line-of-sight detection device can turn on the illumination light source in which the Purkinje image appears on the image, particularly even when the line-of-sight direction is moving fast and greatly. Note that the light source control unit 24 may switch the method of selecting the illumination light source to be lit based on the difference between the detected gaze direction of the current frame and the predicted gaze direction of the next frame. For example, the light source control unit 24 may compare the absolute difference between the detected gaze direction of the current frame and the predicted gaze direction of the next frame with a predetermined threshold value. Then, if the absolute difference value is larger than the predetermined threshold, the light source control unit 24 includes the user's line-of-sight direction predicted at the next shooting by the infrared camera 4 regardless of the line-of-sight direction detected from the latest image. The illumination light source corresponding to the detectable range is turned on at the next shooting. On the other hand, if the difference absolute value is equal to or smaller than a predetermined threshold, the light source control unit 24 falls within a detectable range that includes both the line-of-sight direction detected from the latest image and the user's line-of-sight direction predicted at the next shooting. The corresponding illumination light source may be turned on at the next shooting.

  Furthermore, according to another modification, the lighting model may represent a range in the line-of-sight direction corresponding to a detectable range for each illumination light source. Also in this case, the light source control unit 24 may identify an illumination light source that includes the predicted line-of-sight direction and the detected line-of-sight direction of the current frame within the detectable range with reference to the lighting model.

  The computer program that realizes the function of the processor of the line-of-sight detection device according to the above-described embodiment or its modification may be provided in a form recorded on a computer-readable recording medium such as a magnetic recording medium or an optical recording medium. This recording medium does not include a carrier wave.

  All examples and specific terms listed herein are intended for instructional purposes to help the reader understand the concepts contributed by the inventor to the present invention and the promotion of the technology. It should be construed that it is not limited to the construction of any example herein, such specific examples and conditions, with respect to showing the superiority and inferiority of the present invention. Although embodiments of the present invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made thereto without departing from the spirit and scope of the present invention.

The following supplementary notes are further disclosed regarding the embodiment described above and its modifications.
(Appendix 1)
A pupil region in which the pupil of the eye is reflected, and a corneal reflection image of an illumination light source that is lit among a plurality of illumination light sources are detected from an image generated by the imaging unit photographing the user's eyes. ,
Detecting the user's gaze direction based on the pupillary region and the corneal reflection image;
Based on the user's line-of-sight direction detected from each of the at least two images generated at different timings, predicting the user's line-of-sight direction at the time of the next shooting of the imaging unit,
For each of the plurality of illumination light sources, a detectable range of the line-of-sight direction is defined according to the positional relationship between the illumination light source and the imaging unit, and the predicted line-of-sight direction of the user among the plurality of illumination light sources is At least one of the illumination light sources corresponding to the detectable range included is turned on at the next shooting of the imaging unit,
A computer program for eye-gaze detection for causing a computer to execute the above.
(Appendix 2)
Turning on the illumination light source corresponds to the detectable range including both the user's line-of-sight direction detected from the latest image and the predicted user's line-of-sight direction among the plurality of illumination light sources. The computer program for eye-gaze detection according to appendix 1, wherein at least one illumination light source is turned on.
(Appendix 3)
Turning on the illumination light source corresponds to the detectable range including both the user's line-of-sight direction detected from the latest image and the predicted user's line-of-sight direction among the plurality of illumination light sources. The computer program for eye-gaze detection according to appendix 2, wherein when there are three or more illumination light sources to be turned on, the two most distant illumination light sources among the three or more illumination light sources are turned on.
(Appendix 4)
The lighting of the illumination light source turns on two of the illumination light sources corresponding to the detectable range including the predicted user's line-of-sight direction among the plurality of illumination light sources. Computer program for eye gaze detection.
(Appendix 5)
Turning on the illumination light source means that, when there are three or more illumination light sources corresponding to the detectable range including the predicted user's line-of-sight direction, the three or more illumination light sources. The computer program for line-of-sight detection according to appendix 4, which turns on the two most distant illumination light sources among the illumination light sources.
(Appendix 6)
A pupil region in which the pupil of the eye is reflected, and a corneal reflection image of an illumination light source that is lit among a plurality of illumination light sources are detected from an image generated by the imaging unit photographing the user's eyes. ,
Detecting the user's gaze direction based on the pupillary region and the corneal reflection image;
Based on the user's line-of-sight direction detected from each of the at least two images generated at different timings, predicting the user's line-of-sight direction at the time of the next shooting of the imaging unit,
For each of the plurality of illumination light sources, a detectable range of the line-of-sight direction is defined according to the positional relationship between the illumination light source and the imaging unit, and the predicted line-of-sight direction of the user among the plurality of illumination light sources is At least one of the illumination light sources corresponding to the detectable range included is turned on at the next shooting of the imaging unit,
A gaze detection method including the above.
(Appendix 7)
A pupil region in which the pupil of the eye is reflected, and a corneal reflection image of an illumination light source that is lit among a plurality of illumination light sources are detected from an image generated by the imaging unit photographing the user's eyes. A gaze detection unit that detects the gaze direction of the user based on the pupil region and the corneal reflection image;
A predicting unit that predicts the user's line-of-sight direction at the next shooting of the imaging unit based on the user's line-of-sight direction detected from each of the at least two images generated at different timings;
For each of the plurality of illumination light sources, a detectable range of the line-of-sight direction is defined according to the positional relationship between the illumination light source and the imaging unit, and the predicted line-of-sight direction of the user among the plurality of illumination light sources is A light source control unit that turns on at least one of the illumination light sources corresponding to the included detectable range at the next imaging of the imaging unit;
A line-of-sight detection apparatus comprising:

1 Digital signage system (Gaze detection device)
2 Display device 3-1 to 3-n Illumination light source 4 Infrared camera 5 Input device 6 Storage medium access device 7 Memory 8 Processor 9 Storage medium 10 Case 21 Gaze detection unit 22 Gaze position estimation unit 23 Prediction unit 24 Light source control unit

Claims (5)

A pupil region in which the pupil of the eye is reflected, and a corneal reflection image of an illumination light source that is lit among a plurality of illumination light sources are detected from an image generated by the imaging unit photographing the user's eyes. ,
Detecting the user's gaze direction based on the pupillary region and the corneal reflection image;
Based on the user's line-of-sight direction detected from each of the at least two images generated at different timings, predicting the user's line-of-sight direction at the time of the next shooting of the imaging unit,
For each of the plurality of illumination light sources, a detectable range of the line-of-sight direction is defined according to the positional relationship between the illumination light source and the imaging unit, and the predicted line-of-sight direction of the user among the plurality of illumination light sources is At least one of the illumination light sources corresponding to the detectable range included is turned on at the next shooting of the imaging unit,
A computer program for eye-gaze detection for causing a computer to execute the above.   Turning on the illumination light source corresponds to the detectable range including both the user's line-of-sight direction detected from the latest image and the predicted user's line-of-sight direction among the plurality of illumination light sources. The computer program for eye gaze detection according to claim 1, wherein at least one illumination light source is turned on.   Turning on the illumination light source corresponds to the detectable range including both the user's line-of-sight direction detected from the latest image and the predicted user's line-of-sight direction among the plurality of illumination light sources. The computer program for line-of-sight detection according to claim 2, wherein when there are three or more illumination light sources to be turned on, the two most distant illumination light sources among the three or more illumination light sources are turned on. A pupil region in which the pupil of the eye is reflected, and a corneal reflection image of an illumination light source that is lit among a plurality of illumination light sources are detected from an image generated by the imaging unit photographing the user's eyes. ,
Detecting the user's gaze direction based on the pupillary region and the corneal reflection image;
Based on the user's line-of-sight direction detected from each of the at least two images generated at different timings, predicting the user's line-of-sight direction at the time of the next shooting of the imaging unit,
For each of the plurality of illumination light sources, a detectable range of the line-of-sight direction is defined according to the positional relationship between the illumination light source and the imaging unit, and the predicted line-of-sight direction of the user among the plurality of illumination light sources is At least one of the illumination light sources corresponding to the detectable range included is turned on at the next shooting of the imaging unit,
A gaze detection method including the above. A pupil region in which the pupil of the eye is reflected, and a corneal reflection image of an illumination light source that is lit among a plurality of illumination light sources are detected from an image generated by the imaging unit photographing the user's eyes. A gaze detection unit that detects the gaze direction of the user based on the pupil region and the corneal reflection image;
A predicting unit that predicts the user's line-of-sight direction at the next shooting of the imaging unit based on the user's line-of-sight direction detected from each of the at least two images generated at different timings;
For each of the plurality of illumination light sources, a detectable range of the line-of-sight direction is defined according to the positional relationship between the illumination light source and the imaging unit, and the predicted line-of-sight direction of the user among the plurality of illumination light sources is A light source control unit that turns on at least one of the illumination light sources corresponding to the included detectable range at the next imaging of the imaging unit;
A line-of-sight detection apparatus comprising:

Images