JP6856200B2 - Line-of-sight detection and calibration methods, systems, and computer programs - Google Patents

Description

The present invention relates to a calibration method in line-of-sight detection.

Conventionally, a line-of-sight detection technique capable of measuring the line-of-sight position has been known (for example, Patent Documents 1 and 2). It is used for psychological research on vision, medical tests, and applied research to man-machine interfaces. In such a line-of-sight detection technique, the positions of both eyes and the size and shape of the eyeballs differ depending on the person, so that a calibration process is required when detecting the line-of-sight.

As a conventional calibration method, for example, a 3 × 3 calibration point as shown in FIG. 10 is displayed on a display, and the subject is asked to pay attention to these calibration points in order, and the pupil of the subject at that time is examined. There is a way to shoot with a camera.

Japanese Patent No. 3726122 Japanese Patent No. 3834636

However, according to the conventional calibration method, calibration cannot be performed unless the gazing point is gazed at. Therefore, when detecting the line of sight by an animal such as a marmoset or a Japanese macaque, in the case of a child, or in the case of a subject suffering from dementia, etc. There are problems such as it is difficult to have the gaze point of view, or it takes time and effort to gaze at the gaze after training.

In view of such a problem, an object of the present invention is to provide a line-of-sight detection and calibration method that does not require training for a subject to gaze at a gazing point.

In order to solve the above problems, one aspect of the present invention includes a step of displaying visual stimulus information and line-of-sight information representing the movement of the subject's line of sight while displaying the visual stimulus information by photographing the subject's eyeball. And the step of acquiring the line-of-sight information, while shifting the line-of-sight information in the time axis direction, with respect to the horizontal position and the vertical position of the locus information indicating the locus of the visual stimulus information being displayed and the line-of-sight information. The step of calculating the difference between the line-of-sight information and the line-of-sight information at each time, and the count number of each value of the difference between the horizontal position and the vertical position of the locus information and the line-of-sight information at each calculated time are the largest. It is a line-of-sight detection and calibration method including a step of calibrating the line-of-sight information based on the horizontal position and the vertical position at the time of

In addition, another aspect of the present invention is a display unit that displays visual stimulus information and a line of sight that acquires line-of-sight information that represents the movement of the subject's line of sight while displaying the visual stimulus information by photographing the eyeball of the subject. While shifting the line-of-sight information in the time axis direction with the information acquisition unit, the locus information and the locus information and the vertical position of each of the horizontal and vertical positions of the locus information indicating the locus of the visual stimulus information being displayed and the line-of-sight information are described. The difference calculation unit that calculates the difference between the line-of-sight information and the line-of-sight information at each time, and the count number of each value of the difference between the locus information and the line-of-sight information at the horizontal position and the vertical position at each calculated time are the largest. It is a line-of-sight detection and calibration system including a calibration unit that calibrates the line-of-sight information based on the horizontal position and the vertical position when the number of lines increases.

Another aspect of the present invention is a computer program for causing a computer system to execute the above-mentioned line-of-sight detection and calibration method.

It is a figure which shows an example of the structure of the line-of-sight detection calibration system which executes the line-of-sight detection calibration method which concerns on one Embodiment of this invention. It is a figure which shows an example of the processing flow diagram of the line-of-sight detection calibration method which concerns on one Embodiment of this invention. It is a figure which shows an example of the graph which shows the movement of the visual stimulus information and the movement of the eyes of a subject. It is a figure which shows an example of the graph which shows the movement of the visual stimulus information and the movement of the eyes of a subject. It is a figure which shows an example of the graph which shows the movement of the visual stimulus information and the movement of the eyes of a subject. It is a figure which shows an example of the graph which shows the movement of the visual stimulus information and the movement of the eyes of a subject after executing the line-of-sight detection calibration method which concerns on one Embodiment of this invention. It is a figure explaining the extraction of the calibration point. It is a figure which shows the other experimental data. It is a figure which shows after the calibration of other experimental data. It is a figure which shows an example of the conventional calibration point.

The line-of-sight detection calibration method according to the present embodiment is different from the conventional calibration method, and is a calibration method that does not require special training even for animals such as marmosets and children. In general, animals and humans have the property that if there is something moving in front of them, they will naturally follow it. Therefore, in the line-of-sight detection and calibration method according to the present embodiment, the visual stimulus information to be followed by the eyes is moved in a predetermined trajectory, and the movement of the visual stimulus information and the subject (line-of-sight detection) are applied to the line of sight of the animal or human being to follow the line of sight. The human or animal that is the target of the above is referred to as the "subject"), and the movement of the line of sight is recorded. Then, by comparing these data and extracting a portion where the visual stimulus information and the movement of the eyes of the subject are synchronized, the calibration process in the line-of-sight detection is performed.

Hereinafter, the line-of-sight detection and calibration method according to the present embodiment will be described with reference to the drawings. In each of the figures referred to in the following description, the same parts as those of the other figures are indicated by the same reference numerals.

(Configuration of line-of-sight detection and calibration system)
FIG. 1 is a diagram showing an example of a configuration of a line-of-sight detection and calibration system 1 that executes the line-of-sight detection and calibration method according to the present embodiment. The line-of-sight detection configuration system shown in FIG. 1 includes a display 10 that presents visual stimulus information, an infrared illumination device 12 that illuminates the eyeball 54 of the subject 50 with infrared rays, a mirror 14 that reflects only infrared rays, and an eyeball of the subject 50. It is configured to include a camera 20 for photographing the state of 54, a computer device 16 for presenting visual stimuli for outputting display information for displaying visual stimulus information on the display 10, and a computer device 18 for measuring the line of sight of the subject 50. To.

When the computer device 16 for presenting the visual stimulus outputs the display information for displaying the visual stimulus information on the display 10, the visual stimulus information is displayed on the display 10 in a predetermined trajectory. Further, the head 52 of the subject 50 is fixed in a state of facing the display 10. In addition, the infrared illumination device 12 illuminates the pupil in order to photograph the pupil darkly. When the subject 50 follows the visual stimulus information displayed on the display 10 with the eyes (eyeballs) 54, the camera 20 photographs the state of the eyeballs 54 through the mirror 14 that reflects only infrared rays. The shooting information of the camera 20 is output to the line-of-sight measurement computer device 18. Further, the visual stimulus presentation computer device 16 outputs the presentation position of the visual stimulus information to the line-of-sight measurement computer device 18 via, for example, a network. Further, the line-of-sight detection calibration process according to the present embodiment, which will be described later, can be executed, for example, in the line-of-sight measurement computer device 18.

The visual stimulus information is displayed on the display 10 so that the subject can follow it with his / her eyes, but it is preferable to use information that attracts the subject's interest as much as possible. For example, there is a research result that marmosets prefer to see facial images (The Journal of Neuroscience, January 22, 2014, 34 (4): 1183-1194, 1183 Active Vision in Marmosets: A Model System for Visual Neuroscience).

Here, in the line-of-sight detection and calibration system 1 as shown in FIG. 1, the line-of-sight position measured by the camera 20 and the position actually viewed by the subject 50 are determined by the positional relationship between the head 52 (eyeball 54) and the camera 20. There may be a deviation between the horizontal direction and the vertical direction. The line-of-sight detection calibration method according to the present embodiment is a method for calibrating such a deviation.

(Processing flow)
FIG. 2 is a diagram showing an example of a processing flow diagram of the line-of-sight detection calibration method according to the present embodiment.

When the display information for displaying the visual stimulus information is output from the computer device 16 for presenting the visual stimulus to the display 10, the visual stimulus information is displayed on the display 10 (step S102). At this time, the presentation position of the visual stimulus information is output to the line-of-sight measurement computer device 18 (step S104). While the visual stimulus information is displayed on the display 10, the state of the eyeball 54 of the subject 50 is photographed by the camera 20, and the photographed information (line-of-sight information) is output to the line-of-sight measurement computer device 18 (step S106). Steps S102 to S106 are repeated until the display of the visual stimulus information is completed (step S108: No). When the visual stimulus information moves on the display in a predetermined locus, the locus information is provided to the line-of-sight measurement computer device 18 before the calibration process (step S110) (step S110). (Output), and while the visual stimulus information is being displayed, only the photographing information (line-of-sight information) related to the movement (line of sight) of the subject's eyeball may be output to the line-of-sight measurement computer device 18.

When the display of the visual stimulus information is completed (step S108: Yes), calibration is performed using the locus information of the visual stimulus information and the line-of-sight information of the subject (step S110). The calibration method in step S110 will be described in detail below.

(Gaze detection calibration method)
Hereinafter, the calibration process executed in step S110 will be described in detail with reference to the drawings.

3A and 3B are graphs showing the movement of the visual stimulus information acquired by the line-of-sight measurement computer device 18 and the movement of the eyes of the subject (marmoset in this example). The broken line in the graph represents the movement of the visual stimulus information, and the solid line represents the movement of the subject's eyes (hereinafter, also referred to as “line-of-sight information”). Further, in the graph of FIG. 3A, the vertical axis is the horizontal angle and the horizontal axis is the time axis, and represents the time-dependent changes of the visual stimulus information and the line-of-sight information, respectively. Further, in the graph of FIG. 3B, the vertical axis is the vertical angle and the horizontal axis is the time axis, and represents the time-dependent changes of the visual stimulus information and the line-of-sight information, respectively. Further, in the graph of FIG. 3 (c), the data of FIGS. 3 (a) and 3 (b) are represented by an angle in the vertical direction on the vertical axis and an angle in the horizontal direction on the horizontal axis.

In each graph of FIG. 3, when the subject follows the visual stimulus information with his / her eyes, the movement of the visual stimulus information (broken line) and the movement of the subject's eyes (solid line) are similar, but the subject is visual. There is a slight time lag when following the stimulus information visually. First, this time difference is extracted.

FIG. 4 (a-1) shows changes over time in the horizontal direction between the movement of visual stimulus information (broken line) and the movement of the subject's eyes (solid line) (corresponding to FIG. 3 (a)). The graph of FIG. 4A-2 is a graph obtained by plotting the result of subtracting the value of the solid line from the broken line at each time on such a graph. Further, FIG. 4 (a-3) is a graph of the total number of plots at each angle (vertical axis: degree) for this graph.

In the calibration process of the present embodiment, the above process is repeated while shifting the eye movement (solid line) of the subject in the graph of FIG. 4 (a-1) in the time axis direction. In the case of this example, since the data is acquired at 500 Hz, the solid line graph is shifted in the time axis direction by 1/500 second. Further, FIGS. 4 (b-1) to 4 (b-3) are graphs in the case where the movement of the visual stimulus information and the vertical change of the eye movement of the subject are processed in the same manner as in the horizontal direction. Is shown.

FIG. 5 shows an example in which the process described in FIG. 4 is repeated while shifting the solid line in the time axis direction. FIG. 5 (a-1) shows the solid line data of the graph of FIG. 4 (a-1) shifted to the right on the time axis. Further, each graph of FIGS. 5 (a-2) and 5 (a-3) is a graph generated by the same processing as in FIGS. 4 (a-2) and 4 (a-3).

Here, comparing the graph of FIG. 4 (a-3) with the graph of FIG. 5 (a-3), the maximum value of the total number of plots (count) is about about the graph of FIG. 4 (a-3). It is 108, which is about 191 in the graph of FIG. 5 (a-3). That is, the maximum value of the total number of plots is larger in FIG. 5 (a-3) than in FIG. 4 (a-3). In this way, while shifting the graph (solid line) of the subject's eye movement in the time axis direction, the total number of plots (count) at each angle is calculated, and the time when the maximum value of the total number of plots becomes the largest. It is estimated that the amount of shift in the axial direction is the time difference when the subject visually follows the visual stimulus information. In this example, the maximum value of the total number of plots at each angle was the largest when the solid line graph was shifted by about 200 milliseconds. Further, FIGS. 5 (b-1) to 5 (b-3) show graphs in the vertical direction when the same processing as in the horizontal direction is performed. If the timing (time difference) at which the maximum value of the total number of plots is maximized differs between the horizontal direction and the vertical direction, it is perpendicular to the total number of plots in the horizontal direction while shifting the horizontal direction and the vertical direction by the same amount of time. You may want to find the time when the sum of the total number of directional plots is maximized.

Further, assuming that the graph of FIG. 5 shows the case where the maximum value of the total number of plots is the largest, the angle (vertical axis) when the total number of plots is the maximum (about 191 pieces) in the horizontal direction is It is +3 degrees (Fig. 5 (a-3)). The angle (vertical axis) when the total number of plots is maximum (about 191) in the vertical direction is -13 degrees (FIG. 5 (b-3)). These angles represent the deviation between the camera and the subject's eyeball in the horizontal and vertical directions.

By the above processing, it is possible to correct the time difference when the subject follows the visual stimulus information with the eyes, and the deviation of the position between the camera 20 and the subject's eyeball 54 is estimated. The graph of FIG. 6 is a calibration of the graph of FIG. 3 on the vertical axis (angle) and the horizontal axis (time axis) based on the estimated time and position deviation. The graphs of FIGS. 6A, 6B, and 6C correspond to the graphs of FIGS. 3A, 3B, and 3C, respectively. Compared with FIG. 3, since the time and position deviations are calibrated in each graph of FIG. 6, the movement of the visual stimulus information (broken line) and the movement of the subject's eyes (solid line) are closer to each other as a whole. You can see that it has become.

Further, a point close to the plot of the broken line graph of FIG. 6 (c) and the plot of the solid line graph can be a calibration point. Actually, for example, the distance between the plots of the broken line graph and the solid line graph may be extracted as a calibration point within a predetermined distance. 7 (a) and 7 (b) are clarified plots on the graphs of FIGS. 6 (a) and 6 (b), and FIG. 7 (c) is a broken line graph and a solid line graph, respectively. It is a graph generated by extracting plots in which the distance between plots is less than 4 degrees.

If the number of calibration points extracted as described above is too small, the process for determining the calibration points may be restarted from the beginning.

As described above, the calibration points extracted by the line-of-sight detection calibration method according to the present embodiment have an advantage that the number of calibration points is larger than that in the conventional case. In addition, even if the subject is an animal, a child, a person suffering from dementia, etc., for which it is difficult for the subject to consciously gaze at the calibration point as in the conventional calibration method, the calibration is performed. It is possible to perform more accurate line-of-sight detection.

(Other experimental examples)
8 and 9 are diagrams showing other experimental examples. FIG. 8 is a diagram showing a locus of visual stimulus information and a graph of the movement of the eyes of the subject, corresponding to FIG. Further, FIG. 9 is a diagram showing a graph corresponding to the above-mentioned FIG. 6 after the data shown in FIG. 8 is calibrated by executing the line-of-sight detection calibration method of the present embodiment. Overall, it can be seen that the locus of visual stimulus information (broken line graph) and the movement of the subject's eyes (solid line graph) overlap more in FIG. 9 than in FIG.

(Hardware configuration)
A computer device (for example, a line-of-sight measurement computer device 18) that executes the line-of-sight detection and calibration method according to the present embodiment can be realized by a hardware configuration similar to that of a general computer device. The computer device includes, for example, a processor, RAM, ROM, hard disk device, removable memory, communication interface (wired / wireless), display / touch panel, speaker, keyboard / mouse, and the like.

Then, each process in the line-of-sight detection calibration method of the present embodiment described above can be realized by the cooperation of hardware and software. For example, each process can be executed by the processor reading a program stored in the hard disk device in advance into the memory and executing the program. The hardware configuration described above is merely an example, and the present invention is not limited to this.

Further, the line-of-sight detection and calibration system of the present embodiment described above has a display unit (for example, display 10) that displays visual stimulus information and the movement of the subject's line of sight while displaying the visual stimulus information by photographing the eyeball of the subject. The line-of-sight information acquisition unit (for example, the line-of-sight measurement computer device 18) that acquires the line-of-sight information representing the above, and the locus information indicating the locus of the visual stimulus information being displayed and the line-of-sight while shifting the line-of-sight information in the time axis direction. A difference calculation unit (for example, a line-of-sight measurement computer device 18) that calculates the difference between the locus information and the line-of-sight information at each time for each of the horizontal position and the vertical position with respect to the information, and the locus information and the line-of-sight information. A calibration unit that calibrates line-of-sight information based on the horizontal and vertical positions when the count of each value of the difference at each calculated time of the horizontal position and the vertical position is maximum (for example, a computer device for line-of-sight measurement). It can be said that it is a line-of-sight detection and calibration system including 18).

Although one embodiment of the present invention has been described so far, it goes without saying that the present invention is not limited to the above-described embodiment and may be implemented in various different forms within the scope of the technical idea.

The scope of the present invention is not limited to the exemplary embodiments illustrated and described, but also includes all embodiments that provide an effect equal to that intended by the present invention. Furthermore, the scope of the present invention is not limited to the combination of the features of the invention defined by each claim, but may be defined by any desired combination of the specific features of all the disclosed features. ..

1 Eye-gaze detection and calibration system 10 Display 12 Infrared illumination device 14 Mirror that reflects only infrared rays 16 Computer device for presenting visual stimuli 18 Computer device for eye-gaze measurement 50 Subject 52 Head 54 Eyeball

Claims (5)

Steps to display visual stimulus information and
A step of acquiring gaze information representing the movement of the gaze of the subject while displaying the visual stimulus information by photographing the eyeball of the subject, and
While shifting the line-of-sight information in the time axis direction, each of the locus information and the line-of-sight information is provided for each of the horizontal and vertical positions of the locus information indicating the locus of the visual stimulus information being displayed and the line-of-sight information. Steps to calculate the difference in time and
Based on the horizontal position and the vertical position when the count number of each value of the difference between the horizontal position and the vertical position of the locus information and the line-of-sight information at each of the calculated times is the largest. Steps to calibrate line-of-sight information and
Line-of-sight detection calibration method including. The step of calibrating the line of sight is the horizontal position and the horizontal position when the count number of each value of the difference between the horizontal position and the vertical position of the locus information and the line of sight information at each of the calculated times is the largest. The first aspect of claim 1, wherein the line-of-sight information is calibrated by using a portion where the line-of-sight information and the locus information are synchronized in a state where the line-of-sight information or the locus information is shifted by the amount of the vertical position. Line-of-sight detection and calibration method. The line-of-sight detection and calibration method according to claim 2, wherein the step of calibrating the line of sight is a place where the distance between the line-of-sight information and the locus information is less than a predetermined threshold value at the synchronized portion. A display unit that displays visual stimulus information and
A line-of-sight information acquisition unit that acquires line-of-sight information representing the movement of the subject's line-of-sight while displaying the visual stimulus information by photographing the subject's eyeball.
While shifting the line-of-sight information in the time axis direction, each of the locus information and the line-of-sight information is provided for each of the horizontal and vertical positions of the locus information indicating the locus of the visual stimulus information being displayed and the line-of-sight information. A difference calculation unit that calculates the difference at time, and
Based on the horizontal position and the vertical position when the count number of each value of the difference between the horizontal position and the vertical position of the locus information and the line-of-sight information at each of the calculated times is the largest. A calibration unit that calibrates line-of-sight information,
A line-of-sight detection and calibration system equipped with. A computer program for causing a computer system to execute the line-of-sight detection calibration method according to any one of claims 1 to 3.

Images