JP2019063310A - Sight line detection device, sight line detection program, and sight line detection method - Google Patents

Abstract

To accurately detect a sight line of an observer who observes a target.SOLUTION: On the basis of the difference between sight line data of the left eye of an observer who observes a target and sight line data of the right eye of the observer, a computer finds (step 801) an index indicating variation in sight lines of the observer. Then, on the basis of the index indicating the variation, the computer determines (step 802) the retention of the sight line of the observer and on the basis of a result of the retention determination, determines (step 803) the position of the sight line of the observer.SELECTED DRAWING: Figure 8

Description

  The present invention relates to a gaze detection apparatus, a gaze detection program, and a gaze detection method.

  The gaze detection technology is a technology for specifying a place where an observer who observes an object is looking, and is used in various scenes. For example, a screen of a personal computer (PC), a display shelf of a product in a store, or the like can be an object to be observed.

  There is also known a pointing device based on a line-of-sight input that causes a cursor to be displayed corresponding to the user's gaze point on a display, and a technique for removing saccade components from the output of a binocular eye movement measurement sensor (for example, Patent Document 1 and Patent Reference 2). There is also known a method of evaluating visual fatigue based on fixation involuntary eye movement (see, for example, Non-Patent Document 1).

WO 2009/019760 pamphlet Japanese Patent Application Laid-Open No. 7-308290

Takeshi Goshi and Sakamoto Kazuyoshi, "Study on visual fatigue evaluation of VDT work with fixation eye movement (flick)", Ergonomics, Vol. 32, No. 2, P.I. 87-97, 1996

  In the case where the determination of the user's line of sight is determined using the technique of Patent Document 1, the correct determination result may not necessarily be obtained.

  Such a problem occurs not only in the line of sight of the user looking at the PC screen but also in the case of detecting the line of sight of the observer observing another object.

  In one aspect, the present invention aims to accurately detect the line of sight of an observer observing an object.

In one scheme, the program causes the computer to perform the following processing.
(1) The computer obtains an index indicating the variation in the line of sight of the observer based on the difference between the line of sight data of the left eye of the observer observing the object and the line of sight data of the right eye of the observer.
(2) The computer makes a determination on the standing of the line of sight of the observer based on the index indicating the variation.
(3) The computer determines the line-of-sight position of the observer based on the determination result of the stop determination.

  According to the embodiment, it is possible to accurately detect the line of sight of the observer who observes the object.

It is a figure showing a look sensor. It is a figure which shows the gaze data which shows the gaze position on a screen. It is a figure which shows the detection result of a gaze position. It is a figure which shows the change of a gaze position. It is a figure which shows gaze data generated from a low rate picture. It is a figure which shows the relationship between the sample number of gaze position, and the minimum inclusion circle. It is a functional block diagram of a gaze detection device. It is a flowchart of eye gaze detection processing. It is a functional block diagram which shows the example of a gaze detection apparatus. It is a figure which shows gaze detection by a gaze sensor. It is a figure which shows congestion. It is a figure which shows the difference of the horizontal direction of gaze data. It is a figure which shows the difference of the orthogonal | vertical direction of gaze data. It is a figure which shows the change of the gaze position of both eyes. It is a figure which shows the gaze data corresponding to the change of the gaze position of both eyes. It is a flowchart which shows the specific example of a gaze detection process. It is a block diagram of an information processor.

Hereinafter, embodiments will be described in detail with reference to the drawings.
FIG. 1 shows an example of a gaze sensor detecting a gaze of a user observing a PC screen. Images 111 to 113 are displayed on the screen 102 of the display device 101, and a line-of-sight sensor 103 is provided below the screen 102. The gaze sensor 103 includes a light emitting diode (LED) 121 and a camera 122. At this time, the gaze position of the user on the screen 102 may move between the image 111 and the image 113.

  The LED 121 emits infrared light to the face of the user looking at the screen 102, and the camera 122 captures the face of the user. The PC connected to the line-of-sight sensor 103 generates line-of-sight data indicating the line-of-sight position from the image of the eye of the user captured by the camera 122. The state indicated by the line-of-sight data generated from the image of the eye includes a large line-of-sight movement called saccade, a stop at which the line-of-sight stays in one place, and a line-of-sight variation called fixation eye movement.

  FIG. 2 shows an example of gaze data indicating a gaze position on the screen 102. Assuming that the coordinates in the horizontal direction of the screen 102 are x-coordinates and the coordinates in the vertical direction are y-coordinates, the gaze data can be represented by coordinates (x, y) of the gaze position of the user changing in time series. The horizontal axis of the graph of FIG. 2 represents the number of frames of the image captured by the camera 122, and the vertical axis represents the x-coordinate of the gaze position.

  The line of sight stops around each of the gaze position 201 to the gaze position 203 at which the user gazes on the screen 102, and a saccade in which the line of sight rapidly moves occurs between the two gaze positions. Then, while the line of sight is stationary, a fixation slight movement in which the position of the line of sight varies is observed. The user recognizes the object while the gaze is stationary.

  For example, the saccade between the gaze position 201 and the gaze position 202 appears as line-of-sight data 211 indicating a rapid increase of the x-coordinate, and the involuntary eye movement at the gaze position 203 as line-of-sight data 212 indicating a small increase and decrease in x-coordinate. appear.

  FIG. 3 shows an example of the detection result of the gaze position around the gaze position. When the user is gazing at the gaze position 301, the detection result 302 of the gaze position tends to vary around the gaze position 301. Possible causes of such variations include fine eyeball shaking, involuntary eye movement, detection error due to image processing, and the like.

  Among these, it is considered that the amplitude of the variation of the line-of-sight angle due to the involuntary eye movement is about 0.8 degrees at maximum, and about 0.3 degrees on average. For example, in Non-Patent Document 1, it is reported that the detection range of the amplitude of the flick included in the fixation micromotion is 3.0 ′ to 50.0 ′ and the average amplitude is 17.7 ′. On the other hand, depending on the distance between the user and the camera 122 or the resolution of the image, the viewing angle may vary by several degrees or more.

  The eye movement detecting device of Patent Document 1 calculates the inclusive circle (minimum included circle) of the smallest area including the predetermined number of latest gaze positions, and the next detected gaze position is included in the smallest included circle. It is determined that the user's eyeball has stopped. As a result, even when the gaze position moves slightly within the minimum included circle, or even when there is a detection error of the gaze angle, it can be determined that the user is gazing at a point intentionally.

  However, as described above, not only the eye movement but also the movement of a large line of sight is mixed in the line-of-sight data, and the line-of-sight position changes even by a large line of sight movement. It is not always possible.

  FIG. 4 shows an example of the change of the sight line position on the PC screen. The horizontal and vertical axes represent the x and y coordinates of the screen, respectively. In an actual operation using a PC, the line-of-sight data contains a large amount of large line-of-sight movement, and it is difficult to set an appropriate minimum inclusion circle because the variation during the line-of-sight suspension is unclear. If the gaze determination is performed based on an inappropriate minimum inclusion circle, the accuracy of the determination result is reduced.

  Further, in an environment such as an embedded device installed on a product shelf, since the frame rate of the image of the camera 122 is relatively low, the number of samples of the gaze position is small, and the amount of variation can be determined statistically. Have difficulty.

  FIG. 5 shows an example of gaze data generated from a low rate image. The horizontal axis represents time, and the vertical axis represents the x coordinate of the gaze position. The frame rate of the image is 100 ms / frame, and the line-of-sight data 501 to the line-of-sight data 503 represent the line-of-sight position of the stationary line-of-sight.

  Since human's average gaze time is about several hundreds of ms, in the case of 100 ms / frame, only a few or so eye gaze positions are generated for one stop location. In this case, if the amount of variation is determined using about 10 line-of-sight positions, a large movement of the line of sight is mixed in, and the accuracy of the amount of variation decreases. In the example of FIG. 5, when the variation amount is determined using all of the line-of-sight data 501 to the line-of-sight data 503, a large line-of-sight movement is mixed.

  Therefore, it is conceivable to determine in advance the range 511 of the variation of the sight line position and determine that the sight line is stationary when the sight line data is included in the range 511. However, since the actual variation fluctuates according to conditions such as the distance between the user and the camera or the individual difference of the user's eyeball, if the stop determination is performed based on the range 511 determined in advance, the correct determination result is necessarily It can not always be obtained.

  FIG. 6 shows an example of the relationship between the sample number of the gaze position and the minimum included circle. In the detection result of FIG. 6A, since the number of samples is large, the minimum inclusive circle 611 for the gaze position 601 and the minimum inclusive circle 612 for the gaze position 602 can be clearly distinguished. Therefore, it is possible to accurately determine the stop with respect to the line of sight based on the minimum included circle 611 and the minimum included circle 612.

  On the other hand, in the detection result of FIG. 6B, since the number of samples is small, the line-of-sight position 614 when looking at another gaze position is mixed in the minimum inclusive circle 613 for the gaze position 603. For this reason, it is difficult to make a determination on the stay using the minimum included circle 613, and the accuracy of the determination on the stay decreases.

  FIG. 7 shows an example of the functional configuration of the gaze detection apparatus according to the embodiment. The gaze detection apparatus 701 in FIG. 7 includes a storage unit 711, a calculation unit 712, and a determination unit 713. The storage unit 711 stores line-of-sight data of the left eye of the observer who observes the object and line-of-sight data of the right eye of the observer. The calculation unit 712 and the determination unit 713 perform the gaze detection process using the gaze data stored in the storage unit 711.

  FIG. 8 is a flowchart showing an example of gaze detection processing performed by the gaze detection apparatus 701 of FIG. 7. First, the calculation unit 712 obtains an index indicating a variation in the line of sight of the observer based on the difference between the line of sight data of the left eye and the line of sight data of the right eye (step 801). Next, the determination unit 713 performs a determination on the viewer's gaze based on the index calculated by the calculation unit 712 (step 802), and determines the viewer's gaze position on the basis of the determination result of the stay determination. (Step 803).

  According to the line-of-sight detection device 701 of FIG. 7, the line of sight of the observer observing the object can be detected with high accuracy.

  FIG. 9 shows a specific example of the visual axis detection device 701 of FIG. The gaze detection apparatus 701 in FIG. 9 includes a storage unit 711, a calculation unit 712, a determination unit 713, a detection unit 911, and an output unit 912. The camera 901 corresponds to, for example, the camera 122 in FIG. 1, captures an image of the observer's face, and outputs an image to the gaze detection apparatus 701. The detection unit 911 detects the line-of-sight data 921 of the left eye and the right eye of the observer from each of the images of a plurality of times included in the video output by the camera 122 and stores the data in the storage unit 711. The image at each time is sometimes called a frame.

  For example, the line-of-sight data 921 indicates the line-of-sight position of the left eye and the right eye in the plane facing the face of the observer. In the case of the display device 101 of FIG. 1, the screen 102 corresponds to a plane facing the face of the observer. Further, when the camera 901 is installed on the product shelf, a virtual plane covering the front of the product shelf corresponds to a plane facing the face of the observer.

  FIG. 10 shows an example of gaze detection by the gaze sensor 103 of FIG. The LED 121 emits an infrared ray 1002 to the face 1001 of the observer who is viewing the screen 102, and the camera 122 captures the face 1001. In this case, the detection unit 911 calculates the eye gaze position of the observer based on the positional relationship between the coordinates of the pupil shown in the image and the coordinates of the corneal reflection by the corneal reflection method.

  When the observer is looking at the gaze sensor 103, the difference between the coordinates of the pupil (PupX, PupY) and the coordinates of the corneal reflection (PrkX, PrkY) is 0, and when looking away from the gaze sensor 103 , The difference of their coordinates occurs. Therefore, in the corneal reflection method, the direction of the sight line is detected from the difference between the coordinates (PupX, PupY) and the coordinates (PrkX, PrkY). The line-of-sight position of one eye detected by the corneal reflection method from the i-th frame is represented by coordinates (Xi, Yi) on the screen 102, and can be obtained by the following equation.

Xi = A * (PupX-PrkX) + W / 2 (1)
Yi = H + B * (PupY-PrkY) (2)

  In the equation (1), W represents the number of pixels in the horizontal direction of the screen 102, and H in the equation (2) represents the number of pixels in the vertical direction of the screen 102. The coefficient A and the coefficient B are obtained in advance by calibration. The gaze position (Xi, Yi) can also be calculated from the gaze vector.

  The calculation unit 712 calculates an index 923 indicating the variation of the line of sight based on the difference between the line-of-sight data 921 for the left eye and the line-of-sight data 921 for the right eye, and stores the index 923 in the storage unit 711. The determination unit 713 calculates the threshold 924 for determination of stay based on the index 923, and stores the threshold 924 in the storage unit 711. Further, the determination unit 713 calculates the variation 922 of the line-of-sight data at each time from the line-of-sight data 921 of the left eye and the right eye detected at each of a plurality of times within the predetermined period, and stores the variation 922 in the storage unit 711.

  Next, the determination unit 713 performs the determination of suspension with respect to the variation 922, and when the variation 922 at each time within a predetermined period is smaller than the threshold 924, determines that the line of sight of the observer is stationary. Then, the determination unit 713 determines the gaze position 925 of the observer based on the determination result of the stop determination, and the output unit 912 outputs the gaze position 925.

  By calculating the difference between the line-of-sight data 921 for the left eye and the line-of-sight data 921 for the right eye, the saccades included in the line-of-sight data 921 for each eye are canceled, and it becomes possible to extract variations representing fixation eye movement. When the observer is looking at an object such as the screen 102, since both eyes gaze at the same position, the gaze positions of both eyes are considered to be the same except for the difference in variation. At this time, the difference in the left-right direction (horizontal direction) of the eye-gaze data 921 of both eyes is affected by congestion, but the difference in the vertical direction is not affected by congestion.

  FIG. 11 shows an example of congestion. Convergence refers to the movement of the eye when the two eyes simultaneously turn inward as the observer gazes at the object. As the object moves back and forth, convergence occurs as the subject is adjusted in focus. The convergence causes the eyeball to move in the lateral direction but does not move in the vertical direction, so the vertical difference of the gaze data 921 is not affected by the convergence.

  FIG. 12 shows an example of the difference in the horizontal direction of the eye-gaze data 921 of both eyes. The horizontal axis represents the number of frames, and the vertical axis represents the x-coordinate of the gaze position. The graph 1201 shows the change of the x coordinate of the gaze position of the left eye, the graph 1202 shows the change of the x coordinate of the gaze position of the right eye, and the graph 1203 shows the difference between the graph 1201 and the graph 1202. It can be seen from the graph 1203 that the horizontal difference in the gaze position changes significantly due to congestion.

  FIG. 13 shows an example of the difference in the vertical direction of the eye-gaze data 921 for both eyes. The horizontal axis represents the number of frames, and the vertical axis represents the y-coordinate of the gaze position. The graph 1301 shows the change of y-coordinate of the line-of-sight position of the left eye, the graph 1302 shows the change of y-coordinate of the line-of-sight position of the right eye, and the graph 1303 shows the difference between the graph 1301 and the graph 1302. It can be seen from the graph 1303 that the change in the vertical difference of the gaze position is smaller than the change in the horizontal difference and is not affected by congestion.

  Therefore, the calculation unit 712 calculates the difference between the vertical line-of-sight position indicated by the left-eye line-of-sight data 921 and the vertical line-of-sight position indicated by the right-eye line-of-sight data 921 and calculates the index 923 based on the difference. . For example, the change ΔLy in the vertical direction of the line-of-sight position of the left eye and the change ΔRy in the vertical direction of the line-of-sight position of the right eye are expressed by the following equations.

ΔLy = ΔSLy + ΔVLy (3)
ΔRy = ΔSRy + ΔVRy (4)

  In equation (3), ΔSLy represents a large vertical movement due to the left eye saccade, and ΔVLy represents the vertical dispersion of the left eye gaze position. In the equation (4), ΔSRy represents a large vertical movement due to the saccade in the right eye, and ΔVRy represents a vertical dispersion of the position of the right eye gaze. Since the left eye and the right eye look at the same height, ΔSLy and ΔSRy can be regarded as almost the same value. Therefore, the difference between ΔLy and ΔRy is given by the following equation.

ΔLy−ΔRy = (ΔSLy + ΔVLy) − (ΔSRy + ΔVRy)
= ΔVLy-ΔVRy (5)

  From equation (5), the variance of the difference Ly-Ry between the y-coordinate Ly of the line-of-sight position of the left eye and the y-coordinate Ry of the line-of-sight position of the right eye is the sum of the variance with Ly and the variance with Ry. It can be seen that it is represented. Further, the dispersion due to the variation of Ly and the dispersion due to the variation of Ry can be regarded as substantially the same value. Therefore, it is possible to use half of the Ly-Ry dispersion as the dispersion associated with the Ly dispersion or the Ry dispersion. Furthermore, the variance of the horizontal variation of the gaze position can also be regarded as substantially the same value as the variance of the vertical variation of the gaze position.

  FIG. 14 shows an example of the change in the gaze position of both eyes. The line-of-sight position 1401 and the line-of-sight position 1402 of the left eye of the observer who is viewing the screen 102 move on the screen 102 according to the position of the image to be gazed.

  FIG. 15 shows an example of line-of-sight data 921 corresponding to the change of the line-of-sight position in FIG. The horizontal axis represents the number of frames, and the vertical axis represents the y-coordinate of the gaze position. The graph 1501 shows the change of y-coordinate of the line-of-sight position of the left eye, the graph 1502 shows the change of y-coordinate of the line-of-sight position of the right eye, and the graph 1503 shows the difference between the graph 1501 and the graph 1502.

  By collecting the log of the gaze position 1401 and the gaze position 1402 in a fixed period and calculating the difference of the y coordinate of those gaze positions, even when the gaze moves frequently, the variation amount of the gaze position is always It becomes possible to grasp. As a result, an appropriate threshold value for stopping determination can be determined based on the variation amount of the gaze position, and the accuracy of the stopping determination can be improved. For example, the variance or standard deviation of the y-coordinate difference of the gaze position detected in a fixed period can be used as the index 923 indicating the dispersion of the gaze.

  The difference between the line-of-sight data 921 for the left eye and the line-of-sight data 921 for the right eye need not necessarily be the difference in the vertical direction of the eye position, but may be the difference in eye position in a predetermined direction intersecting the horizontal direction. . For example, if the left-right straight line connecting the left eye and right eye is not parallel to the x-axis on the screen 102, the y-axis on the screen 102 is not perpendicular to the left-right straight but a predetermined angle At the intersection. In this case, the difference between the y-coordinates of the line-of-sight positions of both eyes represents the difference between the line-of-sight positions in a predetermined direction intersecting the horizontal direction.

  FIG. 16 is a flowchart showing a specific example of the gaze detection process performed by the gaze detection apparatus 701 shown in FIG. First, the detection unit 911 detects the visual line data 921 of the left eye and the right eye of the observer from the image output by the camera 122 (step 1601).

  For example, from the image of the i-th frame, the x coordinate LXi of the eye position of the left eye, the y coordinate LYi of the eye position of the left eye, the x coordinate RXi of the eye position of the right eye, and the y coordinate RYi of the eye position of the right eye Ru. LXi, LYi, RXi, and RYi may be coordinates of the gaze position on the screen 102 of FIG. 1, and coordinates of the gaze position on a virtual plane set at a position where an object such as a commodity is present It may be

  Next, the detection unit 911 checks whether the observer to be monitored has switched to another observer (step 1602). For example, when a new observer appears after the observer disappears from the image, the detection unit 911 can determine that the observer has switched.

  When the observer is switched (YES in step 1602), the detection unit 911 resets the detection processing by deleting the line-of-sight data 921 accumulated in the storage unit 711 (step 1603). Then, the detection unit 911 checks whether the line-of-sight data 921 of the latest N (N is an integer of 2 or more) frames is accumulated in the storage unit 711 (step 1604).

  If the line-of-sight data 921 of the N most recent frames is not accumulated (step 1604, NO), the determination unit 713 uses the default threshold as the threshold TH for determination of stopping (step 1606).

  Next, the determination unit 713 calculates the variation 922 of the gaze data at each time from the gaze data 921 of the left eye and the right eye detected at each of a plurality of times in a predetermined period, and performs the standstill determination for the variation 922 ( Step 1607).

  For example, using the variance of coordinates of the gaze position as the variation 922 of the gaze data 921 detected from the i-th frame, and using the nearest M (M is an integer of 2 or more) frames as the predetermined period, the determination unit In 713, Vi representing the variation 922 can be calculated by the following equation.

Vi = ((LXi + RXi) / 2-Avex) ²
+ ((LYi + RYi) / 2-Avey) ² (6)

  However, it is assumed that M <N and i ≧ M. For example, when the frame rate is 100 ms / frame, a value of about 5 to 20 may be used as N, and a value of 2 or more and less than N may be used as M.

  In equation (6), (LXi + RXi) / 2 represents the average value of LXi and RXi, and (LYi + RYi) / 2 represents the average value of LYi and RYi. Avex in Equation (7) represents a value obtained by averaging the average values of LXj and RXj over M frames in the range of j = i−M + 1 to i. Also, Avey in Equation (8) represents a value obtained by averaging the average values of LYj and RYj over the same M frames.

  In the stop determination, the determination unit 713 compares M variations Vj in the range of j = i−M + 1 to i with the threshold TH, and if Vj ≦ TH for all j (step 1607, YES), the sight line is It is determined that the mobile terminal is staying (step 1608). In this case, the determination unit 713 calculates the gaze position 925 from the coordinates of the gaze position of the M frames. As the gaze position 925, statistical values such as an average value or a median value of M coordinates can be used. Then, the visual axis detection device 701 updates i by i = i + 1, and repeats the processing of step 1601 and subsequent steps for the image of the next frame.

  On the other hand, if Vj> TH for one or more j (step 1607, NO), the determination unit 713 determines that the sight line is not stopped, and the sight line detection device 701 performs step 1601 for the next frame image. Repeat the subsequent processing.

  If i <M in step 1607, the determination unit 713 omits the stop determination and repeats the processing after step 1601. If the observer is not switched at step 1602 (step 1602, NO), the detection unit 911 performs the process of step 1604.

  If the line-of-sight data 921 of the N most recent frames is accumulated (step 1604, YES), the calculation unit 712 calculates the index 923 using the accumulated line-of-sight data 921 and the determination unit 713 determines the index 923 The threshold value 924 is calculated from (step 1605).

  For example, the calculation unit 712 can use the moving average of the variance of the difference between LYi and RYi in N frames as the index 923. In this case, the calculation unit 712 calculates div representing the index 923 according to the following equation.

  In the equation (10), ave represents an average value of RYi-LYi in N frames in the range of j = i−N + 1 to i, and div in the equation (9) represents RYi−LYi in the same N frames. Represents the variance. In this case, using the positive real number K, the determination unit 713 can calculate TH representing the threshold 924 according to the following equation.

TH = K * (div / 2) (11)

  Then, the determination unit 713 performs the process of step 1607 and subsequent steps using the threshold value TH of equation (11). Once the line-of-sight data 921 of N frames has been accumulated in step 1604, the threshold value TH calculated in step 1605 is used unless the observer switches.

  According to such line-of-sight detection processing, even when the frame rate of the video is low, it is possible to determine an appropriate threshold for the stop determination based on the difference between the eye-gaze data 921 of both eyes. Accuracy is improved.

  By the way, in the gaze detection process of FIG. 16, since the moving average of the variance of the difference between LYi and RYi is used as the index 923, the period of the N most recent frames is used as the accumulation period for accumulating the gaze data 921. . When M <N, the period of the latest M frames used to calculate the variation Vi is included in the accumulation period. However, the accumulation period does not necessarily have to include the period of M frames, and may be an earlier period.

  Further, the determination unit 713 can use the standard deviation as the variation 922 instead of the variance of the coordinates of the sight line position. In this case, the calculation unit 712 calculates the index 923 using the standard deviation instead of the variance of the difference between LYi and RYi in N frames.

  The configuration of the visual axis detection device 701 shown in FIGS. 7 and 9 is merely an example, and some components may be omitted or changed depending on the application or conditions of the visual axis detection device 701. For example, in the visual axis detection device 701 of FIG. 9, when the visual axis data 921 is detected by an external device, the detection unit 911 can be omitted.

  The flowcharts illustrated in FIGS. 8 and 16 are merely examples, and some processes may be omitted or changed in accordance with the configuration or conditions of the gaze detection apparatus 701. For example, in the gaze detection process of FIG. 16, when the observer does not switch, the processes of steps 1602 and 1603 can be omitted.

  The line-of-sight sensor 103 shown in FIG. 1 and the line-of-sight detection shown in FIG. 10 are merely examples, and the configuration and position of the line-of-sight sensor 103 change according to the application or conditions of the line-of-sight detection device 701. When the camera 122 shoots the observer using visible light, the LED 121 can be omitted.

  The line-of-sight data and the line-of-sight position shown in FIGS. 2 to 6 and FIGS. 12 to 15 are merely examples, and the line-of-sight data and the line-of-sight position change according to the image to be photographed. The congestion shown in FIG. 11 is only an example, and the degree of congestion changes according to the observer.

  The equations (1) to (10) are merely examples, and the gaze detection apparatus 701 may perform the gaze detection process using another calculation equation.

  FIG. 17 shows a configuration example of an information processing apparatus (computer) used as the gaze detection apparatus 701 in FIG. 7 and FIG. The information processing apparatus of FIG. 17 includes a central processing unit (CPU) 1701, a memory 1702, an input device 1703, an output device 1704, an auxiliary storage device 1705, a medium drive device 1706, and a network connection device 1707. These components are connected to one another by a bus 1708. The camera 901 in FIG. 9 may be connected to the bus 1708.

  The memory 1702 is, for example, a semiconductor memory such as a read only memory (ROM), a random access memory (RAM), or a flash memory, and stores programs and data used for processing. The memory 1702 can be used as the storage unit 711 in FIGS. 7 and 9.

  The CPU 1701 (processor) operates as the calculation unit 712 and the determination unit 713 in FIGS. 7 and 9 by executing a program using the memory 1702, for example. The CPU 1701 operates as the detection unit 911 in FIG. 9 by executing a program using the memory 1702.

  The input device 1703 is, for example, a touch panel, a keyboard, a pointing device, or the like, and is used to input an instruction or information from a user or an operator. The output device 1704 is, for example, a display device, a printer, a speaker, etc., and is used to inquire the user or the operator and to output the processing result. The output device 1704 can be used as the output unit 912 in FIG.

  The auxiliary storage device 1705 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device or the like. The auxiliary storage device 1705 may be a hard disk drive. The information processing apparatus can store programs and data in the auxiliary storage device 1705, load them into the memory 1702, and use them. The auxiliary storage device 1705 can be used as the storage unit 711 in FIGS. 7 and 9.

  The medium drive device 1706 drives a portable recording medium 1709 and accesses the recorded contents. The portable recording medium 1709 is a memory device, a flexible disk, an optical disk, a magneto-optical disk or the like. The portable recording medium 1709 may be a Compact Disk Read Only Memory (CD-ROM), a Digital Versatile Disk (DVD), or a Universal Serial Bus (USB) memory. The user or the operator can store programs and data in this portable recording medium 1709, load them into the memory 1702, and use them.

  Thus, computer readable recording media for storing programs and data used for processing include physical (non-temporary) such as the memory 1702, auxiliary storage device 1705, and portable recording medium 1709. A recording medium is included.

  A network connection device 1707 is a communication interface circuit connected to a communication network such as a Local Area Network (LAN), the Internet, etc., and performing data conversion involved in communication. The information processing device can receive programs and data from an external device via the network connection device 1707, load them into the memory 1702, and use them. The network connection device 1707 can be used as the output unit 912 of FIG.

  Note that the information processing apparatus does not have to include all the components shown in FIG. 17, and some of the components may be omitted depending on the application or conditions. For example, when an interface with a user or an operator is unnecessary, the input device 1703 and the output device 1704 may be omitted. When the information processing apparatus does not use the portable recording medium 1709 or the communication network, the medium driving device 1706 or the network connection device 1707 may be omitted.

  While the disclosed embodiments and their advantages have been described in detail, those skilled in the art can make various changes, additions, and omissions without departing from the scope of the present invention as set forth in the claims. I will.

The following appendices will be further disclosed regarding the embodiment described with reference to FIGS. 1 to 17.
(Supplementary Note 1)
Based on the difference between the line-of-sight data of the left eye of the observer who observes the object and the line-of-sight data of the right eye of the observer, an index indicating variation in the line of sight of the observer is determined.
It is determined based on the index that the observer is staying in line of sight,
The gaze position of the observer is determined based on the determination result of the stop determination.
A gaze detection program for causing a computer to execute processing.
(Supplementary Note 2)
The line of sight according to appendix 1, wherein the difference represents a difference between the line-of-sight position of the left eye and the line-of-sight position of the right eye in a predetermined direction intersecting with the horizontal direction in a plane facing the face of the observer. Detection program.
(Supplementary Note 3)
The sight line detection program according to claim 2, wherein the predetermined direction is a direction perpendicular to the left and right direction.
(Supplementary Note 4)
The computer calculates a threshold value for determining the stay based on the index, and variation in line-of-sight data at each of the plurality of times from the line-of-sight data of the left eye and the right eye detected at each of a plurality of times in a predetermined period And determining that the line of sight of the observer is stationary if the variation of the line-of-sight data at each of the plurality of times is smaller than the threshold value. Gaze detection program described.
(Supplementary Note 5)
The index is a variance or a standard deviation of the differences detected at each of a plurality of times in a first period including the predetermined period or in a second period before the predetermined period. Gaze detection program.
(Supplementary Note 6)
A storage unit that stores line-of-sight data of the left eye of the observer observing the object and line-of-sight data of the right eye of the observer;
A calculation unit for obtaining an index indicating a variation in the line of sight of the observer based on a difference between the line of sight data of the left eye and the line of sight data of the right eye;
A determination unit which performs stop determination on the line of sight of the observer based on the index, and determines the line-of-sight position of the observer on the basis of the determination result of the stop determination;
A gaze detection apparatus comprising:
(Appendix 7)
The line of sight according to claim 6, wherein the difference represents a difference between the line-of-sight position of the left eye and the line-of-sight position of the right eye in a predetermined direction intersecting with the horizontal direction in a plane facing the face of the observer. Detection device.
(Supplementary Note 8)
The visual axis detection device according to claim 7, wherein the predetermined direction is a direction perpendicular to the left and right direction.
(Appendix 9)
The determination unit calculates a threshold of the determination for stopping based on the index, and from line-of-sight data of the left eye and the right eye detected at each of a plurality of times within a predetermined period, the line-of-sight data at each of the plurality of times The variation is calculated, and when the variation in line-of-sight data at each of the plurality of times is smaller than the threshold value, it is determined that the line of sight of the observer is stationary. The gaze detection apparatus according to claim 1.
(Supplementary Note 10)
The index represents a variance or a standard deviation of the differences detected at each of a plurality of times in a first period including the predetermined period or in a second period before the predetermined period. Gaze detection device.
(Supplementary Note 11)
The computer is
Based on the difference between the line-of-sight data of the left eye of the observer who observes the object and the line-of-sight data of the right eye of the observer, an index indicating variation in the line of sight of the observer is determined.
It is determined based on the index that the observer is staying in line of sight,
The gaze position of the observer is determined based on the determination result of the stop determination.
A gaze detection method characterized by
(Supplementary Note 12)
The line of sight according to appendix 11, wherein the difference represents a difference between the line-of-sight position of the left eye and the line-of-sight position of the right eye in a predetermined direction intersecting with the horizontal direction in a plane facing the face of the observer. Detection method.
(Supplementary Note 13)
The gaze detection method according to claim 12, wherein the predetermined direction is a direction perpendicular to the left and right direction.
(Supplementary Note 14)
The computer calculates a threshold value for determining the stay based on the index, and variation in line-of-sight data at each of the plurality of times from the line-of-sight data of the left eye and the right eye detected at each of a plurality of times in a predetermined period And determining that the line of sight of the observer is stationary if the variation of the line of sight data at each of the plurality of times is smaller than the threshold value. The gaze detection method described.
(Supplementary Note 15)
The index is a variance or a standard deviation of the differences detected at each of a plurality of times in a first period including the predetermined period or in a second period before the predetermined period. Gaze detection method.

101 Display Device 102 Screen 103 Line-of-Sight Sensor 111-113 Image 121 LED
122, 901 cameras 201 to 203, 301, 601 to 603 gaze positions 211, 212, 501 to 503, 921 gaze data 302 detection results 511 range 611 to 613 minimum inclusive circle 614, 925, 1401, 1402 gaze position 701 gaze detection device 711 storage unit 712 calculation unit 713 determination unit 911 detection unit 912 output unit 922 variation 923 index 924 threshold 1001 face 1002 infrared rays 1201 to 1203, 1301 to 1303, 1501 to 1503 graph 1701 CPU
1702 Memory 1703 Input Device 1704 Output Device 1705 Auxiliary Storage Device 1706 Media Drive Device 1707 Network Connection Device 1708 Bus 1709 Portable Storage Media

Claims (7)

Based on the difference between the line-of-sight data of the left eye of the observer who observes the object and the line-of-sight data of the right eye of the observer, an index indicating variation in the line of sight of the observer is determined.
It is determined based on the index that the observer is staying in line of sight,
The gaze position of the observer is determined based on the determination result of the stop determination.
A gaze detection program for causing a computer to execute processing.   The difference according to claim 1, wherein the difference represents a difference between the line-of-sight position of the left eye and the line-of-sight position of the right eye in a predetermined direction intersecting with the horizontal direction in a plane facing the face of the observer. Gaze detection program.   The eye gaze detection program according to claim 2, wherein the predetermined direction is a direction perpendicular to the left and right direction.   The computer calculates a threshold value for determining the stay based on the index, and variation in line-of-sight data at each of the plurality of times from the line-of-sight data of the left eye and the right eye detected at each of a plurality of times in a predetermined period 4. The method according to any one of claims 1 to 3, wherein when the variation in line-of-sight data at each of the plurality of times is smaller than the threshold value, it is determined that the line of sight of the observer is stationary. The gaze detection program described in.   The index is a variance or standard deviation of the differences detected at each of a plurality of times in a first period including the predetermined period or a second period before the predetermined period. Gaze detection program described. A storage unit that stores line-of-sight data of the left eye of the observer observing the object and line-of-sight data of the right eye of the observer;
A calculation unit for obtaining an index indicating a variation in the line of sight of the observer based on a difference between the line of sight data of the left eye and the line of sight data of the right eye;
A determination unit which performs stop determination on the line of sight of the observer based on the index, and determines the line-of-sight position of the observer on the basis of the determination result of the stop determination;
A gaze detection apparatus comprising: The computer is
Based on the difference between the line-of-sight data of the left eye of the observer who observes the object and the line-of-sight data of the right eye of the observer, an index indicating variation in the line of sight of the observer is determined.
It is determined based on the index that the observer is staying in line of sight,
The gaze position of the observer is determined based on the determination result of the stop determination.
A gaze detection method characterized by