JP2012065781A - Line-of-sight detection device, and correction coefficient determination program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To carry out line-of-sight calibration without imposing a burden on a user.SOLUTION: The line-of-sight detection device 100 determines a temporary correction coefficient every time the user selects an object. The line-of-sight detection device 100 determines a correction coefficient by putting importance on the temporary correction coefficient determined when the gaze level of the user is high, based on the relation between the selected object and the other objects, out of a plurality of correction coefficients. The line-of-sight detection device 100 thus utilizes user operation to determine the correction coefficient without the user being aware of the line-of-sight calibration.

Description

  The present invention relates to a line-of-sight detection device and the like.

  As a technique for detecting a person's line of sight, there is a technique for irradiating a user's eyeball with near infrared rays or the like and detecting the line of sight from the center of the corneal curvature obtained from the position of the reflected image on the corneal surface and the center of the pupil. By using this technique, it is possible to determine the position on the display where the user is gazing. For example, the user can select an object by gazing at the object on the display without operating the keyboard.

  However, there may be an error between the position on the display determined by the line of sight detected using the reflection image on the corneal surface and the position where the user is actually gazing. If an error occurs, an unintended object may be selected when the user tries to select a predetermined object on the display by his / her line of sight. Factors that cause errors include the refractive index of light on the corneal surface, the shape of the eyeball, and the deviation of the reference point with respect to the center of the eyeball.

  In order to correct the above error, line-of-sight calibration is performed. In this line-of-sight calibration, a correction coefficient that fills an error between the position on the display obtained from the reflection image on the cornea surface and the actual position on the display is calculated. By using this correction coefficient, there is no error, and the user can comfortably select an object on the display.

  Here, an example of conventional line-of-sight calibration will be described. FIG. 16 is a diagram for explaining an example of conventional line-of-sight calibration. As shown in FIG. 16, in the related art, the display 10 is divided into a plurality of regions, and the pattern 11 is displayed in the regions. This pattern 11 moves in the order of the arrows. The user follows the pattern 11 displayed on the display in order. In the prior art, for each area where the pattern 11 is displayed, an error between the position on the display 10 determined from the line of sight and the position of the area is obtained, and a correction coefficient is calculated so as to reduce this error.

Japanese Patent No. 4385197 JP 2006-285715 A

  However, the above-described conventional technique has a problem that the burden on the user is large when the line-of-sight calibration is executed.

  For example, when the line-of-sight calibration is executed by the method shown in FIG. 16, the user continues to watch the pattern 11 displayed on the display 10, which takes time, which is a heavy burden on the user.

  The disclosed technique has been made in view of the above, and an object thereof is to provide a gaze detection apparatus and a correction coefficient calculation program capable of performing gaze calibration without imposing a burden on the user.

  In the disclosed gaze detection device, when an object displayed on the screen is selected by the user, the user performs gaze input based on the relationship between the selected object and another object displayed on the screen. A correction coefficient calculation unit for calculating a correction coefficient for correcting the error in the case. Further, the disclosed gaze detection device detects coordinates on the screen on which the user is gazing, based on the correction coefficient and the feature amount related to the eye obtained from an image including the user's eye as a subject. It is a requirement to have a coordinate position detection unit.

  According to the disclosed gaze detection apparatus, gaze calibration can be executed without imposing a burden on the user.

FIG. 1 is a diagram illustrating a functional configuration of the visual line detection device according to the present embodiment. FIG. 2 is a diagram illustrating an example of a data structure of the feature amount management table. FIG. 3 is a diagram illustrating an example of a data structure of personal parameters. FIG. 4 is a diagram illustrating an example of the data structure of the object coordinate data. FIG. 5 is a diagram illustrating an example of the data structure of the correction coefficient management table. FIG. 6 is a diagram for explaining the feature amount of the eye. FIG. 7 is a diagram (1) illustrating a user operation example. FIG. 8 is a diagram (2) illustrating a user operation example. FIG. 9 is a diagram (1) for explaining the relationship between the density of objects and the degree of gaze of the user. FIG. 10 is a diagram (2) for explaining the relationship between the density of objects and the degree of gaze of the user. FIG. 11 is a diagram for explaining the behavior when the mouse pointer is moved to a target object. FIG. 12 is a diagram illustrating a relationship with a moving speed when the mouse pointer is moved to an object to be selected. FIG. 13 is a diagram illustrating the relationship between the line of sight θ and the coordinates on the screen. FIG. 14 is a flowchart illustrating a processing procedure when the line-of-sight detection apparatus calculates a correction coefficient. FIG. 15 is a diagram illustrating a hardware configuration of a computer constituting the visual line detection device according to the embodiment. FIG. 16 is a diagram for explaining an example of conventional line-of-sight calibration.

  Embodiments of a line-of-sight detection device and a correction coefficient calculation program disclosed in the present application will be described below in detail with reference to the drawings. Note that the present invention is not limited to the embodiments.

  A configuration of the line-of-sight detection apparatus according to the present embodiment will be described. FIG. 1 is a diagram illustrating a functional configuration of the visual line detection device according to the present embodiment. As illustrated in FIG. 1, the line-of-sight detection apparatus 100 includes a storage unit 101, a camera 102, an eye detection unit 103, a feature amount extraction unit 104, an input unit 105, a screen display unit 106, and an operation information acquisition unit 107. The line-of-sight detection apparatus 100 includes a selected object determination unit 108, an importance level determination unit 109, a correction coefficient calculation unit 110, a gaze position detection unit 111, and a screen control unit 112.

  The storage unit 101 is a storage unit that stores various data for executing the user's line-of-sight calibration. As shown in FIG. 1, the storage unit 101 stores a feature amount management table 101a, personal parameters 101b, object coordinate data 101c, a correction coefficient management table 101d, and correction coefficient data 101e.

  Among these, the feature amount management table 101a is a table that holds feature amounts. This feature amount is a value specified based on the pupil center of the eye and the cornea reflection image. The description of the pupil center of the eye and the cornea reflection image will be given later. FIG. 2 is a diagram illustrating an example of a data structure of the feature amount management table. As shown in FIG. 2, the feature amount management table 101a stores feature amounts in order.

  The personal parameter 101b holds the eyeball radius for each user. FIG. 3 is a diagram illustrating an example of a data structure of personal parameters. As shown in FIG. 3, the personal parameter 101b holds user identification information for identifying a user and an eyeball radius in association with each other.

  The object coordinate data 101c holds the coordinates of the object on the screen. FIG. 4 is a diagram illustrating an example of the data structure of the object coordinate data. As shown in FIG. 4, the object coordinate data 101c holds object identification information for identifying an object and coordinates in association with each other. These coordinates are the coordinates of the object on the screen.

  The correction coefficient management table 101d is a table that holds correction coefficient candidates. In the following description, a correction coefficient candidate is referred to as a temporary correction coefficient. FIG. 5 is a diagram illustrating an example of the data structure of the correction coefficient management table. As shown in FIG. 5, the correction coefficient management table 101d holds a temporary correction coefficient and an importance flag in association with each other. Here, the importance flag is a flag for identifying whether or not the corresponding temporary correction coefficient is important.

  The correction coefficient data 101e is data that fills an error between the position on the display obtained from the reflection image on the cornea surface and the actual position on the display. The gaze position detection unit 111 described later detects the gaze position of the user using the correction coefficient data 101e.

  Returning to the description of FIG. The camera 102 is a device that captures a user's face. When the camera 102 captures a user's face, it is assumed that visible light or infrared light is irradiated to the user's eyes. The camera 102 outputs the captured image data to the eye detection unit 103 and the gaze position detection unit 111.

  The camera 102 outputs image data to the eye detection unit 103 in accordance with the control signal output from the input unit 105. This control signal is a signal output from the input unit 105 at the timing when the user clicks the mouse or touches the touch panel. The camera 102 always outputs image data to the gaze position detection unit 111.

  The eye detection unit 103 is a processing unit that detects an eye region included in the image data. After detecting the eye area, the eye detection unit 103 outputs image data of the eye area to the feature amount extraction unit 104.

  An example of processing in which the eye detection unit 103 detects an eye region will be described. For example, the eye detection unit 103 holds a pattern including the features of a person's eyes, and determines the region where the eye exists from the image data by matching this pattern with the image data. Alternatively, the eye detection unit 103 may extract eye feature points such as the eyes and the corners of the eye from the image data, and determine the eye region based on the extracted feature point positions.

  The feature amount extraction unit 104 is a processing unit that extracts eye feature amounts from image data of the eye region. The feature amount extraction unit 104 registers the extracted feature amount in the feature amount management table 101a.

  For example, the feature amount extraction unit 104 detects the pupil center of the eye and the cornea reflection image from the image data, and extracts the distance between the pupil center and the cornea reflection image as the eye feature amount. Here, the corneal reflection image is an image generated when light enters the eyes of the user. FIG. 6 is a diagram for explaining the feature amount of the eye. As shown in FIG. 6, a pupil center 21 and a cornea reflection image 22 exist in the user's eye 20. For example, the eye feature amount corresponds to the distance 23 from the position of the pupil center 21 to the position of the cornea reflection image 22.

  Note that the feature value extracted by the feature value extraction unit 104 corresponds to the timing when the user clicks the mouse or touches the touch panel.

  The input unit 105 is an input device used by a user to input various types of information to the line-of-sight detection device 100. For example, the input unit 105 corresponds to a keyboard, a mouse, or the like. The user operates the input unit 105 to select an object displayed on the screen display unit 106. Here, the object corresponds to, for example, an icon, a folder, or a window button. Further, when a user logs in, the input unit 105 is operated to input an ID (Identification), a password, and the like. Information operated by the user is output from the input unit 105 to the operation information acquisition unit 107 and the correction coefficient calculation unit 110.

  For example, the input unit 105 outputs a control signal to the camera 102 in accordance with the timing when the user clicks the mouse or the timing when the user touches the touch panel.

  FIG. 7 is a diagram (1) illustrating a user operation example. In the example shown in FIG. 7, the user 30 operates the mouse or the like to move the mouse pointer 31 and clicks the window button 32. As described above, when the button 32 is clicked, it is considered that the user 30 is gazing at the button 32.

  The screen display unit 106 corresponds to a display device such as a display or a touch panel. For example, the user selects an object by touching the object displayed on the screen display unit 106. Information operated by the user is output from the screen display unit 106 to the operation information acquisition unit 107.

  FIG. 8 is a diagram (2) illustrating a user operation example. In the example shown in FIG. 8, the user 30 touches the icon 33 on the touch panel with the user's hand 30a. Thus, when the icon 33 is touched, it is considered that the user 30 is gazing at the icon 33.

  The operation information acquisition unit 107 acquires user operation data from the input unit 105 and the screen display unit 106, and outputs the acquired operation data to the selected object determination unit 108. This operation data includes coordinates on the screen clicked by the user with the mouse pointer or coordinates on the screen touched by the user. The operation data includes the moving speed of the mouse pointer from when the mouse pointer is clicked to a predetermined time. The unit of the moving speed of the mouse pointer is, for example, the number of pixels / second.

  The selected object determination unit 108 is a processing unit that determines the object selected by the user based on the operation data. For example, the selected object determination unit 108 compares the clicked coordinates with the object coordinate data 101c to determine the object selected by the user. In the following description, an object selected by the user is referred to as a selected object.

  For example, if the clicked coordinates are (X1, Y1), the selected object determination unit 108 determines the object with the object identification number “1001” as the selected object. Similarly, when the touched coordinates are (X1, Y1), the selected object determination unit 108 determines the object with the object identification number “1001” as the selected object. The selected object determination unit 108 outputs the identification information of the selected object to the importance level determination unit 109. Further, the selected object determination unit 108 outputs the operation data to the importance level determination unit 109.

  The importance determination unit 109 is a processing unit that determines the importance of the temporary correction coefficient based on the relationship between the selected object and other objects. The importance level determination unit 109 is an example of a gaze determination unit. Here, the provisional correction coefficient is a coefficient that corrects an error between the selected object and the coordinates on the screen obtained from the line of sight when the user selects the selected object. The provisional correction coefficient is calculated by the correction coefficient calculation unit 110 described later every time the user selects the selected object. The importance level determination unit 109 determines the importance level of the temporary correction coefficient based on “distance between objects”, “density of objects”, and “moving speed of the mouse pointer”. The importance level determination unit 109 outputs the determination result to the correction coefficient calculation unit 110.

  First, a process in which the importance level determination unit 109 determines the importance level based on the distance between objects will be described. If the distance between the selected object and another object is small, it is considered that the user is gazing at the selected object in a narrower area, and the degree of gazing is high. For this reason, when the distance between objects is small, the importance of the temporary correction coefficient calculated | required at this time is high. On the other hand, if the distance between the selected object and another object is large, it is considered that the user is gazing at the selected object in a wide area, and the degree of gazing is low. For this reason, when the distance between objects is large, the importance of the temporary correction coefficient calculated | required at this time is low.

  Specifically, the importance determination unit 109 determines the coordinates of the selected object by comparing the identification information of the selected object with the object coordinate data 101c. Then, the importance level determination unit 109 compares the coordinates of the selected object with the object coordinate data 101c, and calculates the distance between the coordinates of the selected object and the coordinates of the object closest to this coordinate. The importance level determination unit 109 determines that the importance level of the temporary correction coefficient is high when the calculated distance is less than the predetermined distance. On the other hand, the importance level determination unit 109 determines that the importance level of the temporary correction coefficient is low when the calculated distance is equal to or greater than the predetermined distance.

  A process in which the weight determination unit 109 determines the importance based on the density of the object will be described. FIG. 9 and FIG. 10 are diagrams for explaining the relationship between the object density and the user's gaze condition. As shown in FIG. 9, when other objects 35 to 41 are dense around the selected object 34, it is considered that the user is gazing at the narrow area 34 a including the selected object, Is expensive. For this reason, when the density of the object is high, the importance of the temporary correction coefficient obtained at this time is high. On the other hand, as shown in FIG. 10, when other objects 43 are not dense around the selected object 42, the user is considered to be viewing a wide area 42a including the selected object. The degree of gaze is low. For this reason, when the density of the object is low, the importance of the temporary correction coefficient obtained at this time is low.

  Specifically, the importance determination unit 109 determines the coordinates of the selected object by comparing the identification information of the selected object with the object coordinate data 101c. Then, the importance level determination unit 109 compares the coordinates of the selected object with the object coordinate data 101c, and counts the number of objects included in a predetermined range centered on the coordinates of the selected object. The importance level determination unit 109 determines that the importance level of the temporary correction coefficient is high when the counted number is equal to or greater than the predetermined number. On the other hand, the importance level determination unit 109 determines that the importance level of the temporary correction coefficient is low when the counted number is less than the predetermined number.

  A process in which the importance level determination unit 109 determines the importance level based on the moving speed of the mouse pointer will be described. FIG. 11 is a diagram for explaining the behavior when the mouse pointer is moved to a target object. Generally, when the distance between the object 32 to be selected and the mouse pointer 31 is large, the user moves the mouse pointer at high speed. Then, as the distance between the object 32 to be selected and the mouse pointer 31 decreases, the user decreases the moving speed to finely adjust the position of the mouse pointer 31.

  FIG. 12 is a diagram illustrating a relationship with a moving speed when the mouse pointer is moved to an object to be selected. The vertical axis in FIG. 12 indicates the speed of movement speed, and the horizontal axis indicates time. As the time elapses, the mouse pointer approaches the object to be selected. In FIG. 12, it is assumed that an object to be selected is clicked at time T. As shown in FIG. 12, as the time approaches T, the moving speed decreases.

  That is, when the moving speed of the mouse pointer is low, it is considered that the user is gazing at the object to be selected, and the degree of gazing is high. For this reason, when the moving speed is low, the importance of the temporary correction coefficient obtained at this time is high. On the other hand, if the moving speed is high, it is considered that the user is not gazing at the object to be selected, and the degree of gazing is low. For this reason, when the moving speed is high, the importance of the temporary correction coefficient obtained at this time is low.

  Specifically, the importance level determination unit 109 calculates the average value of the moving speed of the mouse pointer from the timing when the selected object is selected to the timing before a predetermined time. Then, the importance level determination unit 109 determines that the importance level of the temporary correction coefficient is high when the average value of the moving speed is less than the predetermined speed. On the other hand, the importance level determination unit 109 determines that the importance level of the temporary correction coefficient is low when the average value of the moving speed is equal to or higher than the predetermined speed.

  The correction coefficient calculation unit 110 is a processing unit that calculates a correction coefficient based on a plurality of temporary correction coefficients and the importance of the temporary correction coefficient. First, after the correction coefficient calculation unit 110 has described the process of calculating the temporary correction coefficient, the process of calculating the correction coefficient will be described.

A process in which the correction coefficient calculation unit 110 calculates a temporary correction coefficient will be described. The correction coefficient calculation unit 110 acquires user identification information from the input unit 105, and acquires eyeball radius information corresponding to the user identification information from the personal parameter 101b. Then, the correction coefficient calculation unit 110 calculates the user's line of sight θ based on the feature amount of the feature point management table 101a and the eyeball radius. When the feature amount is x and the eyeball radius is r, the line of sight θ can be calculated by the equation (1).
θ = Asin ⁻¹ (x / r) + B (1)
A and B in the formula (1) are predetermined constants.

After calculating the line of sight θ, the correction coefficient calculation unit 110 calculates the coordinates on the screen at which the user is gazing with the line of sight θ. FIG. 13 is a diagram illustrating the relationship between the line of sight θ and the coordinates on the screen. As shown in FIG. 13, when the distance between the screen 40 and the user's eye 41 is D, the relationship between the coordinate X and the horizontal line of sight θ ₁ can be expressed by Expression (2).
X = D × {a × tan θ ₁ + b} (2)
Although not shown, the relationship between the Y coordinate and the line of sight θ _{2 in the} direction of gravity can be expressed by Expression (3). A and b included in the equations (2) and (3) are provisional correction coefficients.
Y = D × {a × tan θ ₂ + b} (3)

  The correction coefficient calculation unit 110 detects the coordinates of the selected object from the object coordinate data 101c, and the temporary correction coefficient a, which minimizes the difference between the detected coordinates and the coordinates obtained from the equations (2) and (3). Specify the value of b. The correction coefficient calculation unit 110 sequentially calculates temporary correction coefficients each time an object is selected by the user, and registers the temporary correction coefficients in the correction coefficient management table 101d.

  The correction coefficient calculation unit 110 acquires the determination result of the importance level determination unit 109 at the time when the temporary correction coefficient is calculated. When it is determined that the importance of the temporary correction coefficient is high, the importance flag is set to “ON” when the temporary correction coefficient is stored in the correction coefficient management table 101d. On the other hand, when it is determined that the importance of the temporary correction coefficient is low, the correction coefficient calculation unit 110 sets the importance flag when storing the temporary correction coefficient in the correction coefficient management table 101d. "Turn off.

  Subsequently, a process in which the correction coefficient calculation unit 110 calculates a correction coefficient based on the calculated temporary correction coefficient will be described. The correction coefficient calculation unit 110 weights temporary correction coefficients whose importance level flag is “on” among the temporary correction coefficients. The correction coefficient calculation unit 110 calculates a correction coefficient based on the weighted temporary correction coefficient and other temporary correction coefficients.

  For example, as shown in FIG. 5, if the importance flags of the temporary correction coefficients a2 and b2 are turned on and the importance flags of the other temporary correction coefficients a1, b1, a3, and b3 are turned off, Correction coefficients a2 and b2 are weighted to calculate correction coefficients a and b. In this case, for example, the correction coefficient calculation unit 110 calculates the correction coefficient a by (a1 + a2 × 2 + a3) / 4, and calculates the correction coefficient b by (b1 + b2 × 2 + b3) / 4. The correction coefficient calculation unit 110 updates the correction coefficient data 101e with the calculation result. For example, the correction coefficient calculation unit 110 repeatedly executes the above process every time the user selects the selected object and the feature amount is registered in the feature amount management table 101a.

  The gaze position detection unit 111 is a processing unit that detects coordinates on the screen on which the user is gazing. The gaze position detection unit 111 outputs the detected coordinates to the screen control unit 112.

  Specifically, the gaze position detection unit 111 extracts the feature amount of the user in the same manner as the eye detection unit 103 and the feature amount extraction unit 104. The gaze position detection unit 111 calculates the user's line of sight θ from the feature amount and the eyeball radius in the same manner as the correction coefficient calculation unit 110. After calculating the line of sight θ, the gaze position detection unit 111 calculates the coordinates to be watched by the user based on the line of sight θ, the correction coefficient of the correction coefficient data 101e, and equations (2) and (3). The line-of-sight position detection unit 111 repeatedly executes the above processing.

  The screen control unit 112 is a processing unit that executes various screen controls based on the detection result of the gaze position detection unit 111. For example, if the object and the coordinates obtained from the line of sight overlap for a predetermined time, the screen control unit 112 determines that the object has been clicked and selects the object. In addition, the screen control unit 112 scrolls the screen in accordance with the change direction of the coordinates of the line of sight. In addition, when the coordinates of the line of sight do not change for a predetermined time, the image control unit 112 enlarges the area of the coordinates.

  Next, a processing procedure of the eye gaze detection apparatus 100 according to the present embodiment will be described. FIG. 14 is a flowchart illustrating a processing procedure when the line-of-sight detection apparatus calculates a correction coefficient. For example, the process shown in FIG. 14 is executed when the user clicks the mouse. As shown in FIG. 14, the line-of-sight detection device 100 acquires operation data (step S101), and determines a selected object (step S102).

  The line-of-sight detection apparatus 100 determines the importance of the temporary correction coefficient from the relationship between the selected object and other objects, and updates the correction coefficient management table 101d (step S103). The line-of-sight detection device 100 calculates a correction coefficient based on the correction coefficient management table 101d (step S104). The process shown in FIG. 14 is repeatedly executed when the user clicks the mouse.

  Next, effects of the line-of-sight detection device 100 according to the present embodiment will be described. The line-of-sight detection apparatus 100 obtains a provisional correction coefficient every time the user selects an object. Then, the line-of-sight detection device 100 places emphasis on the temporary correction coefficient calculated when the degree of gaze of the user is high among a plurality of temporary correction coefficients based on the relationship between the selected object and other objects. Then, a correction coefficient is calculated. Since the line-of-sight detection apparatus 100 calculates the correction coefficient by taking advantage of the user's operation, the user is not made aware of the line-of-sight calibration. For this reason, when gaze calibration is performed, the burden placed on the user can be reduced.

  Further, since the line-of-sight detection device 100 performs weighting of the temporary correction coefficient based on the distance between the objects, the value of the temporary correction coefficient calculated when the user's gaze condition is high is used as the correction coefficient value. Can be reflected. For this reason, the accuracy of the correction coefficient is improved, and the coordinates of the user's gaze can be accurately calculated.

  In addition, since the line-of-sight detection apparatus 100 weights the temporary correction coefficient based on the density of the object, the temporary correction coefficient value calculated when the user's gaze condition is high is calculated based on the correction coefficient value. It can be reflected. For this reason, the accuracy of the correction coefficient is improved, and the coordinates of the user's gaze can be accurately calculated.

  In addition, since the line-of-sight detection device 100 weights the temporary correction coefficient based on the moving speed of the mouse pointer, the value of the temporary correction coefficient calculated when the user's gaze is high is used as the correction coefficient. It can be reflected by the value. For this reason, the accuracy of the correction coefficient is improved, and the coordinates of the user's gaze can be accurately calculated.

  The line-of-sight detection device 100 acquires information on the eyeball radius corresponding to the user from the personal parameter 101b, and calculates a correction coefficient using the information on the eyeball radius. For this reason, an error due to individual differences can be eliminated and the correction coefficient can be calculated accurately.

  Next, another example when the line-of-sight detection apparatus calculates a correction coefficient will be described. First, a case where the correction coefficient is calculated based on the density of the object will be described. An event in which an object is selected at a certain timing is referred to as event 1, and an event in which a different object is selected at a different timing from event 1 is referred to as event 2. The conditions for event 1 and event 2 are as follows.

  In event 1, it is assumed that there are five other objects in the 50 pixel square around the selected object. The other objects are referred to as other objects 1 to 5, respectively. The distance between the selected object and the other object 1 is a distance p11, the distance between the selected object and the other object 2 is p12, and the distance between the selected object and the other object 3 is p13. Further, the distance between the selected object and the other object 4 is p14, and the distance between the selected object and the other object 5 is p15. Further, the temporary correction coefficients calculated when the selected object is selected are a1 and b1.

  In event 2, it is assumed that two other objects exist within 50 pixels square around the selected object. The other objects are referred to as other objects 1 and 2, respectively. The distance between the selected object and the other object 1 is a distance p21, and the distance between the selected object and the other object 2 is p22. Further, the temporary correction coefficients calculated when the selected object is selected are a2 and b2.

A case where the correction coefficient calculation unit 110 calculates the correction coefficient when the events 1 and 2 are assumed as described above will be described. For example, the correction coefficient calculation unit 110 sets “the number of other objects” existing within 50 pixels square around the selected object as a weighting coefficient. In this method, the correction coefficient calculation unit 110
a = (5 × a1 + 2 × a2) / 7 (4)
b = (5 × b1 + 2 × b2) / 7 (5)
To calculate the correction coefficients a and b.

Further, when events 1 and 2 are assumed as described above, the correction coefficient calculation unit 110 may use “distance between the selected object and another object” as a weighting coefficient. In this method, the correction coefficient calculation unit 110
a = 1 / k × {(1 / p11 + 1 / p12 + 1 / p13 + 1 / p14 + 1 / p15) × a1 + (1 / p21 + 1 / p22) × a2)} (6)
b = 1 / k × {(1 / p11 + 1 / p12 + 1 / p13 + 1 / p14 + 1 / p15) × b1 + (1 / p21 + 1 / p22) × b2)} (7)
To calculate the correction coefficients a and b. K in the equations (6) and (7) is expressed by the equation (8).
k = 1 / p11 + 1 / p12 + 1 / p13 + 1 / p14 + 1 / p15 + 1 / p21 + 1 / p22 (8)

  Next, a case where the correction coefficient is calculated based on the moving speed of the mouse pointer will be described. As an example, it is assumed that there are 5 frames of image data before the moving speed of the mouse pointer is stopped below a reference value. The first frame, the second frame, the third frame, the fourth frame, and the fifth frame are set in time series order. Here, it is assumed that the correction coefficient calculation unit 110 calculates a temporary correction coefficient every time a frame is switched.

  The temporary correction coefficients obtained in the first frame are a1 and b1, the temporary correction coefficients obtained in the second frame are a2 and b2, and the temporary correction coefficients obtained in the third frame are a3 and b3. The provisional correction coefficients obtained in the fourth frame are a4 and b4, and the provisional correction coefficients obtained in the fifth frame are a5 and b5.

  The moving speed of the mouse pointer in the first frame is v1, the moving speed of the mouse pointer in the second frame is v2, and the moving speed of the mouse pointer in the third frame is v3. The moving speed of the mouse pointer in the fourth frame is v4, and the moving speed of the mouse pointer in the fifth frame is v5.

In the case where the temporary correction coefficient and the moving speed are assumed as described above, the correction coefficient calculation unit 110, for example,
a = m × (a1 / v1 + a2 / v2 + a3 / v3 + a4 / v4 + a5 / v5) (9)
b = m × (b1 / v1 + b2 / v2 + b3 / v3 + b4 / v4 + b5 / v5) (10)
To calculate correction coefficients a and b. M in the equations (9) and (10) is expressed by the equation (11).
m = 1 / v1 + 1 / v2 + 1 / v3 + 1 / v4 + 1 / v5 (11)

  By the way, the eye detection device 103, the feature amount extraction unit 104, the operation information acquisition unit 107, the selection object determination unit 108, the importance determination unit 109, the correction coefficient calculation unit 110, the gaze position detection unit 111, the screen control illustrated in FIG. The unit 112 corresponds to various integrated devices. This integrated device corresponds to, for example, an integrated device such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Alternatively, the processing units 103, 104, and 107 to 112 correspond to electronic circuits such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit).

  Further, the storage unit 101 illustrated in FIG. 1 corresponds to, for example, a semiconductor memory device such as a random access memory (RAM), a read only memory (ROM), and a flash memory, or a storage device such as a hard disk or an optical disk. To do.

  The line-of-sight detection apparatus 100 can also be realized by mounting each function of the line-of-sight detection apparatus 100 on an information processing apparatus such as a known personal computer, workstation, mobile phone, PHS terminal, mobile communication terminal, or PDA. .

  FIG. 15 is a diagram illustrating a hardware configuration of a computer constituting the visual line detection device according to the embodiment. As illustrated in FIG. 15, the computer 200 includes a CPU 201 that executes various arithmetic processes, an input device 202 that receives input of data from a user, and a monitor 203. The computer 200 includes a medium reading device 204 that reads a program and the like from a storage medium, and a network interface device 205 that exchanges data with other computers via a network. The computer 200 also includes a plurality of cameras 206 that capture images, a RAM 207 that temporarily stores various types of information, and a hard disk device 208. Each device 201 to 208 is connected to a bus 209.

  The hard disk device 208 stores a correction coefficient calculation program 208a, a gaze position detection program 208b, and various data 208c. The correction coefficient calculation program 208a has the same functions as the eye detection unit 103, the feature amount extraction unit 104, the operation information acquisition unit 107, the selected object determination unit 108, and the correction coefficient calculation unit 110. The gaze position detection unit program 208 b corresponds to the gaze position detection unit 111. The various data 208c includes information corresponding to the feature amount management table 101a, personal parameters 101b, object coordinate data 101d, correction coefficient management table 101d, and correction coefficient data 101e.

  The CPU 201 reads out the correction coefficient calculation unit program 208 a, the gaze position detection program 208 b, and various data 208 c from the hard disk device 208 and develops them in the RAM 207. The correction coefficient calculation program 208a functions as a correction coefficient calculation process 207a. The gaze position detection program 208b functions as a gaze position detection process 207b. The correction coefficient calculation process 207a uses the various data 207c to weight the temporary correction coefficient to calculate a complement coefficient, and the gaze position detection process 207b uses the correction coefficient to watch the user. Detect position.

  Note that the correction coefficient calculation unit program 208 a and the gaze position detection program 208 b described above are not necessarily stored in the hard disk device 208. For example, the computer 200 may read and execute a program stored in a storage medium such as a CD-ROM. Alternatively, the program may be stored in a public line, the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or the like, and the computer 200 may read and execute the program.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Supplementary note 1) When an object displayed on the screen is selected by the user, the user performs a line-of-sight input based on the relationship between the selected object and another object displayed on the screen. A correction coefficient calculation unit for calculating a correction coefficient for correcting the error of
On the screen on which the user is gazing, based on the correction coefficient calculated by the correction coefficient calculation unit and a feature amount related to the eye obtained from an image including the user's eye as a subject A line-of-sight detection apparatus comprising: a coordinate position detection unit that detects coordinates.

(Supplementary Note 2) The line-of-sight detection device further determines whether the user is gazing at the selected object based on the relationship between the selected object and the other object displayed on the screen. A gaze determination unit for determining
The correction coefficient calculation unit calculates a temporary correction coefficient every time the user selects the object, and determines that the gaze determination unit is gazing at the object among the calculated temporary correction coefficients. The temporary correction coefficient calculated when the object is calculated and the temporary correction coefficient calculated when it is determined that the object is not watched are weighted differently, and each weighted temporary correction coefficient is used as a basis. The line-of-sight detection apparatus according to appendix 1, wherein a correction coefficient is calculated.

(Supplementary note 3) The line of sight according to supplementary note 2, wherein the correction coefficient calculation unit weights the temporary correction coefficient based on a distance between the selected object and another object, and calculates the correction coefficient. Detection device.

(Supplementary note 4) The line of sight according to supplementary note 2, wherein the correction coefficient calculation unit calculates a correction coefficient by weighting a temporary correction coefficient based on a density of other objects around the selected object. Detection device.

(Additional remark 5) The said correction coefficient calculation part weights a temporary correction coefficient based on the moving speed of the mouse pointer from the time when the object was selected to the predetermined time before, and calculates a correction coefficient, It is characterized by the above-mentioned. The line-of-sight detection device according to attachment 2.

(Additional remark 6) The said correction coefficient calculation part refers to the memory | storage part which matches and memorize | stores user identification information and an eyeball radius, and respond | corresponds to this user identification information based on the input user identification information The eye gaze detection apparatus according to any one of appendices 1 to 5, wherein an eyeball radius is specified and a correction coefficient is calculated by further using the eyeball radius.

(Appendix 7)
When an object displayed on the screen is selected by the user, an error when the user performs eye-gaze input is corrected based on the relationship between the selected object and another object displayed on the screen. To calculate the correction coefficient to
A process of detecting coordinates on the screen on which the user is gazing based on the calculated correction coefficient and a feature amount related to the eye obtained from an image including the user's eye as a subject. Correction coefficient calculation program to be executed.

(Supplementary Note 8) Further, based on the relationship between the selected object and the other object displayed on the screen, a process of determining whether or not the user is gazing at the selected object Let the computer run,
When calculating the correction coefficient, a temporary correction coefficient is calculated each time an object is selected by the user, and is calculated when it is determined that the object is being watched out of the calculated temporary correction coefficients. The temporary correction coefficient and the temporary correction coefficient calculated when it is determined that the object is not watched are weighted differently, and the correction coefficient is calculated based on each weighted temporary correction coefficient. The correction coefficient calculation program according to appendix 7, characterized by:

(Supplementary note 9) The supplementary note 8, wherein when calculating the correction coefficient, the temporary correction coefficient is weighted based on a distance between the selected object and another object to calculate the correction coefficient. Correction coefficient calculation program.

(Additional remark 10) When calculating the said correction coefficient, weighting of a temporary correction coefficient is performed based on the density of the other object surrounding the selected object, and a correction coefficient is calculated, The additional coefficient 8 characterized by the above-mentioned. Correction coefficient calculation program.

(Supplementary Note 11) When calculating the correction coefficient, the correction coefficient is calculated by weighting the temporary correction coefficient based on the moving speed of the mouse pointer from the time when the object is selected to a predetermined time before. The correction coefficient calculation program according to appendix 8.

(Additional remark 12) When calculating the said correction coefficient, with reference to the memory | storage device which matches and memorize | stores user identification information and an eyeball radius, based on the input user identification information, this user identification information The correction coefficient calculation program according to any one of appendices 7 to 11, characterized in that an eyeball radius corresponding to is specified and a correction coefficient is calculated by further using the eyeball radius.

DESCRIPTION OF SYMBOLS 100 Eye-gaze detection apparatus 101 Memory | storage part 102 Camera 103 Eye detection part 104 Feature-value extraction part 105 Input part 106 Screen display part 107 Operation information acquisition part 108 Selected object determination part 109 Importance determination part 110 Correction coefficient calculation part 111 Gaze position detection part 112 Screen controller

Claims (7)

When an object displayed on the screen is selected by the user, an error when the user performs eye-gaze input is corrected based on the relationship between the selected object and another object displayed on the screen. A correction coefficient calculation unit for calculating a correction coefficient for
On the screen on which the user is gazing, based on the correction coefficient calculated by the correction coefficient calculation unit and a feature amount related to the eye obtained from an image including the user's eye as a subject A line-of-sight detection apparatus comprising: a coordinate position detection unit that detects coordinates. The line-of-sight detection device further determines whether the user is gazing at the selected object based on a relationship between the selected object and the other object displayed on the screen. Having a determination unit,
The correction coefficient calculation unit calculates a temporary correction coefficient every time the user selects the object, and determines that the gaze determination unit is gazing at the object among the calculated temporary correction coefficients. The temporary correction coefficient calculated when the object is calculated and the temporary correction coefficient calculated when it is determined that the object is not watched are weighted differently, and each weighted temporary correction coefficient is used as a basis. The gaze detection device according to claim 1, wherein a correction coefficient is calculated.   The visual axis detection device according to claim 2, wherein the correction coefficient calculation unit weights the temporary correction coefficient based on a distance between the selected object and another object, and calculates the correction coefficient.   The visual axis detection device according to claim 2, wherein the correction coefficient calculation unit calculates a correction coefficient by weighting a temporary correction coefficient based on a density of other objects around the selected object.   The correction coefficient calculation unit calculates a correction coefficient by weighting a temporary correction coefficient based on a moving speed of a mouse pointer from a time when an object is selected to a predetermined time before. The line-of-sight detection device described.   The correction coefficient calculation unit refers to a storage unit that stores the user identification information and the eyeball radius in association with each other, and based on the input user identification information, calculates the eyeball radius corresponding to the user identification information. The eye gaze detection apparatus according to claim 1, wherein the eye gaze radius is specified and the correction coefficient is calculated further using the eyeball radius. On the computer,
When an object displayed on the screen is selected by the user, an error when the user performs eye-gaze input is corrected based on the relationship between the selected object and another object displayed on the screen. To calculate the correction coefficient to
A process of detecting coordinates on the screen on which the user is gazing based on the calculated correction coefficient and a feature amount related to the eye obtained from an image including the user's eye as a subject. Correction coefficient calculation program to be executed.

Images