JP5590487B2 - Gaze measurement method and gaze measurement device - Google Patents

Description

  The present invention relates to a line-of-sight measurement method and a line-of-sight measurement apparatus.

  Gaze information is used in a wide range of fields such as basic research in neuroscience, evaluation of psychological states such as attention and interest, and an input interface to PC devices. In the future, it is assumed that opportunities for obtaining line-of-sight information are not limited to the above, but will increase in daily life.

  From such a background, eye gaze measurement is demanded not only for positional accuracy but also for the eye gaze measuring method that is light on the subject and easy to calibrate.

  In the conventional gaze measurement method using a camera, the center of the pupil has been used for detecting the gaze direction (for example, Patent Documents 1 and 2). In addition, there is a method of measuring the line of sight by measuring the central position of the corneal curvature (Non-Patent Document 1).

JP 2004-167152 A JP 2005-185431 A

“Gaze Detection Method by Measuring Corneal Curvature Center Position Using Two Light Sources” Hiroaki Tanaka, Shinichi Kajita, Ken Kasai, Takashi Takeda; The Institute of Electronics, Information and Communication Engineers; IEICE Technical Report MBE 2008-128 (2009-03); 177-p. 180

  As disclosed in Patent Documents 1 and 2, the use of the pupil center for detection of the line-of-sight direction has the following problems. The size of the pupil changes from moment to moment due to changes in brightness and movement of the gazing point in the perspective direction, and there is no guarantee that concentric circles will be maintained during pupil expansion and contraction. This is a major cause of a decrease in the accuracy of the detected gaze position. Further, when performing calibration for improving the gaze position accuracy, a process of gazing at many indices at known positions is required, and simple measurement cannot be performed.

  In Non-Patent Document 1, gaze measurement is performed by giving a distance from the corneal curvature center to the eyeball rotation center, and the measurement result varies depending on the distance from the given corneal curvature center to the eyeball rotation center. There is a problem with accuracy.

  The present invention has been made in view of the above-described matters, and an object of the present invention is to provide a gaze measurement method and a gaze measurement apparatus that are easy to calibrate with a light burden on a subject.

The line-of-sight measurement method according to the first aspect of the present invention includes:
The eyeball before and after the subject's rotation is photographed in advance by one imaging device, and three or more arbitrary feature points on the eyeball surface are extracted from the captured image, and based on the three-dimensional position of the feature point before and after the eyeball is rotated. to previously obtain the positional relationship between the feature point and the eyeball rotation center Te,
The subject gazes at any three points of interest whose viewing angles between the points are known in order, and when moving between the two points of interest, a predetermined point on the eyeball surface is the eyeball. A first locus circle, a second locus circle, and a third locus circle that move around the rotation axis are obtained, and the first locus circle, the second locus circle, and the third locus circle are obtained. Is obtained as a gaze vector tip, and a gaze vector connecting the eyeball rotation center and the gaze vector tip is obtained as the gaze direction of the subject.
The positional relationship between the line-of-sight vector and the feature point is obtained,
The eyeball is imaged by the imaging device, and the gaze direction of the subject is measured by obtaining the gaze vector based on the position of the feature point on the captured image.
It is characterized by that.

  Further, a branch point of a blood vessel may be extracted as the feature point.

The line-of-sight measurement apparatus according to the second aspect of the present invention is:
One imaging device for imaging the eyeball of the subject;
An arithmetic device that measures the gaze direction of the subject based on the image captured by the imaging device,
The arithmetic unit is:
Three or more arbitrary feature points on the eyeball surface are extracted from an image obtained by imaging the eyeball before and after the rotation of the subject in advance, and the feature point and the eyeball rotation center are determined based on the three-dimensional position of the feature point before and after the eyeball rotation. When the subject moves around two points of interest in order to make a round of arbitrary three points of interest where the viewing angle between the points is known, A first trajectory circle, a second trajectory circle, and a third trajectory circle in which a predetermined point moves around the eyeball rotation axis are obtained, and the first trajectory circle, the second trajectory circle, and Means for obtaining an intersection of the third trajectory circle as a gaze vector tip, obtaining a gaze vector connecting the eyeball rotation center and the gaze vector tip as a gaze direction of a subject, and obtaining a positional relationship between the gaze vector and the feature point When,
That having a means for measuring a gaze direction of the subject seeking the gaze vector based on the position of the feature point on the image of the captured eye by the image pickup device,
It is characterized by that.

The line-of-sight measurement method according to the third aspect of the present invention includes:
The subject gazes at three points whose viewing angles are known, and each point irradiates the subject's cornea with light from the first light source, and reflects the reflected light that has passed through the optical center of the lens after being reflected by the cornea. An image is projected onto the imaging region, a first plane including the first light source, the optical center of the lens, and the first reflected image is obtained, and the cornea is irradiated with light from the second light source. The reflected light that has passed through the optical center of the lens is projected onto the imaging region as a second reflected image, and includes a second light source, an optical center of the lens, and a second reflected image that includes the second reflected image. Determining a plane, and determining the line of intersection of the first plane and the second plane as a straight line passing through the corneal curvature center;
Assuming a first triangular pyramid having a triangle consisting of three corneal curvature centers with the center of eyeball rotation as the bottom, and a bottom having a triangle consisting of the corneal curvature center with three points as the vertex and the optical center of the lens. Analysis of the first triangular pyramid and the second triangular pyramid having the same triangle as the bottom by normalizing the distance from the eyeball rotation center to the corneal curvature center Performing the step of obtaining the position of the eyeball rotation center,
Gaze the subject at an arbitrary point, perform a step of obtaining a straight line passing through the corneal curvature center, and obtaining a straight line passing through the corneal curvature center;
Obtain the straight line connecting the intersection of the sphere centered on the eyeball rotation center normalized by the eyeball radius and the distance from the center of eyeball rotation to the center of corneal curvature, and measure the gaze direction of the subject A process.

The line-of-sight measurement apparatus according to the fourth aspect of the present invention is:
A first light source and a second light source for irradiating the subject's cornea with light;
The light irradiated from the first light source and the second light source is reflected by the cornea and reflected light passing through the optical center of the lens is projected into the imaging region as a first reflected image and a second reflected image, respectively. An imaging device to
The subject gazes at three points whose viewing angles are known, and for each point, the first light source, the optical center of the lens, the first plane including the first reflected image, and the second A light source, an optical center of the lens, and a second plane including the second reflected image are obtained, and a line of intersection between the first plane and the second plane is defined as a straight line passing through the corneal curvature center. Assuming that the first triangular pyramid has a triangular shape with the center of eyeball rotation as the apex and the triangle composed of three corneal curvature centers as the bottom, and the triangle with the optical center of the lens as the apex and consisting of the three corneal curvature centers as the bottom Assuming a second triangular pyramid, and normalizing the distance from the center of eyeball rotation to the corneal curvature center, the first triangular pyramid having the same triangle as the bottom surface and the second triangular pyramid Geometric analysis is performed to find the position of the eyeball rotation center Means, and when the subject gazes at an arbitrary point, the first plane and the second plane are obtained to obtain a straight line passing through the corneal curvature center, and the obtained straight line and the eyeball rotation center to the corneal curvature center are obtained. An arithmetic unit having a means for obtaining a straight line connecting an intersection of a sphere centered on an eyeball rotation center with a distance normalized by an eyeball radius and an eyeball rotation center, and measuring a subject's gaze direction,
It is characterized by that.

  In the line-of-sight measurement method according to the present invention, calibration can be easily performed simply by having a subject gaze at three points. In addition, since the gaze of the subject can be measured with only two light sources and one camera, or only one camera, a small gaze measuring device can be realized. For this reason, the burden on the subject can be reduced.

1 is a perspective view showing a line-of-sight measurement device according to Embodiment 1. FIG. 3 is a schematic diagram illustrating a camera coordinate system in Embodiment 1. FIG. It is a schematic diagram which shows the movement of the feature point accompanying rotation of an eyeball. It is a schematic diagram which shows the locus | trajectory circle to which the front-end | tip of a gaze vector moves when a gaze is moved from the gaze point A to the gaze point B. It is a figure which shows the measurement result of a rotation angle. FIG. 6 is a perspective view showing a line-of-sight measurement device according to Embodiment 2. It is a perspective view which shows the example which installed the imaging device and the light source in the spectacle lens upper surface. 6 is a schematic diagram illustrating a camera coordinate system according to Embodiment 2. FIG. It is a schematic diagram which shows the straight line which passes along a corneal curvature center. It is a schematic diagram which shows the triangle which consists of a corneal curvature center position when gazes at three points | pieces, and an eyeball rotation center. It is a figure which shows the result of the gaze angle at the time of horizontal rotation. It is a figure which shows the result of a gaze position accuracy.

(Embodiment 1)
The line-of-sight measurement method and line-of-sight measurement apparatus according to Embodiment 1 will be described. FIG. 1 shows, as an example, a line-of-sight measurement apparatus 1 that can be used to realize a line-of-sight measurement method. The line-of-sight measurement device 1 transmits an imaging device 10 such as a CCD camera that can image the eyeball of the subject, a calculation device 11 that calculates the direction of the subject's line of sight from the captured image information, and transmits the captured image information to the calculation device 11. A transmission line 12. This example is a spectacle type in which the imaging device 10 is installed on the glasses 13 and can be suitably used because the positional relationship between the eyeball of the subject and the imaging device 10 hardly changes during measurement.

  Further, the spectacle lens 13a may be configured to incorporate a transmissive display so that an external image and an information image from the display element can be superimposed and observed simultaneously.

  For the gaze measurement in the gaze measurement apparatus 1 described above, the following gaze measurement method is referred to.

  The line-of-sight measurement method according to Embodiment 1 previously extracts three or more arbitrary feature points on the eyeball surface of the subject, obtains the positional relationship between the intersection of the line-of-sight direction and the eyeball, and the feature point, and the imaging apparatus In this method, the eyeball is imaged and the gaze direction of the subject is measured based on the position of the feature point on the captured image.

The principle of gaze measurement will be described with reference to the drawings. FIG. 2 shows a camera coordinate system with the optical center of the lens as the origin. It is assumed that the eyeball is a true sphere and rotates around a certain point (eyeball rotation center O _c ). Also, internal parameters (image center, focal length, pixel size) of the camera to be used are known by prior calibration, and the three-dimensional feature points projected on the imaging surface, for example, on a CCD (Charge Coupled Device) surface Assume that the position is obtained from image coordinates.

Three feature points on the eyeball surface are extracted by image processing from a capillary blood vessel pattern or the like included in the photographed image of the eyeball. Then, the eyeball rotation center from the positions of these feature points O _c, and measures the position of an intersection of the line of sight (fixation point and connecting the eyeball rotation center O _c line) and ocular surface (tip of the sight line vector). The horizontal and vertical directions of the line of sight with respect to the camera are calculated from the line-of-sight vector connecting the eyeball rotation center _Oc and the above intersection. The rotation angle around the line of sight is calculated from the position of the feature point with respect to the line-of-sight vector.

The position vector P _i (i = 1, 2, 3) of the feature point on the eyeball surface in the camera coordinate system is a vector p _i = (x _i , y _i , z) toward the feature point projected on the CCD plane. the unknown parameters K _{i (distance} between the lens optical center O and P _i is, K _i times the definition of the distance between the lens optical center O and p _i), and each of the feature points from the vector O _c of the eyeball rotation center the vector _{r i} towards using vector can be represented by the formula _{(1) P i = K i} · vector _{p i} = vector _{O c} + vector _r i (1)

Next, when the rotational movement of the eyeball occurs, as shown in FIG. 3, the feature point vector P _i before rotation is converted into the feature point vector Q _i after rotation, and the feature point vector p _i projected onto the CCD surface is Move to vector q _i . In this case, the feature point vector _{Q i} (defined as _{J i} times the distance between the distance between the lens optical center O and a vector _{Q i} is the lens optical center O and the vector _{q i)} vector _{q i} and the unknown parameters _{J i,} This is expressed by Expression (2) using a vector t _i from the rotation center vector O _c to the vector q _i and a rotation matrix A.
Vector Q _i = J _i · vector q _i = vector O _c + vector t _i
= Vector O _c + rotation matrix A · vector r _i (2)

Vector of the eyeball rotation center _{O c,} the unknown parameters _{K i} and _{J i} the ones obtained by normalizing the eyeball radius as a respective vector _{O n, K 'i, J} ' is set to _i. Since the radius of the eyeball does not change before and after the rotation of the eyeball, the expressions (3) and (4) are derived from the expressions (1) and (2).
| K ′ _i · vector p _i −vector O _n | ² = 1 (3)
| J ′ _i · vector q _i −vector O _n | ² = 1 (4)

By solving equations 3 and 4 of each unknown parameter K _'i, J' _i, the following equation using vector _{O n} (5), represented by (6).

Since the mutual positional relationship between the feature points is unchanged even before and after the rotation of the eyeball, the following equations (7) and (8) are obtained from the relationship between the distances of the three feature points normalized with the eyeball radius set to 1. ), () 9 are obtained.
| K ′ ₁ · vector p ₁ −K ′ ₂ · vector p ₂ | ²
= | J ′ ₁ · vector q ₁ −J ′ ₂ · vector q ₂ | ² (7)
| K ′ ₂ · vector p ₂ −K ′ ₃ · vector p ₃ | ²
= | J ′ ₂ · vector q ₂ −J ′ ₃ · vector q ₃ | ² (8)
| K ′ ₃ · vector p ₃ −K ′ ₁ · vector p ₁ | ²
= | J ′ ₃ · vector q ₃ −J ′ ₁ · vector q ₁ | ² (9)

From equation (5) to (9), since nine scalar equations is obtained by solving these, the rotational center vector O _n of the eye is obtained. Furthermore, unknown parameters K ₁ ′ to K ₃ ′, J ₁ ′ to J ₃ ′ are also obtained.

As described above, the position vector P _'i, the vector Q' before and after rotation of the normalized feature points eyeball radius _i is, K _'i, J' _{because i} is determined, K _'i, J' and _i And can be obtained by equations (10) and (11).
Vector P ′ _i = K ′ _i · vector p _i (10)
Vector Q ′ _i = J ′ _i · vector q _i (11)

  Therefore, all the normalized positions of the feature points before and after the eyeball rotation are determined.

Since the position vector r _'i, t' of the normalized feature points before and after the rotation with respect to the position vector O _n of the eyeball rotation center for even _i, said unknown parameters is determined, Equation (1), obtained from (2) .

Further, a position vector R = (r ′ ₁ , r ′ ₂ , r ′ ₃ ) ^T of the feature point before rotation, and a position vector T = (t ′ ₁ , t ′ ₂ , t ′ ₃ ) ^T after rotation. The feature points before and after the eyeball rotation are expressed by the following equation (12) using the rotation matrix A.
T = AR (12)

  As a result of obtaining the rotation matrix A from the above equation, the direction vector n and the rotation angle φ of the rotation axis of the eyeball are determined.

  Subsequently, the relationship between the line-of-sight direction vector and the feature point is obtained.

Up to now, the positions of the three feature points on the eyeball surface in the posture P and the posture Q of the eyeball are respectively measured by the camera, whereby the vector O _n of the eyeball rotation center in the coordinate system normalized with the eyeball radius set to 1 It has been shown that a position vector r ′ _i of feature points on the eyeball surface and a rotation matrix A from posture P ′ to Q ′ can be determined.

  However, even if the rotation matrix A is given, the rotation angle of the line of sight is not given. This is because the relationship between the positions of the three feature points and the line-of-sight direction is not given. Here, it is possible to determine the positional relationship between the line-of-sight direction and the feature point from the measurement result of the feature point position with respect to each gazing point by selecting three gazing points (the viewing angle between each target is known). Indicates.

First, let θ _i (i = 1, 2, 3) be the line-of-sight rotation angles when any three gazing points A, B, C are watched in the order of A → B → C → A. Table 1 shows the rotation matrix, the center vector of the locus circle, the radius of the locus circle, the line-of-sight rotation angle, the eyeball rotation axis, and the eyeball rotation angle corresponding to the rotation matrix.

When moving the line of sight from the fixation point A to B, the tip of the line-of-sight vector, as shown in FIG. 4, to move from the vector F ₁ to the vector F _2. In this case the line of sight rotation angle _{∠F 1 O n F 2 = θ} 1, the eyeball rotation angle _{∠F 1 O 1 F 2 = φ} 1. The locus circle to which the tip of the line-of-sight vector moves is a circle having a radius R ′ ₁ and having the direction vector n ₁ of the eyeball rotation axis as a normal vector (center vector O _{1 of the} locus circle).

Similarly, when the line of sight is shifted from B to C, there is a locus circle (the center vector O _{2 of the} locus circle) in which the tip of the line-of-sight vector moves. Similarly, there is a locus circle (the center vector O _{3 of the} locus circle) in which the tip of the line-of-sight vector moves when the line of sight is shifted from C to A.

In this way, by moving the line of sight from the gazing point A to B, from B to C, and from C to A, a trajectory circle in which the tip of a total of three gaze vectors moves is obtained, and one intersection of the three trajectory circles is obtained. It is done. Accordingly, the vector from the eyeball rotation center O _n the intersection of the three circular path is line-of-sight vector.

The center vector O _i of the three trajectory circles is expressed by the following equation using the direction vector n _i of the rotation axis and the radius R ′ _i .
Vector O _i = vector O _n + L ′ _i · vector n _i (13)
| Vector O _i −Vector O _n | ² = 1−R ′ _i ² (14)

ここで、正規化座標系において、眼球半径Ｒ＝１、視線ベクトルの回転角θ_ｉは既知であり、眼球回転角φ_ｉは回転前後の特徴点の位置から与えられる。 Here, L ′ _i is an unknown parameter, and if the radius R ′ _i of the locus circle is known, the center vector O _i of the three locus circles can be determined. The radius R ′ _i of the center vector O _i of the locus circle is obtained from the relationship between the rotation angle θ _i of the line-of-sight vector and the eyeball rotation angle φ _i .

Here, in the normalized coordinate system, the eyeball radius R = 1, the line-of-sight vector rotation angle θ _i is known, and the eyeball rotation angle φ _i is given from the position of the feature point before and after the rotation.

The center vectors O ′ _i of the three trajectory circles after the three gazing points are observed in the order of A → B → C → A are expressed by the following equations (16), (17), and (18).
O ′ ₁ = A ₃ A ₂ O ₁ (16)
O ′ ₂ = A ₃ O ₂ (17)
O ′ ₃ = O ₃ (18)
Here, as shown in Table 1, the rotation matrices A ₁ , A ₂ , and A ₃ correspond to line-of-sight movements of A → B, B → C, and C → A, respectively.

And the front-end | tip F of a gaze vector is calculated | required by following Formula as an intersection of three locus | trajectory circles.
| Vector F−Vector O ′ ₁ | ² = R ′ ₁ ² (19)
| Vector F−Vector O ′ ₂ | ² = R ′ ₂ ² (20)
| Vector F−Vector O ′ ₃ | ² = R ′ ₃ ² (21)
| Vector F−O _n | ² = 1 (22)

From the above, the line-of-sight direction vector G is given by the following equation.
Vector F = Vector P ′ + Vector ΔF (23)
Vector G = vector F- vector _O n (24)
Here, the vector ΔF is a directional vector from the feature point vector P ′ to the vector F when the gazing point A is being watched, and represents the positional relationship between the line-of-sight direction and the feature point.

  Accordingly, if the vector ΔF is first obtained, the line-of-sight vector can be calculated from the measurement result of the feature points thereafter. By obtaining the line-of-sight vector, the rotation angle around the line of sight can be easily obtained from the position of the feature point with respect to the line-of-sight vector.

  As described above, in the visual line measurement method according to the present embodiment, since the relationship between the three feature points on the eyeball surface and the direction vector of the visual line is obtained, the visual line is obtained by imaging the three feature points with the camera. Can be measured. Further, the calibration is easy because the relationship between the three feature points on the eyeball surface and the direction vector of the line of sight is obtained simply by having the subject gaze at the three gaze points in order. In addition, since the line of sight can be measured with only one camera, it is possible to reduce the size of the measurement system using this. For this reason, the burden placed on the subject is also reduced.

(Verification experiment)
In order to verify the usefulness of the line-of-sight measurement method according to the present embodiment, a measurement experiment was performed using glass spheres (diameter: 15 mm) marked with three points as feature points. An optical system is installed so that the center of rotation of the glass sphere coincides with the center of the optical stage, and the rotation of the glass sphere from the feature point coordinates when the glass sphere is rotated every 10 degrees within a range of ± 20 deg in both horizontal and vertical angles. Angle measurement was performed.

  FIG. 5 shows a measurement result of the rotation angle measured when the optical stage is manually rotated from −20 deg to 20 deg in each of the horizontal direction and the vertical direction. As a result of measuring the rotation angle of the glass sphere based on the eyeball rotation angle φ with respect to 25 targets, the position error was as small as 0.1 to 1.1 deg.

(Embodiment 2)
A line-of-sight measurement method and a line-of-sight measurement apparatus according to Embodiment 2 will be described. FIG. 6 shows, as an example, a line-of-sight measurement device 2 that can be used to realize the line-of-sight measurement method according to the second embodiment. The line-of-sight measurement device 1 images two light sources 14 and 15 such as LEDs that irradiate light on the subject's cornea, and an imaging device 10 such as a CCD camera that captures each reflected light reflected by the cornea as a reflected image. A calculation device 11 that calculates the gaze direction of the subject from the image information and a transmission line 12 that transmits the captured image information to the calculation device 11 are provided. In this example, the imaging apparatus 10 and the two light sources 14 and 15 are of a spectacle type in which the spectacles 13 are installed. Since the positional relationship between the eyeball of the subject and the imaging apparatus 10 hardly changes during the measurement of the line of sight, it can be suitably used.

  Further, as shown in FIG. 7, a wavelength-selective reflecting mirror that transmits visible light from the outside as the spectacle lens 13 a (for example, HOE: Holographic Optical Element) is used, and the imaging device 10 and the light sources 14 and 15 are connected to the spectacles. You may install in the lens 13a upper surface. As indicated by the arrows, the light from the light sources 14 and 15 is reflected by the spectacle lens 13a and applied to the cornea, and the positional change accompanying the eye movement of the cornea reflection image is picked up by the image pickup device 10 through the same spectacle lens 13a. can do. The apparatus can be reduced in size and weight, and the visual field of the subject can be prevented from being obstructed, so that the line-of-sight position can be objectively known without interfering with the psychological state and behavior of the measurement subject.

  Further, the spectacle lens 13a may be configured to incorporate a transmissive display so that an external image and an information image from the display element can be superimposed and observed simultaneously.

  Furthermore, it is possible to measure three-dimensionally where the subject's gazing point is located in the outside world. As described above, the positional change accompanying the eye movement of the corneal reflection image by the light source projected from the spectacle lens 13a through the reflector (for example, HOE) having an inclination of about 45 degrees created in the spectacle lens 13a, By measuring two-dimensionally with the imaging device 10 attached to the upper surface of the spectacle lens 13a, the instantaneous line-of-sight direction can be measured. Then, the above set can be installed on both lenses of the spectacles 13, and the intersection (gaze point position) of the binocular gaze can be measured from the simultaneous measurement values of the horizontal and vertical gaze directions of each eye. Depending on the configuration of the optical system, it is possible to measure the line of sight of both eyes with only one set. Then, an image of the outside world is captured by separately installing an imaging device at the middle point of both eyes of the spectacles 13, and in order to associate the measured line-of-sight direction with the position of the outside world (measurement value calibration), The line-of-sight position measured on the display by the installed HOE is displayed as a light spot, and feedback is performed so as to coincide with the position watched by the subject, thereby associating the measured value with the true value.

  For the measurement of the line of sight in the line-of-sight measurement device 2, refer to the following line-of-sight measurement method.

  The line-of-sight measurement method according to Embodiment 2 causes the subject to gaze at three points whose viewing angles are known, and irradiates the subject's cornea with light from the first light source at each point, and the reflected light reflected by the cornea A first reflected image is projected onto the imaging region through the optical center of the lens, a first plane including the first light source, the optical center of the lens, and the first reflected image is obtained, and the second The light from the light source is irradiated onto the cornea, and the reflected light reflected by the cornea passes through the optical center of the lens to project a second reflected image onto the imaging region, and the second light source, the optical center of the lens, and the first Obtaining a second plane including two reflection images, obtaining an intersecting line between the first plane and the second plane as a straight line passing through the corneal curvature center, and a three-point corneal curvature with the eyeball rotation center as a vertex. Assuming a first triangular pyramid with the center triangle as the bottom, and the lens optics Assuming a second triangular pyramid with the triangle at the heart and the triangle consisting of the three corneal curvature centers as the bottom, normalizing the distance from the center of eyeball rotation to the corneal curvature center, A step of performing geometric analysis of the first triangular pyramid and the second triangular pyramid to determine the position of the eyeball rotation center, and a step of causing the subject to gaze at an arbitrary point and to obtain a straight line passing through the corneal curvature center. To obtain a straight line passing through the corneal curvature center, and the intersection of the obtained straight line and a sphere centered on the eyeball rotation center obtained by normalizing the distance from the eyeball rotation center to the corneal curvature center by the eyeball radius and the eyeball A step of obtaining a straight line connecting the rotation center and measuring the direction of the subject's line of sight.

  The principle of the line-of-sight measurement method will be described with reference to the drawings. FIG. 8 shows a camera coordinate system with the optical center O of the camera lens as the origin. When infrared light is emitted toward the cornea from the point light sources LED1 and LED2 at two different positions, two erecting virtual images (Purkinje-Sanson images) are formed behind the cornea. By taking these two corneal reflection images P1 and P2 with a camera, the positions of the corneal curvature center and the eyeball rotation center are measured, and the gaze direction is calculated as a straight line connecting these two points. It is assumed that the cornea is a part of a complete sphere, and the eyeball rotates around a certain point.

First, calculation of the corneal reflection image position will be described. First, a cornea reflection image region corresponding to each of the light sources LED1 and LED2 is extracted from the photographed image. Next, the position (x _p , y _p ) of the cornea reflection image P on the photographed image is determined by calculating the center of gravity of the reflection image region using the luminance value of each pixel as shown in Equations 1 and 2. Is done. In Equations 1 and 2, the lower left coordinate of the reflected image region is (x _l , y _l ), the upper right coordinate is (x _r , x _r ), and f (x, y) is the coordinate on the image ( The luminance value at x, y).

Next, the corneal reflection image position P (x ′ _p , y ′ _p , z ′ _p ) of the light source in the camera coordinate system can be expressed by the following equation.
x ′ _p = μx (x _p −x _c ) (3)
y ′ _p = μx (y _p −y _c ) (4)
z ′ _p = D (5)
Here, μ _x and μ _y are conversion coefficients and correspond to the pixel size. x _c and y _c are intersections between the optical axis of the lens and the imaging surface (for example, a CCD surface), and are positions at the center of the image. D is the distance from the image center to the optical center. These are all known constants specific to the camera.

  Subsequently, a straight line passing through the corneal curvature center is calculated with reference to FIG.

  When the eyeball is irradiated with near-infrared light, the light source position, the lens optical center, the position of the corneal reflection image projected on the CCD surface, and the corneal curvature center are located on the same plane. For each light source, one plane consisting of the cornea reflection image, the lens optical center, and the light source is determined. When two light sources L1 and L2 having different positions are used, two planes each including the three points are determined.

  The intersecting line between the two planes is a straight line passing through the lens optical center toward the corneal curvature center vector Q (q).

The optical center is O, and the light source reflection image positions of the two light sources L1 (vector l ₁ ) and L2 (vector l ₂ ) are P ₁ (vector p ₁ ) and P ₂ (vector p ₂ ). Then, a vector n ₁ perpendicular to the plane formed by L ₁ , O, and P ₁ and a normal vector n ₂ perpendicular to the plane formed by L ₂ , O, and P ₂ are obtained from equations (6) and (7), respectively. .
Vector n ₁ = (vector p ₁ -vector o) × (vector l ₁ -vector o) (6)
Vector n ₂ = (vector p ₂ −vector o) × (vector l ₂ −vector o) (7)

As shown in the equation (8), the intersection direction vector v is obtained from the outer product of the vector n ₁ and the vector n ₂ .
Vector v = vector n ₁ × vector n ₂ (8)

The intersection line m is expressed by Expression (9) by using the optical center vector O (o) and the parameter K.
Vector m = vector o + K · vector v (9)

  Subsequently, the eyeball rotation center position is calculated. As shown in FIG. 10, the eyeball rotation center can be obtained from three gazing points with known viewing angles and three straight lines passing through the corneal curvature center.

Assuming that the positions of the corneal curvature centers when the three visual targets are looked at in turn are Q ₁ , Q ₂ , and Q ₃ , these positions are represented by three unknown parameters K ₁ , K ₂ , and K from Equation 9. By using ₃ , it can be expressed as the following formulas (10) to (12).
Vector q ₁ = vector o + K ₁ · vector v ₁ (10)
Vector q ₂ = vector o + K ₂ · vector v ₂ (11)
Vector q ₃ = vector o + K ₃ · vector v ₃ (12)

Here, the angles formed by the straight lines connecting the three corneal curvature center positions Q ₁ , Q ₂ , Q ₃ and the eyeball rotation center position vector E (e) are φ ₁ , φ ₂ , φ ₃ , respectively. When the distance between the center of curvature of the cornea and the center of corneal curvature is R, the lengths of the sides in the triangles Q ₁ , Q ₂ , Q ₃ are expressed by the following expressions (13) to (15) using Expressions (10) to (12). be able to. The left side in the equations (13) to (15) is the length of each side in the triangles Q ₁ , Q ₂ , Q ₃ calculated from the triangle composed of the center of eyeball rotation and the two corneal curvature centers, and the right side is This is the length of each side in the triangles Q ₁ , Q ₂ , and Q ₃ calculated from the triangle formed by the optical center of the lens and the center position of the two corneal curvatures.

Since the equations (13) to (15) include four unknown parameters (R, K ₁ , K ₂ , K ₃ ), they cannot be solved as they are. So, divide both sides of the above equation by R ²
When K ₁ / R = K ′ ₁ , K ₂ / R = K ′ ₂ , and K ₃ / R = K ′ ₃ are set, (13) to (15) are expressed as (16) to (18), respectively. be able to.

By solving this with respect to K ′ ₁ , K ′ ₂ , K ′ ₃ , the corneal curvature center position Q ′ ₁ (vector q ′ ₁ ), Q ′ normalized by the distance R from the center of eyeball rotation to the corneal curvature center. ₂ (vector q ′ ₂ ) and Q ′ ₃ (vector q ′ ₃ ) can be obtained respectively.

Since the normalized corneal curvature center E ′ (vector e ′) is equidistant from Q ′ ₁ , Q ′ ₂ , and Q ′ ₃ , E ′ (vector e ′) is determined from the following equation. Of the two calculated points, a point farther from the camera is E ′.
| Vector e′−vector q ′ ₁ | ² = | vector e′−vector q ′ ₂ | ²
= | Vector e′−vector q ′ ₃ | ² = 1 (19)

It is sufficient to know E ′ (vector e ′) to obtain the line-of-sight direction, but if the radius R is given, the actual eyeball rotation center position E (vector e ′) can also be obtained from the following equation. It's ok.
e = R · e '(20)

Subsequently, the corneal curvature center position is calculated. A normalized corneal curvature center position Q ′ (vector q ′) at an arbitrary line-of-sight position is a sphere having a radius of 1 centered on the normalized center of eyeball rotation and a straight line m obtained from two reflection images. Since K ′ is obtained from the intersection, Q ′ (vector q ′) is determined.
| Vector q′−vector e ′ | ² = 1 (21)
Vector q ′ = K ′ · vector v (22)
A point closer to the camera among the two calculated intersections is defined as Q ′.

From the above, the line-of-sight direction vector g ′ is obtained from Equation 23.
Vector g ′ = vector q′−vector e ′ (23)

  Therefore, when the subject is looking at an arbitrary point, a straight line that passes through the corneal curvature center by the above-described method is obtained, and the distance between this straight line and the center of eyeball rotation to the corneal curvature center is normalized by the eyeball radius. Is obtained, and a straight line connecting the intersection and the eyeball rotation center is obtained, so that the visual line direction of the subject can be measured.

If the eyeball radius R is given, the line-of-sight vector g connecting the center of corneal curvature and the center of eyeball rotation can also be obtained by Expression 24 using the vector g ′.
Vector g = R · Vector g ′ (24)

  In the line-of-sight measurement method according to the present embodiment, the eyeball rotation center position necessary for the line-of-sight calculation is obtained by gazing at three points with known visual angles, so that calibration is easy.

  In addition, since the line of sight can be measured as long as the positions of the two cornea reflection images can be detected, the burden of image processing is reduced, and as a result, high speed and low cost can be expected.

  Further, in the line-of-sight measurement method according to the present embodiment, it is only necessary to have one camera and two light sources, and it is easy to reduce the size of the measurement system, and these are integrated with eyeglasses and HMD (Head Mount Display). Thus, it becomes possible to detect the line of sight without deteriorating the line-of-sight position accuracy even during head movement.

(Verification experiment)
The experimental environment is shown below. A macro lens (VS Technology, VS-LD25) was attached to a CCD camera (manufactured by Sony Corporation, XC-EI50), and a cornea model (optical glass sphere having a diameter of 15.4 mm) was photographed.

  In order to reproduce the horizontal and vertical rotation of the eyeball, the optical stage is a horizontal rotation stage (Sigma Kogyo Co., Ltd., capable of rotating ± 180 deg in the horizontal direction) and a gonio stage (Sigma Sigma Co., Ltd., vertical rotation of ± 30 deg). Possible).

The point where the rotational axes of both stages coincided was taken as the center of rotation of the eyeball, and based on physiological knowledge, a glass ball was placed 5.3 mm ahead of the center of eyeball rotation.

  The front 0 deg direction of the horizontal rotation stage was the optical axis direction of the camera, and the rotation plane was made to coincide with the horizontal plane of the camera.

  Further, near-infrared LEDs (manufactured by Hamamatsu Photonics Co., Ltd., wavelength 880 nm, emission diameter 160 μm) were installed at the top and side of the camera, respectively, and the reflected light from the glass sphere was photographed through a filter that cuts the visible light region.

  Images captured at 30 frames per second (resolution: 640 × 480 pixels) were taken into an image processing computer via a capture board (Buffalo, Inc., CBP-AV), and the position of the reflected image was calculated.

  Then, the visual axis rotation angle was calculated from the position of the reflected image when the optical glass sphere was rotated horizontally and vertically, and the positional accuracy was verified.

  Next, the verification method is shown below. The corneal model was rotated within the range of horizontal ± 40 deg and vertical ± 30 deg by the rotation stage and the gonio stage, and the line-of-sight position was measured. Each stage was rotated in 10 deg increments in the horizontal and vertical directions, the light source reflection image positions at a total of 63 line-of-sight positions were measured, and the line-of-sight direction was determined for each.

  In this verification, since the visual line direction was given by the horizontal angle and the vertical angle using the optical stage, the visual line position accuracy was evaluated by the angle. The horizontal angle was an angle formed by a vector projected from the gaze vector measured on the horizontal rotation plane of the stage and a reference vector (vector in the front 0 deg direction), and the vertical angle was an angle formed by the horizontal rotation plane and the measured vector.

  FIG. 11 shows the line-of-sight angle when the optical stage is manually rotated from −10 degrees to 10 degrees in the horizontal direction and the vertical direction, respectively. The average value and standard deviation of the line-of-sight angle for 1 second (30 frames of frame numbers 2402 to 2430) in the steady state after rotation were 10.001 ± 0.008 deg in the horizontal direction. The vertical direction was 0.048 ± 0.008 deg.

  FIG. 12 shows the line-of-sight angles examined in the range of horizontal ± 40 deg and vertical ± 30 deg. The rotation angle of the optical stage in the horizontal direction is referred to as a horizontal angle, and the rotation angle of the optical stage in the vertical direction is referred to as a vertical angle.

  The grid lattice points in FIG. 12 are the rotation angles of the optical stage. The detected line-of-sight angle is plotted. The detected line-of-sight angle almost coincided with the rotation angle of the optical stage in both the horizontal angle and the vertical angle.

Table 2 shows the maximum line-of-sight angle error, average error, and standard deviation.

  The average error in the range of the line-of-sight angle in the range of horizontal angle ± 20 deg and vertical angle ± 20 deg was 0.187 deg in the horizontal direction and 0.200 deg in the vertical direction. Even when the range of the line-of-sight angle was expanded to a horizontal angle of ± 40 deg and a vertical angle of ± 30 deg, the average error was less than 0.3 deg in the horizontal direction and the vertical direction, and no significant increase in error was observed. From this, it can be seen that the gaze position accuracy obtained by the gaze measurement method according to the present embodiment is high.

  Since it is a gaze measurement method that is easy to calibrate and has little burden on the subject, it can be used in a wide range of fields such as basic research in neuroscience, evaluation of psychological state, and input interface to PC devices.

DESCRIPTION OF SYMBOLS 1 Line-of-sight measurement apparatus 2 Line-of-sight measurement apparatus 10 Imaging apparatus 11 Arithmetic apparatus 12 Transmission line 13 Glasses 13a Glasses lens 14 Light source 15 Light source

Claims (5)

The eyeball before and after the subject's rotation is photographed in advance by one imaging device, and three or more arbitrary feature points on the eyeball surface are extracted from the captured image, and based on the three-dimensional position of the feature point before and after the eyeball is rotated. to previously obtain the positional relationship between the feature point and the eyeball rotation center Te,
The subject gazes at any three points of interest whose viewing angles between the points are known in order, and when moving between the two points of interest, a predetermined point on the eyeball surface is the eyeball. A first locus circle, a second locus circle, and a third locus circle that move around the rotation axis are obtained, and the first locus circle, the second locus circle, and the third locus circle are obtained. Is obtained as a gaze vector tip, and a gaze vector connecting the eyeball rotation center and the gaze vector tip is obtained as the gaze direction of the subject.
The positional relationship between the line-of-sight vector and the feature point is obtained,
The eyeball is imaged by the imaging device, and the gaze direction of the subject is measured by obtaining the gaze vector based on the position of the feature point on the captured image.
A gaze measurement method characterized by the above.   The eye gaze measurement method according to claim 1, wherein a branch point of a blood vessel is extracted as the feature point. One imaging device for imaging the eyeball of the subject;
An arithmetic device that measures the gaze direction of the subject based on the image captured by the imaging device,
The arithmetic unit is:
Three or more arbitrary feature points on the eyeball surface are extracted from an image obtained by imaging the eyeball before and after the rotation of the subject in advance, and the feature point and the eyeball rotation center are determined based on the three-dimensional position of the feature point before and after the eyeball rotation. When the subject moves around two points of interest in order to make a round of arbitrary three points of interest where the viewing angle between the points is known, it moves on the surface of the eyeball. A first trajectory circle, a second trajectory circle, and a third trajectory circle in which a predetermined point moves around the eyeball rotation axis are obtained, and the first trajectory circle, the second trajectory circle, and Means for obtaining an intersection of the third trajectory circle as a gaze vector tip, obtaining a gaze vector connecting the eyeball rotation center and the gaze vector tip as a gaze direction of a subject, and obtaining a positional relationship between the gaze vector and the feature point When,
That having a means for measuring a gaze direction of the subject seeking the gaze vector based on the position of the feature point on the image of the captured eye by the image pickup device,
A line-of-sight measurement device characterized by that. The subject gazes at three points whose viewing angles are known, and each point irradiates the subject's cornea with light from the first light source, and reflects the reflected light that has passed through the optical center of the lens after being reflected by the cornea. An image is projected onto the imaging region, a first plane including the first light source, the optical center of the lens, and the first reflected image is obtained, and the cornea is irradiated with light from the second light source. The reflected light that has passed through the optical center of the lens is projected onto the imaging region as a second reflected image, and includes a second light source, an optical center of the lens, and a second reflected image that includes the second reflected image. Determining a plane, and determining the line of intersection of the first plane and the second plane as a straight line passing through the corneal curvature center;
Assuming a first triangular pyramid having a triangle consisting of three corneal curvature centers with the center of eyeball rotation as the bottom, and a bottom having a triangle consisting of the corneal curvature center with three points as the vertex and the optical center of the lens. Analysis of the first triangular pyramid and the second triangular pyramid having the same triangle as the bottom by normalizing the distance from the eyeball rotation center to the corneal curvature center Performing the step of obtaining the position of the eyeball rotation center,
Gaze the subject at an arbitrary point, perform a step of obtaining a straight line passing through the corneal curvature center, and obtaining a straight line passing through the corneal curvature center;
Obtain the straight line connecting the intersection of the sphere centered on the eyeball rotation center normalized by the eyeball radius and the distance from the center of eyeball rotation to the center of corneal curvature, and measure the gaze direction of the subject A process comprising:
A gaze measurement method characterized by the above. A first light source and a second light source for irradiating the subject's cornea with light;
The light irradiated from the first light source and the second light source is reflected by the cornea and reflected light passing through the optical center of the lens is projected into the imaging region as a first reflected image and a second reflected image, respectively. An imaging device to
The subject gazes at three points whose viewing angles are known, and for each point, the first light source, the optical center of the lens, the first plane including the first reflected image, and the second A light source, an optical center of the lens, and a second plane including the second reflected image are obtained, and a line of intersection between the first plane and the second plane is defined as a straight line passing through the corneal curvature center. Assuming that the first triangular pyramid has a triangular shape with the center of eyeball rotation as the apex and the triangle composed of three corneal curvature centers as the bottom, and the triangle with the optical center of the lens as the apex and consisting of the three corneal curvature centers as the bottom Assuming a second triangular pyramid, and normalizing the distance from the center of eyeball rotation to the corneal curvature center, the first triangular pyramid having the same triangle as the bottom surface and the second triangular pyramid Geometric analysis is performed to find the position of the eyeball rotation center Means, and when the subject gazes at an arbitrary point, the first plane and the second plane are obtained to obtain a straight line passing through the corneal curvature center, and the obtained straight line and the eyeball rotation center to the corneal curvature center are obtained. An arithmetic unit having a means for obtaining a straight line connecting an intersection of a sphere centered on an eyeball rotation center with a distance normalized by an eyeball radius and an eyeball rotation center, and measuring a subject's gaze direction,
A line-of-sight measurement device characterized by that.

Images