JP4055858B2 - Gaze detection system - Google Patents

Description

  The present invention relates to a gaze detection system for obtaining a gaze vector for a camera.

Conventionally, a simple gaze position measurement system for measuring a gaze position has been demanded. If such a system can be provided at a lower price, it can be used, for example, in the following fields.
1) Psychological research on vision
2) Medical examination
3) Man-machine interface (line-of-sight input device)
4) Although it overlaps with 3), it is an auxiliary device for people with disabilities.
5) Built in the head-mounted display and used as a line-of-sight input device.
6) Passive calibration of binocular parallax by using two devices and setting the position of the camera strictly (instead of presenting the target and staring, it just moves the eyes) Only). It can be used for medical examinations, psychological experiments, behavioral experiments, etc.

Conventionally, however, multiple images are captured by a camera, the reflected light shape and pupil shape are measured, and the center of gravity position of the reflected light, the inclination of the minor axis, the minor axis major axis ratio, and the pupil center are calculated from the elliptical approximation of the pupil. There has been no simple gaze position measurement system that measures the gaze position using the above method.
Japanese Patent No. 2600114 IEICE technical report vol.100 no.47 “Development of gaze position measurement system” ISSN0913-5685, TL2000-1-6 “Thinking and Language”, May 12, 2000

  The present invention captures a plurality of images with a camera, measures the reflected light shape and pupil shape of each image, and approximates the reflected light barycentric position and the pupil to an ellipse, thereby reducing the minor axis inclination and the minor axis major axis. The purpose is to obtain a line-of-sight vector for the camera by calculating the ratio and pupil center.

(1) Regarding the relationship between the position of the center of gravity of the reflected light and the center position of the corneal curvature.
Take a human or monkey eyeball with a CCD camera. To illuminate the pupil darkly, illumination is performed with an infrared LED. FIG. 1 shows the positional relationship between the camera, the light source (infrared LED), and the eyeball. In FIG. 1, the camera is illustrated as being positioned approximately in front of the eyeball in the horizontal direction, but the position of the camera can be set at an arbitrary position within a range where the pupil can be photographed. The light source can also be placed at any location where the reflected light shape is on the cornea. The CCD camera captures an image as shown in FIG.

  FIG. 3 is a diagram in which FIG. 1 is simplified and the pupil center and the corneal curvature center are entered. In the simplified FIG. 3, the place where the reflected light barycentric position occurs is shown in FIG. The position of the center of gravity of the reflected light is generated at the position where the cornea and the bisector of the corner connecting the light source, the corneal curvature center, and the camera intersect. A vector connecting the center of corneal curvature and the position of the center of gravity of the reflected light is named a reflected light vector. (It is assumed that the light source is at infinity. The camera is assumed to be parallel projected).

  FIG. 5 shows the positional relationship between the position of the center of gravity of the reflected light and the center of corneal curvature when the eyeball moves. If the light source is sufficiently far away, the angles of the normal vector to the camera surface and the reflected light vector will be the same even if the eyeball rotates. Therefore, even if the eyeball rotates, the distance between the center of corneal curvature projected on the camera and the position of the center of gravity of the reflected light is kept constant. FIG. 5 shows a state where the distance α in the x direction is kept constant. In FIG. 6, even when the line of sight changes, the corneal curvature center projected by the camera is always in the position of α in the x direction and β in the y direction from the center of gravity of the reflected light. FIG. 7 shows the positional relationship of each part of the eyeball viewed from the camera. Assuming that the pupil center, the center of eyeball rotation, and the center of corneal curvature exist on a straight line as shown in FIG. 3, in the projection image of FIG. 7, the center of the pupil (Xp, Yp), the center of eyeball rotation (Xo, Yo), the corneal curvature center (Xk, Yk) exists on the straight line. The reflected light barycentric position (Xr, Yr) is at a position moved by (α, β) from the corneal curvature center. α and β have multiple images, and the straight line connecting the pupil center (Xp, Yp) and the pupil center (Xo, Yo) and the corneal curvature center position (Xk, Yk) = (Xr-α, Yr -β) can be obtained from α and β that minimize the distance.

(2) How to find the center of eyeball rotation-center distance of corneal curvature.
As shown in FIG. 7, the “eyeball rotation center position−corneal curvature center” distance in the image photographed by the camera can be obtained from the corneal curvature center (Xk, Yk) and the eyeball rotation center (Xo, Yo). This distance is a thing in which the actual distance is projected in two dimensions, and is obtained by multiplying the actual distance by a certain constant. The “pupil center-eyeball rotation center” distance in this image can be obtained from the pupil center (Xp, Yp) and the eyeball rotation center (Xo, Yo). Since it is known that the actual “pupil center−center of eyeball rotation” can be obtained (see Patent Document 1), a constant at the time of projection can be obtained from the ratio of the two. By multiplying this constant by the “eyeball rotation center position−corneal curvature center” distance in the image, the actual “eyeball rotation center position−corneal curvature center” distance can be obtained. The relational expression between the actual “eyeball rotation center position-corneal curvature center” distance (rr) and the actual actual “pupil center-eyeball rotation center” (r) is as described later (paragraphs [0021] [0026] reference). (The reflected light barycentric position coordinate is (rx, ry), the deviation of the reflected light barycentric position and the corneal curvature center position in the x direction is expressed as alpha, and the deviation in the y direction is expressed as beta. And instead of Xk, rx (Instead of -alpha (= Xk) and Yk, ry-beta (= Yk) is used.)

3) Gaze direction with respect to the camera.
The direction of the line of sight with respect to the camera can be obtained from the two points on the elliptical short axis extension line in the image taken by the camera (FIG. 7) and the actual distance between the two points. [Expression 3] represents the line-of-sight direction with respect to the camera by the pupil center position (Xp, Yp), the eyeball center position (Xo, Yo), and the pupil center eyeball rotation radius (r). For [Equation 3], instead of the eyeball rotation center (Xo, Yo), the corneal curvature center (Xk, Yk) is used, and the pupil center-eyeball rotation center distance (pupil center rotation radius) (r) is used instead of the pupil center-cornea. By substituting r (pupil center rotation radius) -rr (corneal curvature rotation radius)) at the curvature center distance (2) above, the pupil center (Xp, Yp) and corneal curvature center (Xk, Yk) in FIG. The direction of the line of sight with respect to the camera can be obtained from Similarly, for [Equation 3], instead of the pupil center (Xp, Yp), the corneal curvature center (Xk, Yk) is used, and the corneal curvature is used instead of the pupil center-eyeball rotation center distance (pupil center rotation radius) (r). By substituting the center-to-eye rotation center distance (above 2) rr (corneal curvature rotation radius)), the line-of-sight direction from the center of corneal curvature (Xk, Yk) and the center of eyeball rotation (Xo, Yo) in FIG. You can ask for it.

  The present invention captures a plurality of images with a camera, measures the reflected light shape and pupil shape of each image, and approximates the reflected light barycentric position and the pupil to an ellipse, thereby reducing the minor axis inclination and the minor axis major axis ratio. The pupil center is obtained, and the line-of-sight vector for the camera can be obtained from these values.

  Hereinafter, a first example of the line-of-sight detection method and system of the present invention will be described with reference to FIGS. FIG. 1 is a diagram illustrating a positional relationship among a camera, a light source, and an eyeball. Take a human or monkey eyeball with a CCD camera. To illuminate the pupil darkly, illumination is performed with an infrared LED. Although the camera is illustrated as being positioned in front of the eyeball in a substantially horizontal direction, the position of the camera can be set at an arbitrary position within a range where the pupil can be photographed. The light source can also be placed at any location where the reflection position is on the cornea. The CCD camera captures an image as shown in FIG.

  FIG. 3 is a diagram in which FIG. 1 is simplified and the pupil center, the corneal curvature center, and the eyeball rotation center are entered. In the simplified FIG. 3, the location where the reflection point occurs is shown in FIG. The reflection point is generated at the position where the cornea and the bisector of the corner connecting the light source, the corneal curvature center, and the camera intersect. A vector connecting the corneal curvature center and the reflection point is named a reflected light vector. (It is assumed that the light source is at infinity. The camera is assumed to be projected in parallel.)

FIG. 5 shows the positional relationship between the reflection point and the corneal curvature center when the eyeball moves. If the light source is sufficiently far away, the angle of the reflected light vector to the center of corneal curvature will be the same even when the eyeball rotates. Therefore, even when the eyeball rotates, the distance between the center of corneal curvature and the reflection point is kept constant. FIG. 5 shows only the distance in the x direction. FIG. 6 shows the positional relationship between the corneal curvature center and the reflection point when the image is taken from the camera direction.
The positional relationship among the reflection point, the corneal curvature center, the pupil center, and the eyeball rotation center is shown in FIG. When this diagram is simplified, FIG. 8 is obtained.

The camera captures multiple images, and for each image, the reflected light shape and pupil shape are measured, and the reflected light center of gravity and the pupil are elliptically approximated to determine the minor axis inclination, minor axis major axis ratio, and pupil center. The line-of-sight vector for the camera can be obtained from these values.
i) Measurement of reflected light barycentric position 1) A 120x90 pixel image having the pupil center as the center of the region is taken out every 3 pixels to generate a 40x30 image.
2) Take a point above a certain threshold and label it for each chunk.
3) Measure the center of gravity of the mass that is larger than a certain size and closest to the center of the pupil.
It was done by the method.
However, this reflected light barycentric position measurement method will be missed if the size of the reflection point is small, and if there are multiple reflection points, the wrong reflection point will be selected (on the cornea). There is an nth order Purkinje reflection.

ii) About reflected light selection If all detected ones are used and no selection is made, the reflected light reflected on the sclera, not the cornea, is detected and calculated as shown in FIG. It will end up. In such a case, the assumption that the reflection point is on the cornea is lost, and the calculation result becomes inaccurate. In such a case, it is necessary to make the calculation impossible.

iii) About automatic method switching.
In the above-described method, the line-of-sight position can be obtained from the center position of the pupil of the subject and the center of the reflected light after calibration. However, this method has an advantage that measurement is possible even if the head moves, but if the reflection point moves off the cornea, measurement cannot be performed. The measurement range is narrower than when measuring only from the center of the pupil.

iv) Automatic correction of pupil center-eye rotation center method The above method cannot measure accurately if the head moves. Even when measurement is performed with the head fixed, intermittent displacement may occur, in which case recalibration is required.

v) About multiple use of thresholds when detecting pupils.
The pupil detection method performs detection in the following two stages.
1) The image is sampled every 8 pixels and the approximate position of the pupil is detected.
2) Find a rough ellipse and explore the vicinity to detect the edge of the pupil.
This method uses the same threshold value in both stages. When there is a lot of noise (when there are as many dark places as the pupil), rough detection may fail at the first stage. In such a case, it is possible to reduce noise by lowering the threshold value. However, if such processing is performed, the edge of the pupil may not be detected accurately.

vi) Pupil edge detection method Using the geometrical properties of the ellipse, remove all but the edge of the pupil. An ellipse has the following geometric properties. As shown in FIG. 10, consider three parallel straight lines l, m, n that intersect with an ellipse. Let l and n be equidistant from m. Assume that the intersections of the ellipse and the straight line l are a and b, and the intersections of the straight line n are c and d. Ellipse and straight line m
Let o be the midpoint of the intersection of. If the middle point of the line connecting the midpoints of a and b and the midpoints of c and d is o ', then o' has the property of overlapping o. A straight line m drawn about the center of the pupil image
Find the point corresponding to o'i (i = 1−N) of a set of N parallel lines equidistant from. The obtained points are distributed on the straight line m. When the number of mistakenly detected edges is small enough, the position where the most points o 'are gathered is
Corresponds to the position of. Points that are away from the position include points that are not on the locus of the ellipse, so they are excluded.
It is possible to set an allowable range by specifying in a program how far away they are to be excluded. If four points contain errors in an complementary manner, they will not be excluded, but such a probability is unlikely to occur. Since the points ai, bi, ci, and di having o'i sufficiently close to o are points on the locus of the ellipse, the coordinates of these points can be substituted into [Equation 1] for accurate ellipse approximation. (See Non-Patent Document 1)
By solving the above simultaneous equations, the coefficient of the following elliptic equation [Equation 2] can be obtained by the method of least squares.

vii) Accuracy of pupil detection (rate of noise entering in the horizontal direction).
There was no quantitative evaluation when the reflected light was applied to the edge of the pupil or the edge of the pupil was not clearly captured due to out of focus.

Next, the gaze detection method and system of the present invention will be described.
i) Measurement of reflected light barycentric position 1) FIG. 11 is a diagram for explaining reflected light barycentric position measurement, and is a view of the eyeball seen from the camera direction, and FIG. 12 shows the eyeball in the lateral direction. It is the figure seen from.
Pupil center position (xp, yp), eyeball rotation center position (xo, yo), center of corneal curvature when the cornea is part of the sphere when the cornea is part of the sphere (alpha, beta ), When the pupil center rotation radius (r) and the corneal curvature center rotation radius (rr) are known, the reflected light barycentric position (rx, ry) can be predicted from the following equation.
rx = (xp-xo) * rr / r + xo + alpha
ry = (yp-yo) * rr / r + yo + beta
2) A 40x30 pixel image centered on the predicted point is generated.
3) Take a point above a certain threshold and label it for each chunk.
4) Measure the center of gravity of the mass that is larger than a certain size and closest to the prediction.

Consider an example in which the reflection point centroid is measured from the image shown in FIG. The cross on the screen represents the pupil center position. The white dot on the cross is the first Purkinje image, and the white dot on the lower right is the second Purkinje image. Both are reflections of illumination light, but a plurality of reflected images appear because the reflection location in the eyeball is different.
1) Reflection when the pupil center position (xp, yp), eyeball rotation center position (xo, yo), corneal curvature center (α, β), pupil center rotation radius r, corneal curvature center rotation radius rr are already known. The optical centroid position (rx, ry) is predicted from the above formula.
2) The predicted position is the position X in FIG. The center of reflection point is obtained from a 40x30 pixel image indicated by a square centered on X.
3) Pick out points above a certain threshold and perform labeling, and select the ones with a certain size or larger and the center of gravity closest to the predicted value. In FIG. 15, the center of reflection point is represented by a cross.

ii) Reflected light selection At the time of calibration, find the average of the reflection point width and height. It is possible to accurately perform measurement by eliminating objects that are significantly different from the width and height of the reflection point during measurement. The size of the reflection point shown in FIG. 16 is 6 pixels vertically and 7 pixels horizontally. This is a sufficiently small difference from the average value. How much error is allowed can be set by a program.
As shown in FIG. 17, when the reflected light is present on the sclera, the sizes are greatly different (vertical 19 pixels and horizontal 50 pixels). In this case, the measurement using the reflection point is abandoned.

iii) About automatic method switching.
If it is no longer possible to calculate with the pupil center-reflection position centroid method, the measurement can be interrupted by automatically switching to the pupil center method.
When the reflection point exists on the sclera, it is not possible to measure the line of sight using the reflection point. However, even in this state, the pupil center coordinates can be measured. The line-of-sight direction can be obtained by the pupil center-eyeball rotation center method. The movement of the eyeball rotation center is dealt with by performing correction by automatic correction.

iv) Eye Gaze Vector by Pupil Center-Eyeball Rotation Center Method The gaze direction can be obtained from the pupil center position (xp, yp) and the eyeball rotation center (xo, yo) pupil center rotation radius r. The line-of-sight direction with respect to the camera can be expressed by the following [Equation 3].

v) Automatic correction of pupil center-eye rotation center method When the head does not move, the pupil rotation center (xo, yo) is a constant value. However, no matter how the head is fixed, it may move. Conventionally, when the head has moved, the calibration is performed again, but a method of automatically calibrating this using a reflection point is proposed.
The eyeball rotation center can be obtained from the pupil center position and the reflected light center of gravity. Pupil center position (xp, yp), reflected light centroid position (rx, ry), distance between reflected light centroid position and corneal curvature center position when the cornea is part of the sphere (alpha, beta ), Pupil center rotation radius (r), and corneal curvature center rotation radius (rr), the rotation center of the eyeball (xo, yo) is obtained by the following equation.
xo = (rx-alpha-xp * rr / r) / (1-rr / r)
yo = (ry-beta-yp * rr / r) / (1-rr / r)
The reflection point may not be measurable, and since it is assumed to be intermittent, the center of eyeball rotation (xo) , yo) is the center of eyeball rotation at that time. By using this value, the line-of-sight direction can be calculated using only the pupil center without recalibration. In addition, if the time is sufficiently short, it can be assumed that the head is not moving, so that it can be used as an auxiliary when the pupil center-reflection point method cannot be used.

vi) Multiple use of thresholds when detecting pupils.
The problem is solved by using different threshold values in the first stage and the second stage. If the brightness does not change, an independent fixed value is set.If the threshold is automatically changed, the ratio of the first-stage threshold (th1) to the second-stage threshold (th2) (a ), (Th1 = a * th2 relationship) is set so that continuous changes can be handled.
As shown in FIG. 18, in the case of the threshold value (th2) necessary for detecting the contour, there is much ambient noise.
As shown in FIG. 19, ambient noise can be reduced by lowering the threshold value (th1).

vii) Pupil edge detection method 1) Obtain an ellipse by a conventional method.
Using the ellipse's geometrical properties, remove all but the edges of the pupil. An ellipse has the following geometric properties. Consider three parallel straight lines l, m, n that intersect an ellipse. Let l and n be equidistant from m. Assume that the intersections of the ellipse and the straight line l are a and b, and the intersections of the straight line n are c and d. Ellipse and straight line m
Let o be the midpoint of the intersection of. If the middle point of the line connecting the midpoints of a and b and the midpoints of c and d is o ', o' has the property that it overlaps o (Fig. 10). A straight line m drawn about the center of the pupil image
Find the point corresponding to o'i (i = 1−N) of a set of N parallel lines equidistant from. The obtained points are distributed on the straight line m. When the number of mistakenly detected edges is small enough, the position where the most points o 'are gathered is
Corresponds to the position of. Points that are away from the position include points that are not on the locus of the ellipse, so they are excluded. It is possible to set an allowable range by specifying in a program how far away they are to be excluded. If four points contain errors in an complementary manner, they will not be excluded, but such a probability is unlikely to occur. o
Since points a i, bi, ci, di having o'i sufficiently close to are points on the locus of the ellipse, the coordinates of these points can be substituted into [Equation 1] for accurate ellipse approximation.

2) In the above case, for example, it is assumed that only a12 is not on the locus of the ellipse for some reason. Then, although b12, c12, and d12 exist on the locus of the ellipse, they are not used for calculating the locus of the ellipse. Therefore, after obtaining an elliptical trajectory, the distance from the obtained elliptical trajectory is calculated for the pupil edge point detected again. If the distance is equal to or less than a certain value, the pupil edge is determined.
3) Add a new point and calculate an ellipse by the method of least squares.
This makes it possible to obtain an ellipse with high accuracy.

viii) Measuring method of eyelid engagement 1) As shown in FIG. 20, from the detected edge of the pupil, an ellipse is approximated to obtain the vertical distance (eh) of the ellipse.
2) The difference (ph) between the maximum value and the minimum value in the vertical direction of the detected pupil edge (used for elliptical approximation) is obtained.
3) Let ph / eh * 100 be the pupil detection rate (%).
How much measurement is interrupted when the eyelid is open by using the pupil detection rate? It is possible to make a quantitative evaluation.

ix) Pupil detection accuracy (rate of noise entering in the horizontal direction).
1) Obtain the difference (ph) between the maximum and minimum vertical values of the pupil edge used for the ellipse approximation.
2) Obtain the number of pupil edges used for elliptical approximation (en).
3) Let (en / 2) / ph * 100 be the edge detection rate (%).
Since en detects the left and right edges at the time of detection, it needs to be divided into two equal parts.
By using the edge detection rate, it is possible to quantitatively evaluate how accurately the edge is detected. FIG. 21 with relatively little noise has an edge detection rate of 70%. Further, FIG. 22 in the case where there is a lot of noise is an edge detection rate of 40%.

It is a figure which shows the positional relationship of a camera and eyes. It is a figure which shows the image | photographed eyeball. It is the figure which simplified FIG. It is a figure which shows a reflective point appearance position. It is a figure which shows the positional relationship of the cornea reflected light gravity center when an eyeball rotates, and the cornea curvature center. It is a figure for demonstrating that the positional relationship of a cornea reflected light gravity center and a cornea curvature center does not change even if an eyeball rotates. It is a figure for demonstrating that a pupil center, a corneal curvature center, and an eyeball rotation center exist on the extension line of the short axis of a pupil. FIG. 8 is a simplified diagram of FIG. 7, showing the positional relationship between the corneal reflection light center of gravity, the pupil center, the corneal curvature center, and the eyeball rotation center. It is a figure for demonstrating the detection of the reflected light at the time of reflecting on the sclera instead of on a cornea. It is a figure for demonstrating the detection method of the edge of a pupil. It is a figure for demonstrating reflected light gravity center position measurement, and is the figure which looked at the eyeball from the camera direction. It is the figure which looked at the eyeball shown in FIG. 11 from the horizontal direction. It is an image for demonstrating the measurement of a reflection point gravity center. It is a figure for demonstrating the method of calculating | requiring a reflective point gravity center. It is a figure for demonstrating the method of selecting the thing with a gravity center nearest to a predicted value. It is a figure for demonstrating the magnitude | size of a reflective point. It is a figure for demonstrating the reflected light which exists in a sclera. It is a figure for demonstrating the surrounding noise at the time of detecting an outline. It is a figure for demonstrating the method of reducing ambient noise by lowering | hanging a threshold value (th1). It is a figure for demonstrating the measuring method of the degree of engagement of an eyelid. It is a figure which illustrates the edge detection rate in case there is comparatively little noise. It is a figure which illustrates the edge detection rate when there is much noise.

Claims (3)

In a subject whose head is fixed, means for capturing a plurality of images with a camera capable of photographing a pupil illuminated by a light source arranged so that the reflection position is located on the cornea, and reflected light for each image Measuring the shape and pupil shape, and means for calculating the reflected light barycentric position and the ellipse approximation of the pupil to calculate the inclination of the minor axis, the minor axis major axis ratio, and the pupil center, and the reflected light barycentric position and the cornea System for measuring the distance in the x direction and the distance in the y direction of the corneal curvature center that is the center of the sphere, and the distance between the center of rotation of the eyeball that is a fixed point when the eyeball rotates and the center of corneal curvature . In a subject whose head is fixed, means for capturing a plurality of images by a camera capable of photographing a pupil illuminated by a light source arranged so that the reflection position is located on the cornea;
Means for measuring the reflected light shape and pupil shape of each captured image, and calculating the inclination of the minor axis, the minor axis major axis ratio, and the pupil center by approximating the reflected light center of gravity and the pupil to an ellipse And
Reflected light is measured using the x-direction distance and y-direction distance between the reflected light center of gravity and the corneal curvature center, the distance between the corneal curvature center and the eyeball rotation center, and the distance between the pupil center and the eyeball rotation center. A line-of-sight detection system consisting of obtaining a line-of-sight vector for the camera from the position of the center of gravity and the center of the pupil. In a subject whose head is not fixed, means for capturing a plurality of images by a camera capable of photographing a pupil illuminated by a light source arranged so that the reflection position is located on the cornea;
Means for measuring the reflected light shape and pupil shape of each captured image, and calculating the inclination of the minor axis, the minor axis major axis ratio, and the pupil center by approximating the reflected light center of gravity and the pupil to an ellipse And
Using the distance between the center of gravity of the reflected light and the center of the corneal curvature in the x direction and the distance in the y direction, the distance between the center of the corneal curvature and the center of the pupil, the line of sight from the center of the reflected light and the center of the pupil to the camera A gaze detection system consisting of finding vectors.