JP2007029126A - Line-of-sight detecting device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve detection precision and to simplify processing in a line-of-sight detecting device for detecting a user's line of sight. <P>SOLUTION: An imaging direction of an imaging device is taken as the depth direction, and two light sources are disposed respectively in the lengthwise direction and the lateral direction of the center of a lens in a plane including the center of the lens. An eye ball image 5 including a user's pupil image and two reflection images p1i, p2i of the light sources is acquired. Then, an intersection of a straight line connecting the center of the lens and the center of a corneal curvature with the imaging device is taken as a reference point ri, and the ordinate of the reflection image of the light source placed in the lateral direction of the imaging device and the abscissa of the reflection image of the light source placed in the lengthwise direction are respectively taken as the ordinate and abscissa of the reference point. The direction of the line of sight is detected from the positional relationship between the pupil image and the reference point. Since coordinates of the reference point are determined only by obtaining the coordinates of the reflection images of the light sources, no calculation errors due to the calculation of the reference point are made, and the processing is simplified. Moreover, a distance between the eyeball and the imaging device is calculated from a distance Dp between the reflection images of two light sources. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

The present invention relates to an apparatus for detecting a user's line of sight, and in particular, a line of sight using a user's eyeball image acquired by the image pickup device and a reflection image of the light source on the eyeball obtained by arranging a light source around an imaging device such as a camera. The present invention relates to a detection device.

Various methods and apparatuses for calculating the rotation angle of the user's eyeball from the image information of the user's eye obtained from an imaging device such as a camera and detecting the direction of the line of sight have already been proposed.

A basic method of eye gaze detection will be described with reference to FIGS. FIG. 2 defines a coordinate system of the imaging device 1 such as a camera.
As shown in the left figure, the z-axis is taken with the imaging direction of the imaging device as the axis in the depth direction, the horizontal axis on the plane including the lens center is the x-axis, and the vertical axis is the y-axis.
Each axis intersects vertically. The right figure is a view of the imaging device as seen from the front. FIG. 3 is a diagram for explaining the principle of the gaze detection method.
The imaging device 1 and the user's eyeball 3 are viewed from above (y-axis positive direction).
The distances from the lens center o of the lens 2 of the imaging device to the imaging device 1 and the eyeball 3 are respectively represented as F0 and F1, and the corneal curvature radius is represented as Rc.
In FIG. 3, when the intersection between the user's line of sight 4 and the corneal surface is e, the user's line of sight direction is represented by a vector <ce> (hereinafter referred to as a line of sight vector), and the line of sight is a straight line connecting these points. ce (hereinafter referred to as line of sight ce).
(Here, in the text, the vector notation from point a to point b is <ab>.)
Various objects representing the intersection point e between the user's line of sight and the corneal surface have been proposed, but a typical one is the center position of the pupil or iris on the corneal surface.
Further, since the pupil is inside the cornea and a refracted image appears on the surface of the cornea, there is a method in which the center of the pupil considering refraction is set as the intersection point e (Non-patent Document 1).
In the following description, the center of the pupil will be described as the intersection point e between the line of sight and the corneal surface. In the gaze detection, this gaze vector <ce>
The purpose is to derive the sight line ce.

The line-of-sight vector <ce> is based on the lens center o as <ce> = <oe> − <oc> (1)
And a vector <oe> from the lens center o to the pupil center e and a vector <oc> from the corneal curvature center c.
Here, the intersection point between the segment oc connecting the lens center o and the corneal curvature center c and the corneal surface is defined as a reference point r,
The points corresponding to the pupil center and the reference point in the image projected on the imaging device through the lens system, respectively,
ei = (Eix, Eiy, Eiz)
ri = (Rix, Riy, Riz)
It expresses. Here, Eiz = Riz = −F0.
Since the corneal curvature center c and the reference point ri on the imaging device are on the same straight line passing through the lens center o, the vector <oc>
<Oc> = {(F1 + Rc) / F0} × <rio>
It can be expressed.
The pupil center e on the corneal surface and the pupil center ei on the imaging device are also on the same straight line passing through the lens center o.
Here, if the straight line extending from the lens center in the z-coordinate direction and the corneal curvature center are not far apart, or if the distance F1 from the lens center to the eyeball is sufficiently long, the z-coordinate of the pupil center on the corneal surface is expressed as F1. Can be approximated.
At this time, the vector <oe>
<Oe> = (F1 / F0) × <eo>
It can be expressed.
(If approximation is not used, it is obtained from the intersection of the straight line oei and the corneal surface.)
Therefore, the line-of-sight vector <ce> is obtained from the equation (1):
<Ce> = {(F1 + Rc) / F0} × <ori> − (F1 / F0) × <oei> (2)
It can be expressed.

Each parameter in equation (2) is verified below.
The distance F0 between the lens center and the imaging device is predetermined.
The distance F1 between the center of the lens and the eyeball image is determined by restraining the user to a specified value by fixing the user's head or by directly measuring F1 using an imaging device, a measuring apparatus, or the like.
The corneal curvature radius Rc can be used because an average human value is known anatomically.
The pupil center ei on the imaging device is detected from the eyeball image acquired by the imaging device.
As shown in FIG. 4, the user's eyeball image 5 is displayed on the imaging device through the lens.
Since the projected image is inverted by the lens, the coordinate system is inverted 180 degrees around the z axis in FIG.
The point ei can be detected from the image information of the pupil 6 and the iris 7 in the eyeball image.
The last remaining parameter necessary for deriving the line-of-sight vector is the reference point ri on the imaging device.
Therefore, how to obtain the reference point ri on the imaging device is a problem in detecting the line of sight.

A technique called corneal reflection method is widely used for deriving the reference point.
In this method, a light source is installed on the imaging device side, and a reflection image (Purkinje image) on the cornea of the light source is acquired together with an image of the eyeball by the imaging device, and a reference point for calculating a line of sight is derived from the reflection image. It is a technique.

The most direct method is to use a half mirror shown in Patent Document 1.
The method will be described with reference to FIG.
The imaging device 1 and the user's eyeball 3 are viewed from the side surface (x-axis negative direction).
A half mirror 8 is inserted at an angle of 45 degrees between the eyeball and the lens 2, and a light source 9 is installed on the half mirror 8. The light from the light source is reflected by the half mirror and a Purkinje image p is formed on the cornea. At this time, the Purkinje image coincides with the reference point.
Therefore, the line-of-sight vector can be derived by substituting the Purkinje image pi projected on the imaging device into the equation (2) as the reference point ri.
However, in the case of this method, light from the light source is irradiated near the center of the pupil.
Usually, infrared light is used as the light source.
In theory, infrared light can be detected by an imaging device, but cannot be perceived by humans.
However, it is known that humans feel stress when the power of infrared light is strong.
Therefore, it is a problem that light from the light source is irradiated near the center of the pupil.

Therefore, the technique as in Non-Patent Document 1 adjusts the position of the light source so that light from the light source does not enter the vicinity of the pupil.
In Non-Patent Document 1, as shown in FIG. 6, the position of the reference point and the Purkinje image is intentionally shifted by installing the light source 9 below the imaging device (in the negative y-axis direction).
The method will be described with reference to FIGS.
The lower diagram in FIG. 7 is a view of the imaging device 1 and the user's eyeball 3 as viewed from the side (in the negative x-axis direction).
7 shows the imaging device 1 and the user's eyeball 3 as viewed from above (y-axis positive direction).
In the upper and lower figures, the z-axis direction is aligned.
FIG. 8 shows the user's eyeball image 5 and Purkinje image pi projected on the imaging device.
Since the light source 9 is installed in the negative y-axis direction from the lens center o as shown in the lower diagram of FIG.
The Purkinje image p is shifted downward from the reference point r.
This deviation Δp is approximately when the distance H from the lens center to the light source, the corneal curvature radius Rc, and the distance F1 from the lens center to the eyeball is sufficiently larger than Rc,
Δp≈ (Rc × H) / 2F1
Can be derived as follows.
Therefore, the y coordinate Riy of the reference point on the imaging device is obtained from the y coordinate Piy of the Purkinje image on the imaging device.
Riy≈Py− (F0 / F1) Δp
Can be approximated.
On the other hand, in the x-axis direction, as shown in the upper diagram of FIG. 7, when the light source 9 is installed in the negative y-axis direction from the lens center, the line segment from the lens center to the corneal curvature center in the xz plane The line segment from the light source to the corneal curvature center matches.
This is always true even if the position of the eyeball moves.
Therefore, the x coordinate Rix of the reference point on the imaging device is equal to the x coordinate Pix of the Purkinje image on the imaging device.
Therefore, as shown in FIG. 8, the reference point ri exists immediately above the Purkinje image pi on the imaging device (in the negative y-axis direction).

A method using two light sources as in Patent Document 2 is also considered.
In Patent Document 2, the light source 9 is installed on both sides (x-axis direction) of the imaging device 1 as shown in FIG.
The method will be described with reference to FIGS.
FIG. 10 is a view of the imaging device 1 and the user's eyeball 3 as seen from above (y-axis positive direction).
FIG. 11 shows a user's eyeball image 5 and two Purkinje images p1i and p2i projected on the imaging device.
When the light source 9 is installed at the same distance G on the left and right of the lens center o, Purkinje images are formed on the left and right p1 and p2 with the reference point r in between.
The x coordinate Rx of the reference point approximately matches the midpoint of the x coordinates P1x and P2x of the two Purkinje images.
At this time, similarly to the reference point on the cornea, the x coordinate Rix of the reference point on the imaging device is also Rix≈ (P1ix + P2ix) / 2.
And approximately coincides with the midpoint between the two Purkinje images P1ix and P2ix on the imaging device.
On the other hand, the y-axis direction is equivalent to the situation in which the x-axis and the y-axis are interchanged in the upper diagram of FIG.
That is, when the light source is arranged on the x axis passing through the center of the lens, the y coordinate Riy of the reference point on the imaging device is equal to the two Purkinje images P1iiy and P2iiy on the imaging device.
Therefore, as shown in FIG. 11, the two Purkinje images p1i and p2i on the imaging device are arranged in parallel with the x axis, and the midpoint thereof is the reference point ri.

Furthermore, in the method of Patent Document 2,
The distance F1 from the center of the lens to the eyeball (exactly where the Purkinje image is generated) is:
F1 × (F1 + Rc) ≈F0 × G × Rc / | P1ix−P2ix | (3)
Satisfy the relationship.
That is, the interval | P1ix−P2ix | between two Purkinje images on the imaging device
From this, the distance F1 between the lens center o and the eyeball can be derived.

Takehiko Ohno, Naoki Takegawa, Atsushi Yoshikawa, “Gaze Measurement System Realizing Simple Calibration with Two-Point Correction”, Transactions of Information Processing Society of Japan, Vol. 44, no. 4, pp. 1136-1149, 2003. JP S61-172552 A JP H02-264632 A

The methods of Non-Patent Document 1 and Patent Document 2 described in the background art are easily affected by measurement errors and calculation errors.

In the case of the method of Non-Patent Document 1,
When deriving the reference point from the Purkinje image, distance information from the light source or lens center to the eyeball is necessary for the y coordinate.
Although it depends on the distance information measuring means, a measurement error is likely to occur in this information.
Moreover, since an approximate expression is used, a calculation error also occurs.

In the case of the method of Patent Document 2, it is considered that the influence of the measurement error is small because the distance information from the light source and the lens center to the eyeball is not necessary as compared with the method of Non-Patent Document 1.
However, strictly speaking, the midpoint of the two Purkinje images is equal to the reference point only when the corneal curvature center c exists on a straight line passing through the lens center and parallel to the z-axis.
Therefore, an approximation error occurs in the x-coordinate of the reference point derived as the center of the corneal curvature, that is, the eyeball position moves away from the straight line.
Of course, a method that does not use the above approximation by the midpoint is also conceivable. However, distance information from the light source and the center of the lens to the eyeball is necessary, and a new problem of high calculation cost arises.

Attention was paid to the method of deriving the x-coordinate Rix of the reference point on the imaging device in Non-Patent Document 1 and the y-coordinate Riy of the reference point on the imaging device in Patent Document 2.
That is, in Non-Patent Document 1, as shown in the upper diagram of FIG. 7, when the light source 9 is installed shifted from the lens center o in the y-axis direction, a line from the lens center to the corneal curvature center in the xz plane. The line segment from the light source to the corneal curvature center geometrically matches.
Therefore, the x coordinate Rix of the reference point on the imaging device 1 is equal to the x coordinate Pix of the Purkinje image on the imaging device.
Similarly, in Patent Document 2,
In the case where the light source is installed shifted from the lens center in the x-axis direction, the line segment from the lens center to the corneal curvature center and the line segment from the light source to the corneal curvature center in the yz plane geometrically match. Therefore, the y coordinate Riy of the reference point on the imaging device is equal to the x coordinate Piy of the Purkinje image on the imaging device.
That is, when the light source is installed in the y-axis direction from the center of the lens, the x coordinate of the reference point on the imaging device is equal to that of the reference point on the imaging device when the light source is installed in the x-axis direction from the lens center. The y-coordinate can be correctly derived without approximation only by detecting the Purkinje image.

Therefore, in the present invention,
Two light sources are arranged so as to be shifted from the lens center in the x-axis direction and the y-axis direction, respectively, and the coordinate value of the Purkinje image on the imaging device corresponding to each light source is detected.
Then, the y coordinate of the Purkinje image corresponding to the light source shifted in the x-axis direction is set as the y coordinate of the reference point on the imaging device, and the x coordinate of the Purkinje image corresponding to the light source shifted in the y-axis direction is the reference on the imaging device. Let it be the x coordinate of the point.

In the present invention,
Since the coordinate value of the Purkinje image on the image pickup device becomes the coordinate value of the reference point on the image pickup device as it is, numerical calculation for deriving the reference point is not required.
Therefore, the reference point detection procedure is simple and the detection speed is fast.
Moreover, since a recent calculation is not used, a calculation error does not occur.
Therefore, the reference point detection accuracy is high.

FIG. 12 shows the positional relationship between an imaging device and a light source according to an embodiment for carrying out the present invention.
For example, as shown in the figure, the light sources 9 are respectively arranged just beside the lens center o of the imaging device 1 (x-axis negative direction) and directly below (y-axis negative direction).
As the light source, a projector such as a light emitting diode capable of emitting infrared rays that does not cause discomfort to the user is used.
An imaging device that can respond to the infrared light region is used.
Furthermore, by attaching a filter that suppresses a response other than infrared light to the lens of the imaging device, the influence of illumination (such as natural light) other than the projector can be suppressed.

When the imaging device is directed toward the user's eyeball, the user's eyeball image 5 and two Purkinje images p1i and p2i corresponding to the light sources in the x-axis direction and the y-axis direction are displayed on the imaging device as shown in FIG.
As described in “Means for Solving the Problem”, the y coordinate Riy of the reference point on the imaging device is equal to the y coordinate P1ii of the Purkinje image p1i corresponding to the light source in the x-axis direction.
In addition, the x coordinate Rix of the reference point on the imaging device is also equal to the x coordinate P2ix of the Purkinje image p2i corresponding to the light source in the y-axis direction.
Therefore, the coordinate values P2ix and P1ii of the two Purkinje images on the imaging device become the coordinate values of the reference point ri on the imaging device as they are.

Further, when the interval between the two light sources is D0 and the interval between the two Purkinje images on the imaging device is Dp, the distance F1 from the lens center to the eyeball is similar to the case of Equation (3) in Patent Document 2,
F1 × (F1 + Rc) ≈F0 × D0 × Rc / Dp (4)
Satisfy the relationship.
That is, the distance F1 from the center of the lens to the eyeball can be derived from the distance D0 between the two light sources and the distance Dp between the two Purkinje images.
Here, the distance D0 between the two light sources is the distance from the lens center o to the light source 9 in the x-axis direction G,
If the distance from the lens center o to the light source 9 in the y-axis direction is H,
D0 = √ (G2 + H2)
And the distance Dp between the two Purkinje images is
Dp = √ {(P1ix−P2ix) 2+ (P1ii−P2ii) 2}
Is calculated.
Similarly, F1 can be derived from the distance between the x coordinates and the distance between the y coordinates of two Purkinje images.

FIG. 13 shows an embodiment when the line-of-sight detection apparatus according to the present invention is applied to a pointing device based on line-of-sight input.
This embodiment will be described with reference to this figure.

As shown in FIG. 13, the line-of-sight detection apparatus according to the present invention is composed of an imaging device such as a camera 1, two light sources 9, and an arithmetic unit 10, and when the present invention is applied to a pointing device based on line-of-sight input, A computer 11 and a display 12 are included.
The user turns his / her face from the front of the imaging device to the imaging device.
First, infrared light is irradiated toward the user's eyes 3 from the two light sources on the lateral side and the lower side of the image pickup device installed as shown in the “best mode”.
A user's eyeball image 5 and two Purkinje images p1i and p2i are displayed on the imaging device.
In the arithmetic device, the pupil region and the iris region are extracted, and the pupil center ei is specified from the information.
Further, the region of the Purkinje image is extracted and the position is specified.
At this time, the position of the Purkinje image can be specified with accuracy (subpixel) exceeding the resolution of the imaging device by using image processing such as centroid calculation.
Then, the coordinate values of the two Purkinje images on the imaging device are directly applied to the coordinate value ri of the reference value on the imaging device.
Then, a line-of-sight vector is derived according to Equation (2).
At this time, the distance F1 between the lens center and the eyeball may be derived by Expression (6), the user's head may be fixed, or F1 may be directly measured by another measuring device. .
Then, a straight line 4 representing the line of sight is obtained from the line-of-sight vector.
The intersection of the line of sight 4 and the plane representing the display 12 is the position 13 at which the user is gazing.
By converting the gaze position into a coordinate value on the computer and setting it as the coordinate value of the cursor 14 indicating the position that can be input on the display, the position control of the cursor by the gaze input can be performed.
This cursor coordinate value is sent to the computer 11, and the computer displays the cursor 14 on the display 12.

Application to the operation of an object based on the line-of-sight direction, such as a pointing device based on the line-of-sight input of the embodiment, or based on the line-of-sight direction of the user built in the camera finder, which is also cited in Patent Document 2 As in the case of photographing, it can be applied to processing of images and devices based on the line-of-sight direction.

It is the figure showing the eyeball image and the Purkinje image which were projected on the imaging device of the form example for implementing this invention. FIG. 3 is a three-dimensional view (left) and a front view (right) of an imaging device used for eye-gaze detection and its coordinate system. It is the figure which looked at the apparatus system for demonstrating the principle of a basic gaze detection method from the upper surface. It is a figure showing the eyeball image projected on the image pick-up device by a basic look calculation method. It is a figure from the side of an apparatus system for explaining a gaze detection method of patent documents 1. 3A and 3B are a three-dimensional view (left) and a front view (right) illustrating a positional relationship between the imaging device and the light source of Non-Patent Document 1. It is the figure (bottom) which looked at the apparatus system for demonstrating the gaze detection method of a nonpatent literature 1 from the side, and the figure (top) seen from the upper surface. It is the figure showing the eyeball image and the Purkinje image which were projected on the imaging device of nonpatent literature 1. They are the three-dimensional figure (left) and the front view (right) showing the positional relationship of the imaging device of patent document 2, and a light source. It is the figure which looked at the apparatus system for demonstrating the gaze detection method of patent document 2 from the upper surface. It is the figure showing the eyeball image and the Purkinje image which were projected on the imaging device of patent document 2. FIG. It is a figure showing the positional relationship of the imaging device and light source of the form example for implementing this invention. It is a figure showing the Example at the time of applying the gaze detection apparatus which concerns on this invention to the pointing device by gaze input.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Lens of imaging device 3 User's eyeball 4 User's line of sight 5 User's eyeball image projected on imaging device 6 User's pupil image projected on imaging device 7 Use projected on imaging device Iris image 8 Half mirror 9 Light source 10 Gaze direction computing device 11 Computer 12 Computer display 13 User gaze position 14 Cursor position

Claims (1)

The imaging direction of the imaging device is the depth direction, and two light sources are installed on the plane including the lens center in the vertical direction and the horizontal direction of the lens center, respectively, and the imaging device includes the pupil image of the user and the reflected images of the two light sources. An image acquisition means for acquiring an ocular image, and an ordinate of a reflection image of a light source installed in a lateral direction of the imaging device, with an intersection point on a linear imaging device connecting a lens center and a corneal curvature center as a reference point, and A line-of-sight detection method for detecting a line of sight from the positional relationship between the pupil image and the reference point, with the abscissa of the reflected image of the light source installed in the vertical direction as the ordinate and abscissa of the reference point, and the two reflections A line-of-sight calculation apparatus comprising: a distance measuring unit that calculates a distance between the user's eyeball and the imaging device from a distance between images.

Images