JPH05199995A - Pupil photographing device and visual line detector - Google Patents

Abstract

PURPOSE:To detect a visual line with high accuracy while permitting the move ment of a head in a wide range without mounting. CONSTITUTION:The eyeball 30 is irradiated with the near IR light from a light source 5 and the reflected light thereof is imaged to an image pickup image 4 in a camera 1. On the other hand, the light reflected by a half mirror 3 in a lens system is branched to two beams which are then imaged onto a CCD line sensor 11 by respective lenses 10. An out of focus is detected by an out-of- focus detection processing section 12 from the spacing therebetween. The image of the pupil is then picked up by executing focusing and zooming until the spacing is minimized. The rotating angle of the eyeball is calculated in accordance with the image of the image pickup image 4 from the moving quantity of the lenses of this time, by which the visual line detection is executed.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a pupil photographing apparatus for photographing a pupil or a corneal reflection image while focusing on the movement of the image and detecting the position of the pupil or the corneal reflection image, and for detecting the line of sight without wearing the pupil using image processing. The present invention relates to a line-of-sight detection device.

[0002]

2. Description of the Related Art Considering the application of image processing for performing line-of-sight detection by a non-wearing means, it is necessary to extract feature points, detect position of feature points, and estimate eyeball position and rotation angle. Conventionally, a pupil, a corneal reflection image, and the like are often used as the characteristic points of the sight line detection. It is also known that a method of illuminating the eyes and photographing the reflected light with a camera is effective for extracting these. Therefore, if the spatial positions of these feature points can be obtained, they can be converted into the line of sight.

[0003]

By the way, there are various methods for obtaining the spatial position of a feature point, and a typical one is stereo image measurement, but two cameras are required, and
There is a problem that the interface environment of the machine is rather large. Therefore, measurement with a monocular camera is desirable, but in this case, there is a problem that it is difficult to obtain distance information. There is a method of obtaining the distance by combining ultrasonic waves, etc., but sufficient accuracy cannot be obtained.

Further, since the corneal reflection image is photographed by applying illumination, the spatial position of this feature point changes depending on the position of illumination. Therefore, it is important to generate this image at a position where it can be easily converted into the line of sight.

The present invention efficiently illuminates the pupil and the corneal reflection image and changes the adjusting mechanism of the camera so as to focus on this characteristic point. The distance information is calculated from the driving amount of the adjusting mechanism at that time. Then, by estimating the position and rotation angle of the eyeball, a pupil imaging device and a line-of-sight detection device for detecting the line of sight with high accuracy while allowing the movement of the head in a wide range by a non-wearing means are provided. It is to be realized.

[0006]

SUMMARY OF THE INVENTION A pupillary photographing apparatus according to the present invention comprises an illuminating device for irradiating near infrared light forward from near the optical axis of a lens of a camera, and a fundus or corneal surface for illumination by the illuminating device. A lens system that collects the reflected light and forms a pupil image or a corneal reflection image on the imaging surface, and a half mirror that is provided in the middle of the lens system and branches an optical path.
The light reflected by the half mirror is split into two, and each image is formed, and the focus blur detection processing unit that detects the focus blur from the distance between the image formation points and the distance between the image formation points are the smallest. Thus, a focus adjustment drive mechanism for driving the focus adjustment unit of the lens is provided.

Further, a line-of-sight detecting apparatus according to the present invention uses the above-mentioned pupil photographing apparatus, and calculates a distance from a lens movement amount by a focus adjustment driving mechanism to a pupil or a corneal reflection image to be photographed. And an image processing unit that extracts a pupil and corneal reflection image from an image obtained from a pupil imaging device, and an eyeball rotation angle that calculates an eyeball rotation angle from the distance information and position information of the pupil and corneal reflection image on the screen. And a line-of-sight detection unit that detects the line of sight from the distance information and the eyeball rotation angle.

[0008]

In the pupil photographing apparatus of the present invention, by irradiating light from near the optical axis of the lens of the camera, the reflected light from the retina is captured and the pupil can be photographed brightly. Further, since the corneal reflection image formed at this time is generated on the line connecting the camera and the center of curvature of the cornea, the position of the center of corneal curvature can be calculated by determining the position of the light reflected by the cornea, and the line of sight can be easily obtained.

[0009] Further, it is branched by a half mirror, and further 2
Since the focus blur is detected by the imaging system of the reflected light that is branched into two, the focus adjustment unit can control the mechanism unit so that the blur is reduced. Also, from the movement amount of the mechanism unit at this time, Distance information is obtained.

Further, in the visual axis detection device according to the present invention, the corneal reflection image by illumination is used from near the optical axis of the lens of the camera, the camera actively works in auto focus, and the pupil or corneal reflection image is constantly displayed on the screen. By capturing in the center, the vector from the center of the pupil to the corneal reflection image on the photographic screen can be approximated by a linear or quadratic equation with respect to the eye rotation angle, so there are two points on the image (pupil and corneal reflection image)
The line of sight can be calculated from the position of and the distance to the eyeball.

[0011]

FIG. 1 shows an embodiment of the present invention. In the figure, 1 is a camera, 2 is a zoom lens, 3 is a half mirror, 4 is an image sensor, 5 is a light source, 6 is a condenser lens, 7 is a near-infrared band pass filter, and 5-7 are illumination devices. Is configured. 8 is a reflecting mirror, 9 is a near infrared band pass filter, 10 is a lens for a focus blur detection sensor, 11 is a CC
It is a D line sensor, and a defocus detection sensor is composed of 10. Reference numeral 12 is a focus blur detection processing unit, 13 is a focus adjustment drive mechanism, 14 is a zoom drive mechanism, 15 and 16 are movement amount measurement units of the focus and the zoom lens, 17 is a distance calculation unit, and 18 is a pupil and corneal reflection image. Image processing unit, 1
9 is an eyeball rotation angle calculation unit, 20 is a line-of-sight detection unit, 21 is a camera drive control unit, and 30 is an eyeball. A portion 100 surrounded by a dotted line indicates a pupil photographing device, and a reference numeral 200 indicates a line-of-sight detecting device using the pupil photographing device 100. Also,
(A) is an output from the image processing unit 18, and (b) is an output from the distance calculation unit 17, both of which are input to the camera drive control unit 21, and a zoom command (c) from the camera drive control unit 21.
Is input to the zoom drive mechanism 14, and a rotation command (d)
Is a drive mechanism (not shown) for changing the direction of the camera 1.
Added to.

Next, the operation will be described. The light emitted from the light source 5 is reflected by the reflecting mirror 8 and illuminates the human eyeball 30 from the optical axis of the zoom lens. Part of this light enters the eyeball 30 from the pupil, is reflected by the fundus, passes through the pupil again, and is captured by the camera 1. Therefore, the pupil is photographed brightly. Also, some light is reflected on the corneal surface of the eyeball 30, but since the cornea is a convex surface, a virtual image (corneal reflex image) is formed in the eyeball 30.
make. This image is captured by the camera 1. The reflected light captured is split into two by the half mirror 3 in the camera 1. One is captured by the image sensor 4 placed on the image plane, and the image processing unit 18 separates and extracts the pupil and the corneal reflection image.

The other half branched by the half mirror 3 is imaged on the CCD line sensor 11 by the two lenses 10 forming a pair. Since the image becomes large when there is blur, the distance between two points on the CCD line sensor 11 becomes long. When this is detected, the focus adjustment drive mechanism 13 and the zoom drive mechanism 14 are moved so as to reduce the blur. Further, the lens drive system is provided with an encoder, and the movement amount of the zoom lens 2 is measured by the movement amount measuring units 15 and 16. This amount of movement is converted into a distance by collating the amount of movement of the zoom lens 2 previously obtained by calibration and data relating to the distance. The eye rotation angle calculation unit 19 calculates the eye rotation angle based on the distance data obtained as the output of the distance calculation unit 17 and the vector amount from the center of the eye 30 toward the corneal reflection image obtained as the output of the image processing unit 18. To be done.

Further, in the present system, when the spatial position of the eye changes due to the movement of the head, the camera 1 is rotated by the camera drive control unit 21 so as to follow it. The rotation angle (pan, tilt) at this time is determined by the line-of-sight detection unit 20.
Sent to. From the distance data of the distance calculator 17, the rotation angle of the camera 1, and the position data of the characteristic points on the image, the spatial position of the pupil or corneal reflection image, which is the characteristic point, can be obtained by means not shown in the figure.

Since the spatial position of the pupil or the corneal reflection image is near the center of the eyeball, these positions can be approximated to the center of the eyeball, or a predetermined offset can be added to the center position of the eyeball. The line-of-sight detection unit 20 detects the line of sight from the eyeball center position and the rotation angle of the eyeball rotation angle calculation unit 19.

FIG. 2 shows an example of a visual axis detection method when the positions of the pupil and the corneal reflection image are known. X (W) -Y (W)
-Z (W) is a coordinate system that describes the gaze target to be obtained. The initial position and the initial rotation angle of the camera 1 are known in this coordinate system. The position L of the light source 5 is assumed to be provided in front of the zoom lens 2 as described with reference to FIG.

When the positions of the light source 5 and the eyeball 30 are apart from each other, the light incident on the corneal surface becomes almost parallel light. Assuming that the corneal surface is a sphere, the corneal reflection image is r / on the line connecting the light source 5 and the corneal curvature center K when viewed from the position of the camera 1.
It occurs at the point 2 (r: radius of curvature of the cornea). Corneal curvature radius r
Can vary from individual to individual, but can be measured in advance by a different method. Therefore, the position Pr of the corneal reflection image,
When the center position Pu of the pupil and the position L of the light source 5 are known,
The optical axis A of the eyeball 30 is obtained as follows. In addition, G shows a line of sight.

[0018]

[Equation 1] The line of sight G can be defined as a straight line having a constant inclination with the optical axis A as shown in FIG. This inclination is X (W) -Y
It can be estimated by preparing an optotype whose position is known in the (W) -Z (W) system, obtaining an optical axis when gazing at this, and comparing it with the line of sight G.

FIG. 3 is an algorithm for calculating the eyeball center 31 from the position of the pupil or corneal reflection image, estimating the eyeball rotation angle from the positions of the pupil and corneal reflection image on the screen, and integrating these to calculate the line of sight. The example corresponds to claim 2. In FIG. 3A, 31 is the center of the eyeball, 32 is the pupil, and both eyes 30 are inclined by the angle θ with respect to the optical axis A.
The screens 33 of the camera 1 are as shown in FIGS. 2B and 2C, respectively. The pupil image 34 closer to the camera 1 becomes larger than that farther away. A vector is shown by an arrow from the pupil center 35 to the corneal reflection image 36. The camera 1 rotates following the movement of the eyeball 30, and also performs focus adjustment, zooming, and the like. Therefore, in the camera image, the pupil 32 or the cornea reflection image 36 comes near the center of the screen 23. Considering the vector connecting the center 35 of the pupil and the corneal reflection image 36 in the screen 33 in this situation, this direction is determined by the eyeball rotation angle (horizontal θ, vertical ψ). Also, absolute values are θ, ψ, and camera 1
Is determined by the distance d from the eyeball 30 to the magnification E of the camera 1. That is, the following formula is established.

[0020]

[Equation 2] Therefore, if E, d, Vx, Vy are obtained as described above, the eyeball rotation angles θ, ψ can be known. Further, since the position of the eyeball center 31 is similarly known, the camera 1 and the eyeball center 3 are also known.
Set the eye coordinate system with the line segment connecting 1 as the Z axis, and set θ, ψ
The optical axis can be found by finding the line segment with the eyeball rotation angle as. The line of sight can be obtained by adding a predetermined inclination to the optical axis.

Here, f in the equations (3) and (4) is unknown, but Vx, when a target is previously instructed and viewed,
It can be obtained by associating with Vy.

FIG. 4 is an example showing an application of line-of-sight detection.
In the figure, 100 is a pupil image capturing apparatus, and 101 is an ultrasonic sensor including an oscillator and a receiver. Even if the user moves the shooting range Z1 as shown in the figure, as long as the camera 1 shown in FIG. 1 catches the eyeball 30 by following the camera zoom or tracking range Z2, the position of the gaze screen 300 You can detect whether you are looking at. However, the camera 1 has an eyeball 30
When the head moves at high speed to zoom in, the eyeball 30 may be off the gaze screen 300. Therefore, when the ultrasonic sensor 101 detects the approximate position of the head and the eyeball 30 is off the screen and the search is difficult only with the camera 1, control is performed to point the camera 1 in the detection direction of the ultrasonic sensor 101. ..

[0023]

The pupil photographing apparatus according to the present invention provides an illuminating device which irradiates near infrared light from near the optical axis of the lens of the camera to the front, and a light which is reflected on the fundus or corneal surface by the illumination of this illuminating device. A lens system that condenses and forms a pupil image or a corneal reflection image on the imaging surface, a half mirror that is provided in the middle of this lens system to branch the optical path, and the light reflected by this half mirror is split into two. After that, each image is formed, and the focus blur detection processing unit that detects the focus blur from the distance between the image formation points and the focus adjustment that drives the focus adjustment unit of the lens to minimize the distance between the image formation points Since the driving mechanism is provided, the pupil can be imaged without using a monocular camera. In addition, it is possible to reliably detect defocus.

Further, the visual axis detection device according to the present invention uses the pupil photographing device, and a distance calculator for calculating a distance from the lens movement amount by the focus adjustment drive mechanism to the pupil or corneal reflection image to be photographed. , An image processing unit that extracts a pupil and a corneal reflection image from an image obtained from a pupil imaging device, and an eyeball rotation angle calculation that calculates an eyeball rotation angle from distance information and position information of the pupil and the corneal reflection image on the screen Department,
Since the line-of-sight detection unit that detects the line-of-sight from the distance information and the eyeball rotation angle is provided, the line-of-sight can be detected without wearing.
The device configuration is suitable for miniaturization, and high-speed control is also possible. In the man-machine interface, it is also possible to apply to the instruction input device by the line of sight and the intention extraction.

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of a pupil imaging apparatus and a line-of-sight detection apparatus according to the present invention.

FIG. 2 is a diagram for explaining the principle of line-of-sight detection using a pupil and a corneal reflection image.

FIG. 3 is a diagram for explaining another example of line-of-sight detection.

FIG. 4 is a schematic configuration diagram showing an application example of the present invention.

[Explanation of symbols]

 1 Camera 2 Zoom Lens 3 Half Mirror 4 Image Sensor 5 Light Source 6 Condenser Lens 7 Near-infrared Bandpass Filter 8 Reflector 9 Near-infrared Bandpass Filter 10 Lens 11 CCD Line Sensor 12 Focus Blurring Detection Processing Section 13 Focus Adjustment Drive Mechanism 14 Zoom drive mechanism 15 Moving amount measuring unit 16 Moving amount measuring unit 17 Distance calculating unit 18 Image processing unit 19 Eye rotation angle calculating unit 20 Eye-gaze detecting unit 21 Camera drive control unit 30 Eyeball 100 Pupil imaging device 200 Eye-gaze detecting device

Claims (2)

[Claims] 1. An illuminating device that irradiates near-infrared light forward from near the optical axis of a camera lens, and a pupil image or corneal reflection by collecting light reflected by the fundus or corneal surface by the illumination of this illuminating device. A lens system for forming an image on the image pickup surface, a half mirror provided in the middle of the lens system for branching the optical path, and the light reflected by the half mirror is split into two and then each image is formed. A focus blur detection processing unit that detects focus blur from the distance between the image formation points, and a focus adjustment drive mechanism that drives the focus adjustment unit of the lens so that the distance between the image formation points is minimized. Pupil imaging device characterized by. 2. The pupil photographing apparatus according to claim 1, wherein a distance calculating unit for calculating a distance from a lens movement amount by a focus adjustment drive mechanism to a pupil or a corneal reflection image to be photographed, and a pupil photographing apparatus. An image processing unit that extracts a pupil and a corneal reflection image from the obtained image, an eyeball rotation angle calculation unit that calculates the rotation angle of the eyeball from the distance information and the position information of the pupil and the corneal reflection image on the screen, and A line-of-sight detection device comprising a line-of-sight detection unit that detects a line of sight from distance information and an eyeball rotation angle.