JPH1066678A - Non-contact line-of-sight measurement device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the burdens to a testee, to substantially mitigate the limit of an operation range and to measure the line of sight with high accuracy. SOLUTION: A head part is irradiated with infrared ray light, the output of two cameras 1A and 1B for the head part is used and an eyeball position detection means 31 detects the position of an eyeball E from the reflected light of markers M1, M2 and M3. By using the eyeball position, a tracking control means 32 catches the eyeball E by the camera 2 for the eyeball and tracks it. The position of a cornea reflection line and a pupil center are detected from the output of the camera 2 for the eyeball and a line-of-sight vector is obtained by a line-of-sight vector calculation means 33. A line-of-sight placing means 34 spatially places the line-of-sight vector in a desired coordinate system.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye gaze measuring apparatus, and more particularly to a non-contact eye gaze measuring apparatus which can detect a gaze with higher accuracy using only a simple position marker which does not burden a subject.

[0002]

2. Description of the Related Art Conventional gaze measuring apparatuses are mainly of a type in which a subject's head is fixed and a type in which a detecting device is attached to the subject's head. Both of these significantly impaired the subject's usual behavior and put a great burden on the subject in gaze measurement. In recent years, non-contact types have been made to reduce the burden on the subject, but the measurement range is extremely limited because the detection device is just remote from the subject. However, the measurement can only be performed when the subject cannot move. The method of detecting the line of sight from a face image using image processing or the like, which has been proposed as a method of removing the restriction on the movement of the subject, requires time for image processing and has poor time characteristics. Further, since the gaze is detected by the same camera as the face detection, there is a disadvantage that accuracy is poor.

[0003]

The subject's gaze is sufficiently high so as to reduce the burden on the subject and to operate the subject at a normal operation speed in a sufficient range for working on the desk, so that the subject is not burdened. In order to provide a device that captures with high accuracy, it is important to provide a method for fairly accurately grasping the eyeball position of a subject.

Further, since the position of the eyeball can be measured from the three-dimensional measurement, it is necessary to remove the error factor of the three-dimensional measurement at the same time. When there are a plurality of light sources, the reflection of light from the plurality of light sources may be an error factor, and there may be eyeglasses, skin, and the like that reflect light in addition to a marker or eyeball attached to the face.

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and has as its object to reduce the burden on a subject, to operate the subject at a normal operating speed in a range sufficient for working on a desk, Is to provide a device that captures the image with sufficiently high accuracy.

[0006] In addition, operations on a desk, particularly operations using a computer, are performed by looking at a monitor or a desk.
The line of sight moves on a plurality of arbitrary planes. Accordingly, it is an object of the present invention to provide a device that can place a line of sight on an arbitrary plane and specify what the user is looking at.

[0007]

SUMMARY OF THE INVENTION The present invention is a non-contact gaze measuring apparatus using a plurality of head cameras and a high-magnification ophthalmic camera. Eyeball position detection for detecting reflected light from at least three markers attached to the head with the plurality of head cameras, detecting the position of the head and the direction of the face, and detecting the position of the eyeball. Means, an infrared light irradiating means for applying infrared light to an eyeball position detected by the plurality of head cameras,
Only the eyeball is captured by the high-magnification eyeball camera, tracking control means that constantly tracks the eyeball by the information from the head camera, and the position of the corneal reflection image and the pupil center are detected from the output of the eyeball camera, The apparatus includes a line-of-sight vector calculation unit for obtaining a line-of-sight vector, and line-of-sight setting unit for converting the line-of-sight vector into a desired coordinate system and spatially setting the line-of-sight position and the line-of-sight direction.

[0008] The infrared irradiating means comprises a plurality of infrared light irradiating sections arranged around a camera lens.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention relates to a non-contact gaze measuring device for detecting gaze and measuring for a long time with very high accuracy. By detecting the marker that reflects infrared light with multiple head cameras and measuring the position of the marker three-dimensionally,
Measure position and tilt accurately.

[0010] By arranging at least three markers so that the eyeball to be measured is located at the center of gravity of the marker, it can be assumed that the eyeball exists at the center of the head coordinate system with high accuracy.
The center of gravity is calculated from the position of the marker, and assuming that the center of the eyeball is located near the center of gravity, infrared light is irradiated, and tracking is performed by an eyeball camera described later in synchronization with the infrared light. In addition, the displacement between the center of the eyeball and the center of gravity is constantly measured, and by correcting the displacement, the position of the center of the eyeball is accurately set and the head coordinate system is calculated.

Next, infrared light is emitted in accordance with the center of the three-dimensionally measured head coordinate system, and the eyeball camera tracks in synchronization with the infrared light as described above. The camera for the eyeball captures the pupil image and the corneal reflection image, finds the pupil center from the pupil image,
Even if there is no pupil at the center of the eyeball camera, the coordinates of the pupil center are corrected. Then, a gaze vector is obtained from the corneal reflection image and the corrected pupil center coordinates.

The line of sight vector accurately detected by the above means is projected by calculation on an arbitrary predetermined plane, and the line of sight is laid on the plane. The line of sight laid on this arbitrary plane is presented as a model. This allows the observer to immediately know where the target object is looking at any time.

[0013]

DESCRIPTION OF THE PREFERRED EMBODIMENTS First, an eyeball position detecting method will be described.

FIG. 1 is a diagram in which a marker is attached around the eyeball E to be measured. H indicates a head. M1,
M2 and M3 (hereinafter, M is used when it is not necessary to distinguish them) show examples of three markers attached to the spectacle frame. These markers M are three-dimensionally measured by a plurality of head cameras 1A and 1B shown in FIG. 2, and the positions of the markers M are observed.

FIG. 2 shows a case where two head cameras 1A and 1B are used. How far the marker M is away from the head cameras 1A and 1B and the direction of the direction are calculated by comparing the marker information of the head cameras 1A and 1B with each other. Thereby, the approximate position of the center of the eyeball E can be measured. It is obtained by the head position measuring device (not shown) in the image processing unit 3 based on the X, Y, and Z information. In the drawings, reference numeral 2 denotes an eyeball camera, and reference numeral 4 denotes an infrared light irradiation unit.

FIG. 3 shows the arrangement of head cameras 1A and 1B for irradiating infrared light to the head position and measuring the position of the subject's head from a marker M attached to the head H, and the irradiation of infrared light. FIG. 4 is a diagram showing an example of a position of a means 4. 8 is the observation target,
In this example, it is indicated that the display is on a monitor and a keyboard. In order to prevent the infrared light from being directly irradiated to the head cameras 1A and 1B, the infrared light irradiating means 4 is arranged around the camera lens as shown in FIG.

FIG. 4 illustrates the above in more detail. The infrared light irradiation means 4 shown in FIG. 3 is provided on the lens 5 of the main body of the head camera 1A (1B). Reference numeral 6 denotes a polarizing plate, and reference numeral 7 denotes a bandpass filter that transmits only infrared light. In FIG. 4, the polarization plate 6 is tilted by 90 degrees by the right and left cameras, so that the infrared light from the infrared light irradiation means 4 provided on the left and right of the observation target 8 in FIG. Noise light can be removed.

That is, in the configuration as shown in FIG. 5A, the right head camera 1B can irradiate, for example, only light in a laterally deflecting direction because of the presence of the polarizing plate 6, and conversely, the same as the irradiating direction. Only light polarized in the direction can enter. Therefore, noise generated by other light sources will not be incident. In the case of a plurality of light sources, the noise is removed by providing an appropriate band-pass filter 7 and identifying which light source receives the light. FIG. 5B schematically shows this.

FIG. 6 shows a mechanism in which the eyeball camera 2 for catching the eyeball E tracks the eyeball E based on the head position information.
It is. 32 is a tracking control means, 2 is an eyeball camera with a depth direction adjustment focus, 9 is a horizontal position adjustment mirror, 10 is a vertical position adjustment mirror, 11 is an optical axis, and M is a marker. . In FIG. 6, as an example,
Regarding X and Y, each is shown by a method using a mirror for dedicated tracking. Z is adjusted by a focus function inside the camera. Thereby, the eyeball camera 2 can always correctly capture the eyeball E with accurate focus.

FIG. 6 will be described in more detail. First, FIG.
7 will be described. FIG. 7 shows a means in which the light source and the camera track in synchronization. 23
Denotes a half mirror, which emits the infrared light to the outside by the infrared light irradiating means 4 and simultaneously emits the incident light to the eyeball camera 2. Reference numeral 9 denotes a horizontal position adjustment mirror for tracking the movement of the subject in the horizontal direction (X), and reference numeral 10 denotes a vertical position adjustment mirror for tracking the movement in the vertical direction (Y). Reference numeral 21 denotes a band pass filter, and reference numeral 22 denotes an eyeball photographing lens. The eyeball camera 2 and the infrared light irradiating means 4 are arranged conjugate to the half mirror 23. Moreover, since the mirrors are located behind the tracking mirrors 9 and 10, even if the mirrors are driven by the movement of the user, the infrared light is always illuminated on the same trajectory under the same illumination condition and the photographing can be performed. An advantage of this arrangement is that since the imaging region and the illumination region coincide with each other, it is possible to specify an infrared light irradiation range that is extremely narrow. That is, it is possible to reduce the size of the infrared light source and increase the irradiation density, and this advantage is great.

FIG. 8 is a block diagram showing the configuration of one embodiment of the present invention. In FIG. 8, reference numeral 31 denotes an eyeball position detecting means for processing the outputs of the head cameras 1A and 1B. 3
Reference numeral 2 denotes the above-described tracking control means, which performs tracking control of the eyeball E by the eyeball camera 2. Reference numeral 33 denotes a line-of-sight vector calculation unit that processes the output of the eyeball camera 2. Reference numeral 34 denotes a line-of-sight arrangement means for displaying the line of sight of the subject.

Next, the operation of the embodiment of FIG. 8 will be described mainly with reference to FIGS. Note that (S1) to (S12) in FIG. 9 indicate each step.

The eyeball position detecting means 31 calculates the distance from the position of the marker M to the midpoint of the head cameras 1A and 1B, the head H
, And the approximate position of the eyeball E is detected (S1). The center of gravity of the three markers M1 to M3 is obtained, and a correction described later is added (S2), and it is assumed that the eyeball E exists here (S3). The focus of the eyeball camera 2 is set based on the obtained eyeball position (S4), and the tracking control means 32 sets the two position adjustment mirrors 9, 10. As a result, the eyeball camera 2 captures the eyeball E, and thereafter tracks the eyeball E (S5).

The line-of-sight vector calculation means 33 processes the image obtained from the camera for eyeball 2 and extracts a pupil image (S6). Since the pupil image includes many images other than the pupil, those noise portions are removed (S7), and the center position is calculated (S8). Since it was assumed that the X, Y, and Z of the tracking information of the eyeball camera 2 were at the center of the head cameras 1A and 1B, these were corrected from the detected pupil center and X ', Y', Z Ask for '.

At the same time, the corneal reflection image is captured by the eyeball camera 2 (S9), and the pupil center position and the position of the corneal reflection image, X ', Y', Z ', and the eyeball diameter from the physiological point of view are determined. A line-of-sight vector is created using the value of the eyeball center (S10), (S11). The created line-of-sight vector is geometrically mapped to an arbitrary plane, which has been measured in advance, and is placed on the plane as a line of sight (S1).
2).

FIG. 10 is a view for explaining placement of the line of sight by the line of sight placement means 34. A personal computer PC and a keyboard KB are placed on a desk, and planes PL1 and PL2 preset as display planes are prepared. ing. In FIG. 10, the eyeball E of the subject is directed to one of the keys on the keyboard KB. Therefore, an intersection W2 between the line-of-sight vector and the plane PL2 is obtained, and this point W2 is displayed.
This allows the observer to immediately know where the subject is looking.

When the subject looks at the display of the personal computer PC, the intersection W1 between the line-of-sight vector at this time and the plane PL1 is displayed. In this manner, the spatial arrangement of the line-of-sight vectors is performed.

In order to observe the line of sight, the user must place the camera on an arbitrary plane. At that time, when to show to the subject,
You may not show it. In the case of showing, for example, in the case of the plane PL <b> 1 in FIG. 10, it is conceivable that the line of sight is directly displayed on the monitor if it is limited to only the monitor. The plane PL2 can be indicated by a laser pointer or the like. If it is not shown, it can be displayed by taking an arbitrary plane with a camera and superimposing it on the video. In the case of a display device such as a monitor, the image may be distributed and superimposed on a monitor that is invisible to the subject.

The marker M is set so that the eyeball E is located at the center of gravity of the marker. However, this is not necessarily accurate, and the number of the markers M does not need to be three. I just need more. However, considering the rotation of the head H, the head can be tilted only up to 45 degrees at four points, but can be up to 60 degrees at three points. Also,
The eyeball E may not be at the position of the center of gravity, but the correction calculation in that case will be described below. If the actual eyeball position is K and the center of gravity of the marker is G, if the markers are M1 to M3, K = G before correction

[0030]

(Equation 1) Can be corrected with. Here, P1, P2, and P3 are vectors of points obtained by three-dimensional measurement, and PG is a vector of the center of gravity. _Ex and _Ey are correction values in the horizontal and vertical directions, respectively, and are the geometric dimensions of the center of the eyeball E. This correction value is
Assuming that the spectacle frame does not move significantly, the whole center of the eyeball E may be obtained by image processing and its center of gravity G may be captured,
The determination may be made by looking at the image of the eyeball camera 2 capturing the position of the eyeball E in the early stage of the operation of the apparatus. This provides a basis for chasing the eyeball E fairly accurately.

[0031]

As described above, the present invention attaches a simple marker to the vicinity of the eyeball, irradiates it with infrared light that does not affect the subject's behavior, and reflects the marker only from the desired light source. The marker position can be detected with high accuracy by detecting only the reflected light of the marker. For example, when the present invention is not applied, it is difficult to separate the noise of various lighting devices in the room from the marker position and the corneal reflection image, and accordingly, the accuracy is considerably reduced. Further, even if accuracy is sacrificed, the calculation time is enormous and cannot be used for real-time measurement.

Further, since only infrared light is used and no visible light is used, image processing is accelerated, detection can be sufficiently performed within the sampling frequency of the camera, and accordingly, it is possible to follow the movement of the subject. Can be done accurately.

Further, since only the eyeball is photographed by the high-magnification eyeball camera, the line-of-sight vector can be detected with considerably high accuracy. If the conditions are met, it is possible to measure the accuracy within 0.3 degrees of the viewing angle. Moreover, the subject only attaches three or more simple markers and does not perform any other constraints.

Further, since the line of sight can be placed at arbitrary coordinates, for example, when measuring the line of sight when operating the computer environment on a desk, not only the line of sight on the monitor but also the line of sight of the keyboard or other objects Can catch the line of sight.

[Brief description of the drawings]

FIG. 1 is a diagram showing an example of a head detection marker provided around an eyeball to be detected in the present invention.

FIG. 2 is a diagram showing an outline of the operation principle of head detection in the present invention.

FIG. 3 is a diagram illustrating an example of an irradiation position of infrared light for head detection and an arrangement of a head camera according to the present invention.

FIG. 4 is a diagram showing an attachment position of an infrared light irradiating means of the head camera and an attachment position of a bandpass filter and a polarizing plate in the present invention.

FIG. 5 is a diagram showing noise removal by a plurality of light sources according to the present invention.

FIG. 6 is a diagram illustrating an operation principle of tracking an eyeball position by an eyeball camera according to the present invention.

FIG. 7 is an explanatory auxiliary diagram for detailed description of FIG. 6;

FIG. 8 is a block diagram showing a configuration of one embodiment of the present invention.

FIG. 9 is a flowchart for explaining the operation of the embodiment in FIG. 8;

FIG. 10 is a diagram showing a mechanism for laying a line of sight on an arbitrary plane according to the present invention.

[Explanation of symbols]

 M1 to M3 Marker E Eyeball H Head PC Personal computer KB Keyboard PL1, PL2 Plane W1, W2 Intersection between line of sight vector and plane 1A, 1B Head camera 2 Camera for eyeball 21 Bandpass filter 22 Eyeball photographing lens 23 Half mirror 3 Image processing unit 31 Eyeball position detection means 32 Tracking control means 33 Eye vector calculation means 34 Eye placement means 4 Infrared light irradiation means

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Takashi Tate 1-2-7 Nishi-Azabu, Minato-ku, Tokyo NAC Inside (72) Inventor Shigetoshi Mizuno 1-2-7 Nishi-Azabu, Minato-ku, Tokyo Shares In company knack

Claims (2)

[Claims] 1. A non-contact eye-gaze measuring device using a plurality of head cameras and a high-magnification ophthalmic camera, wherein at least one of infrared lights applied to a subject's head is applied to the head. Eyeball position detection means for detecting reflected light from the three markers with the plurality of head cameras, detecting the position of the head and the direction in which the face is facing, and detecting the eyeball position; An infrared light irradiating unit that irradiates infrared light to an eyeball position detected by the camera for use, tracking control that catches only the eyeball with the high-magnification eyeball camera, and always tracks the eyeball with information from the head camera. Means, a line-of-sight vector calculating means for detecting the position of the corneal reflection image and the center of the pupil from the output of the eyeball camera, and calculating a line-of-sight vector, and converting the line-of-sight vector into a desired coordinate system, Space direction Non-contact gaze measuring apparatus characterized by comprising: a sight constellation means for location, the. 2. The non-contact sight line measuring device according to claim 1, wherein the infrared irradiating means is constituted by arranging a plurality of infrared light irradiating sections around a camera lens.