JPH0449943A - Eye ball motion analyzer - Google Patents

Abstract

PURPOSE:To enable the analyzing of movement of a visual line to be realized by a motion of a head and that of a visual line by a motion of an eye ball simultaneously by detecting the motion of the head to be converted into a motion of the eye ball. CONSTITUTION:An eye motion detecting section 5 applies a detection output thereof to a gazing point processing section 7 and a head motion detecting section 6 applies a detection output thereof to the gazing point processing section 7. The processing section 7 computes movement of a visual line of an observer according to a motion of an eye ball and the motion of the head to define a notice point so that a moving component is separated between one gazing point and another from the movement of the eye sight. Then, gazing information separated with the processing section 7 is applied to an image processing section 8 to be converted into an image signal. The image signal is applied to a superimposer 9 and superimposed with a video signal applied from outside to be turned to a video output, which is shown on a display. This enables the analyzing of the gazing movement by the motion of the eye ball and that by the motion of the head simultaneously.

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention relates to an eye movement analysis device, and more particularly to an eye movement analysis device that can detect eye movements and head movements of a subject and display a viewpoint on a display. Regarding.

[Prior Art] A device that detects the eye movement of a subject and displays the gaze point on a display or film was developed as an eye camera, and the most common opportunity to see it is probably due to the driver's eye movements. be. Various methods have been considered for detecting the eye movements of the subject, including a method using corneal reflected light, a method using a contact lens, and an EOG method.

FIG. 10 is a diagram for explaining the principle of detecting eye movement using the scleral reflection method, which utilizes the difference in reflectance between the white and iris of the corneal surface. Referring to Figure 10, left and right eyes E
Sensors 1 and 2 each having three elements are arranged at L and E2. A light emitting element 3 is arranged in the center of the sensor 12,
As the light-emitting element 3, a light-emitting diode that emits infrared radiation with a relatively wide directivity of about ±21° is used. Light receiving elements 4 are provided on both sides of the light emitting element 3. As the light receiving element 4, a photodiode with sharp directivity of approximately ±10° is used. The light projected from the light emitting element 3 to the eyeball has a different reflectance between the black eye and the white of the eye, and by amplifying this difference in reflectance and taking the difference, it becomes a horizontal (left and right) output, and when the sum is taken, it becomes a vertical (up and down) output. The output is

Additionally, the positions of sensors 1 and 2 with respect to the eyeball are different between the horizontal direction and the vertical direction, with the horizontal sensor 1 detecting reflected light at the center of the eyeball above and below, and the vertical sensor 2 detecting the reflected light at the center of the eyeball.
are arranged so as to detect the reflected light below. A horizontal sensor 1 is placed on one eye E1, and a vertical sensor 2 is placed on the other eye E2.If these are used simultaneously, two-dimensional eye movements can be detected and the viewpoint can be displayed on the display. can be displayed. For example, Japanese Patent Laid-Open No. 126140/1983 discloses a method for obtaining information such as the moving speed, moving direction, moving distance, and gaze time of the eyeball from such a method.

When analyzing eye movements using such a device, the subject's head is fixed using a condyle table and only eye movements are recorded. Alternatively, a small camera may be mounted on goggles or eyeglass frames equipped with an eye movement detection sensor to capture images corresponding to the subject's field of view, allowing for free movement of the head while detecting eye movement. It is used. In addition, with conventional eye movement analyzers, measurements are taken when the head is fixed or not, as described here.
Or, even when the head is not fixed, the head movement is not detected, or even if it is detected, it is not detected independently and converted into a line of sight.

[Problems to be Solved by the Invention] As mentioned above, conventional eye movement analyzers do not take head movements into account, so the method of measuring head movements by fixing them with the whites of the cheeks is difficult to analyze eye movements. However, there is a problem in that it is not possible to realize the natural observation behavior of humans who gaze using head movements.

Even with a method that allows free head movement, it does not take into account the movement of the line of sight achieved by head movement, so if you move your head violently and change your line of sight, such as while driving a car, no eye movement will occur. Due to the action of the postural control system, eye movements occur in the exact opposite direction to head movements, such as when walking while gazing at someone's face at a distance. , there was an inconvenience in that the eye movement was recorded or displayed even though the user was actually gazing at the other person's face.

Therefore, the main purpose of this invention is to simultaneously analyze the gaze movement realized by head movement and the gaze movement caused by eyeball movement by detecting head movement and converting it into gaze movement. An object of the present invention is to provide an eye movement analysis device that can perform the following functions.

[Means for Solving the Problems] The present invention is an eye movement analysis device that analyzes the eye movement of an observer, and includes an eye movement detection means that detects the movement of the eyeball, and a head movement analyzer that detects the movement of the head. a detection means; and a gaze point processing means that calculates the movement of the observer's line of sight according to the detected eyeball movements and head movements, and separates the gaze point from the gaze movement by defining the gaze point. , an image processing means for displaying on the image gaze information consisting of the movement of the gaze moving between the gaze points, and a superimposing means for superimposing the gaze information on the video image of the observer. Prepared and configured.

[Operation] The eye movement analysis device according to the present invention detects the movement of the eyeballs and the movement of the head, and calculates the movement of the observer's line of sight according to the movement of the eyeballs and the head. By defining the viewpoint, the gaze point is separated from the gaze movement, and the gaze information consisting of the gaze movement moving between the gaze points is superimposed on the video image being viewed by the observer. do.

[Embodiment of the Invention] An embodiment of the invention will be described in detail below with reference to the drawings.

FIG. 1 is a schematic block diagram of an embodiment of the present invention.

Referring to FIG. 1, an eye movement detection section 5 detects the movement of the observer's eyeballs, and provides the detection output to a gaze point processing section 7. The head motion detection unit 6 detects the movement of the observer's head, and provides the detection output to the gaze point processing unit 7.

The gaze point processing unit 7 calculates the movement of the observer's line of sight according to the eye movement detected by the eye movement detection unit 5 and the head movement detected by the head movement detection unit 6, and determines the gaze point. By defining , we separate the movement component between the gaze points from the gaze movement. The line of sight information separated by the gaze point processing section 7 is given to the image processing section 8 and converted into an image signal. This image signal is applied to a superimposer 9, where it is superimposed on a video signal applied from the outside to become a video output, which is displayed on a display (not shown).

FIG. 2 is a diagram for explaining the movement of the line of sight due to head movement. Next, with reference to FIG. 2, the movement of the line of sight due to head movement will be explained. In general, moving the desired visual object can be achieved by rotating the eyeballs toward the visual object, or by moving the head without moving the eyeballs. Usually, both movements are used together. Head movement involves parallel movement due to the orientation of the feet and spine, as well as movement of the neck, spine,
# There is also a rotational movement realized by L foot etc. This head movement is converted into an eye rotation angle using a method described later to obtain a head movement corrected eye rotation angle α shown in FIG. Movement of the point of gaze is executed by a combination of this head movement correction eyeball rotation angle α and the eyeball rotation angle θ. Here, the sum of these two rotation angles α and θ is collectively defined as the line of sight.

FIG. 3 is a diagram for explaining the method of defining the gaze point from the aforementioned line of sight. During eye movements, even when fixating or gazing, the eye performs minute movements called fixation micromovements. In addition, the head movement correction eyeball rotation angle α due to the vestibulo-ocular reflex 1 posture itself to prevent deviation of the line of sight due to posture shaking also occurs during fixation. In other words, there are very few moments when the line of sight is completely stationary in space. Therefore, it is necessary to separate the gaze point, which occurs when the subject gazes with the intention of receiving information from the visual object, from among the constantly moving gaze movements. For this purpose, it is necessary to set a certain standard and define the gaze point. Regarding this gaze point, one based on eye movement speed has already been reported. (
JP 60-126140: Journal of the Communication Society [Definition of gaze point in images and its application to image analysis J1986゜9
, Vol. J69-D, No. 9.1335-134
2 pages).

One of the features of this invention is that the definition of the gaze point, which is determined based on the eye movement speed, is expanded to include the gaze movement speed including head movement. Specifically, as shown in FIG. 3, when the next sample falls within the range Dth of the point of interest determined by the threshold velocity vth and the sampling period T6 from the current point of interest, the current point of interest is It was set as the same point of fixation as the viewpoint, and when it exceeded this range, it was set as a new point of fixation. As an example of the threshold speed vth,
It is set at 5°/sec based on the nature of follow-up movement during eyeball movement, and is used for practical purposes. For example, if the sampling period is 10m
When 5ec, D th-5゜/5ecXQ, 01
sec-Q, 05″-3′.

FIG. 4 is a diagram for explaining the correspondence between the coordinates of an image viewed by an observer and the coordinates realized by an embodiment of the present invention. As shown in FIG. 4(a), the vertical Dv is a distance HD away from the subject's position.

When observing a horizontal Dh screen, the screen coordinates are (11) - (
Px, Py). The visual angle of the presented image as seen from the subject is a-180/π・tan-' Dh/ (2・D), as shown in FIG. 4(b).
b-180/π・tan-'Dv/ (2・D) Let the gaze movement angle (Xv, Yv) be the screen coordinates (Xp, Yp
) is: Xp-f (Xv)-pju(r-p) ginXv/(a-
Xv)...(1) Yp"'g (Yv)-q+ (s-Q) ・Yv/(
b-Yv)...(2) It is expressed as follows.

FIG. 5 is a diagram showing an example in which the eye movement detecting section 5 and the head movement detecting section 6 shown in FIG. 1 are attached to goggles. The eye movement detecting section 5 is provided with a light emitting diode in the center in the same way as the sensor 1 shown in FIG. . A magnetic sensor is used as the head movement detection section 6.

FIG. 6 is a diagram for explaining the principle of the magnetic sensor used in the head motion detection section 6 shown in FIG. 5. Referring to FIG. 6, the head motion detection unit 6 includes an orthogonal coil that serves as a source 61 and an orthogonal coil that serves as a sensor 62. In response to commands from computer 64, drive circuit 65 drives orthogonal coils of source 61 to generate a magnetic field. When the subject wearing the head movement detection unit 6 moves, the sensor 62
A voltage is induced, and the detection circuit 63 detects this voltage.
The computer 64 calculates the detection output, thereby outputting data corresponding to the movement of the head.

FIG. 7 is a diagram showing the principle of a head coordinate system centered on the subject. Next, the head coordinate system detected by the head motion detection section 6 will be explained with reference to FIG. The head coordinate system is an XYZ coordinate system realized by parallel movement of the subject with respect to the observation target, as shown in Figure 7(a), and a polar coordinate system based on the rotational movement of the head, as shown in Figure 7(b). There are two possibilities. The amount of head movement in each coordinate system is defined as (Hx, Hy, Hz) and (Hφ, Hθ, Hψ). Here, as an example, the direction of approaching the observation target is the Y axis, the horizontal movement direction is the X axis, and the vertical movement direction is the Z axis. Hφ indicates the rotation point of the X axis, that is, the movement of nodding the head up and down, and Hθ indicates the rotation of the Y axis, that is, the movement of tilting the head once from the left shoulder to the right shoulder. Hψ is rotation within the Z axis, and is a movement of rotating the neck from side to side.

The line of sight changes due to horizontal movement of the head (Hx, Hy, Hz), and when this is converted into an eyeball rotation angle (E x, E y), the following equation is obtained.

Ex-180/π・tan-'Hx/ (D×Hy)・
... (3) Ey=180/π・tan-' Hz/ (D+Hy
)... (4) When the head is tilted by Hθ toward the left or right shoulder, the coordinates of the eye movement system rotate. Therefore, it is necessary to convert the eye movement coordinate system (Xe, Ye) tilted by Hθ into a coordinate system (Xe', Ye') orthogonal to the original observation object.

Xe' −Xe ・cosHθ×Ye−sinHθ・・
・(5) Ye' -- From equation 4), it is expressed by the following equation.

Xh-Ex+Hψ-(7)Yh-E
y0Hφ ... (8) Therefore, the movement of the line of sight (Xv, Yv) considering the movement of the head is the (5th)
From equations to equations (8), the following equations (9) and (10)
It is expressed by the formula.

Xv-Xe' 10Xh -(9)Yv-
Ye' + Yh - (10) The above (
By using Equation 9) and Equation (10), it is possible to reproduce the normal movement of the line of sight that is performed by combining head movement and eyeball movement.

FIG. 8 is a flowchart for explaining the overall operation of an embodiment of the present invention, and FIG. 9 is a diagram showing an example of a display screen according to an embodiment of the invention.

Next, referring to FIGS. 1 to 9, the specific operation of one embodiment of the present invention will be described using FIG. 8. First, in the initial settings, coordinate transformation coefficients are used to correctly superimpose the coordinates viewed by the observer and the position data of the line of sight obtained by an embodiment of the present invention onto the image viewed by the observer. and set the formula. By converting the coordinates using the method shown in Figure 4 above, (1)
The line of sight movement angle (Xv) is calculated using the equation and equation (2).

Yv) is converted to screen coordinates (Xp, Yp).

Next, the eye movement data is calibrated using the eye movement detected by the eye movement detection unit and the head movement detected by the head movement detection unit 6. That is, the above (9th
) and Equation (10), the eye movement data and head movement data are converted into line-of-sight movements, and similarly, using Equations (1) and (2), they are converted into screen coordinates. The movement of the line of sight obtained by these transformations is transformed into screen coordinates, but the transformation is relative. In order to reproduce the movement of the subject's line of sight on the screen that captures the visual target that the subject is actually looking at, it is necessary to absolutely match the screen coordinates with the subject's line of sight. Eye movement calibration performs this task.

After the calibration is completed, the start of measurement is commanded by, for example, a key, and the measurement is repeated until the end of the measurement is commanded again by the key or the like.

In the image display, the true movement of the line of sight is determined from the eye movement data and the head movement data using equations (1) to (10), and is displayed on the image that the subject is viewing. At this time, the movement of the line of sight is separated into movement components between the gaze points by the definition of the gaze point, an example of which has already been disclosed. Then, the superimposer 9 superimposes the gaze point and the movement trajectory between the gaze points on the image that the observer is viewing. An example of the display at that time is shown in FIG.

The example shown in FIG. 9 is an example of the movement of the subject's line of sight when he is looking at a girl. Gaze points are color coded according to gaze time, movement trajectories between gaze points are color coded according to speed, and superimposed at corresponding positions in the image. Data such as eye movement, head movement, and number of samples obtained through these methods are recorded on a disk (not shown) or the like.

[Effects of the Invention] As described above, according to the present invention, the movement of the eyeballs and the movement of the head are detected, the movement of the observer's line of sight is calculated according to these detection outputs, and the point of gaze is defined. By doing this, the gaze point is separated from the gaze movement, and the gaze information is superimposed on the moving image that the observer is looking at, so the gaze movement due to eyeball movement and the gaze due to head movement are separated. can be analyzed simultaneously with the movement of

[Brief explanation of the drawing]

FIG. 1 is a schematic block diagram of an embodiment of the present invention. FIG. 2 is a diagram for explaining the movement of the line of sight due to head movement. FIG. 3 is a diagram for explaining the definition of a gaze point. FIG. 4 is a diagram for explaining the correspondence between the coordinates of an image viewed by an observer and the coordinates of a system realized by an embodiment of the present invention. FIG. 5 is a diagram showing an example in which an eye movement detection sensor and a head movement detection sensor are attached to goggles. FIG. 6 is a diagram for explaining the principle of the magnetic sensor used in the head motion detection section shown in FIG. 5. FIG. 7 is a diagram showing the principle of a head coordinate system centered on the subject. FIG. 8 is a flowchart for explaining the specific operation of one embodiment of the present invention. FIG. 9 is a diagram showing an example of a display screen. FIG. 10 is a diagram for explaining the principle of detecting eye movement using the scleral reflex method. In the figure, 5 is an eye movement detection unit, 6 is a head movement detection unit, 7 is a gaze point processing unit, 8 is an image processing unit, 9 is a superimposer, 61 is a source, 62 is a sensor, 63 is a detection circuit, 64 is a computer, and 65 is a drive circuit. Patent Applicant A.T.R. Co., Ltd. Viewing Diagram New IoJmu ζt>b Sample Diagram Diagram Diagram Diagram Diagram. ::IjlNh=h1! Altj#3j1. f-4
/1RtTi'4TGoγ

Claims (1)

[Scope of Claims] An eye movement analysis device for analyzing eye movements of an observer, comprising: eye movement detection means for detecting eye movement; head movement detection means for detecting head movement; and the eye movement detection means. The movement of the observer's line of sight is calculated according to the movement of the eyeballs detected by the means and the movement of the head detected by the head motion detection means, and the movement of the line of sight is determined by defining a point of gaze. a gaze point processing means for separating gaze points, an image processing means for displaying gaze information on an image consisting of gaze movement moving between gaze points, and a gaze point processing means for displaying gaze information on an image, and An eye movement analysis device comprising superimposition means for superimposing onto a video image.