JPH0377533A - Apparatus for measuring gaze attitude and absolute position of gaze point - Google Patents

Abstract

PURPOSE:To enable a gaze point analysis with eye camera to be carried out and a position and direction of a subject's head to be also computed even in case that the head moves around horizontal or vertical axis by setting up four or more characteristic points indicating respective absolute positions in an object whereat the subject is gazing. CONSTITUTION:Nine LED's P1-P9 are arranged as characteristic points on a drafting board 1 at spacings of Dx and Dy in x- and y- directions, respectively. Their positions are defined so that at least four LED's, which are centered around a point gazed at by a subject 2, are included in a photographic range of a visual field camera 3a fixed to the subject's head 2a. The visual field camera 3a fixed to the subject's head 2a outputs image signals of the drafting board in a direction of the subject's face, and eye mark cameras 3b detect directions of the subject's right and left eyes separately and output coordinate signals. It is possible, therefore, to detect an attitude of the subject's head based on data put out of the eye camera and to analyze an absolute position variation of gaze point even in case that the subject moves the gaze point through turning his head up and down or right and left.

Description

Detailed Description of the Invention (Field of Industrial Application) The present invention is a method for detecting the head of a subject wearing an eye camera based on the visual field image signal output from the visual field camera and the signal of the feature point placed within the target field of view. Regarding the device for measuring the posture of the
The present invention relates to a device that makes it possible to measure the absolute position of a gaze point within a target field of view when the head is rotated vertically and horizontally from these signals and the coordinates of the gaze point on a visual field image.

(Conventional technology) An eye camera is a device that analyzes a person's gaze point.

The eye camera includes a field of view camera and an eye mark camera, and is used while being fixed to the subject's head. At this time, the visual field camera photographs the subject's front visual field, the eye mark camera detects the deviation of the eyeball from the front, and outputs the position of the gaze point as coordinates on the visual field image.

Therefore, if measurements are taken with the subject's head fixed, the movement of the gaze point within the field of view when the subject observes the object is determined by the coordinates of the eyemark output from the eyemark camera. can be automatically analyzed by tracking. However, when you move your head while gazing at a point within the target field of view, or when you move or rotate your head to move the point of gaze, the coordinates (relative coordinates) output from the eye mark camera will change. , it no longer shows changes in the coordinates (absolute coordinates) of the gaze point within the actual field of view. Therefore, when it is necessary to analyze the gaze point when natural head movements are involved, humans can sequentially track visual field images displaying eye marks that indicate the position of the gaze point, and then The absolute coordinates of the point of interest are read in the system, and it was not possible to automatically measure the absolute coordinates of the point of interest and analyze the amount and direction of movement.

As a way to solve this problem, by setting multiple feature points at fixed positions within the target field of view and detecting the coordinates of the feature points on the field of view image, we can detect the deviation of the field of view image from the reference position. A method has been proposed that allows changes in the absolute coordinates of the gaze point to be automatically analyzed even when the head moves by correcting the coordinates of the gaze point detected by an eyemark camera.

As one of them, Japanese Patent Application Laid-Open No. 62-53630 discloses that a transformation matrix of a visual field image during measurement with respect to a reference image of the first frame is obtained, and an eye mark signal is converted to coordinates on the reference image to calculate the absolute position. A method has been proposed for detecting the absolute coordinates of the gaze point when head movement is involved.

(Problem to be Solved by the Invention) However, the method described in the above publication requires movement of the subject's head in a direction parallel to the field of view, movement in the normal direction of the nucleus, and movement of the subject's head in the normal direction. However, it cannot accommodate rotations of the head in the left-right or up-down directions, that is, movements that involve rotation of the head around a vertical axis or rotations of the head around a horizontal axis in the left-right direction. In actual observation of the target field of view, the subject very often moves the point of gaze by turning the head up and down and left and right, so the above method does not significantly restrict the subject's movement. Without this, there was a problem that automatic analysis of gaze points could not be performed.

Therefore, the present invention detects the posture of the subject's head based on the data output from the eye camera even when the subject moves the gaze point by rotating the head up and down or left and right. ,
The objective is to obtain a device that can automatically analyze changes in the absolute position of the gaze point.

(Means for Solving the Problems) In the present invention, four or more feature points P, ~P, are installed within the field of view l of the object to be observed, and at least any four of these feature points
The mutual positional relationship of the feature points P, ~P is determined so that the feature points P, . By making the brightness of this feature point sufficiently higher than the surroundings, the position coordinates on the visual field image 4 can be detected by image processing. Each detected feature point may be sequentially emitted in synchronization with a vertical synchronization signal every one screen scan (for example, 1/30 seconds) of the visual field image, or the emitted light color may be changed and color image processing may be performed. Be identifiable.

The apparatus of the present invention includes a feature point signal detection means 3 that separates the signal of the feature point P+"P* from the image signal of the visual field camera 3a.
2, a feature point coordinate detection means 33 for determining the coordinates of the detected feature point images pl to p on the visual field image 4, and distances Md□, dX□ on the visual field image 4 between adjacent feature points. ,dyI
,dy! The distance calculation means 34 between feature points calculates the distance between feature points.

Furthermore, the device of the present invention serves as a means for detecting posture changes including horizontal and vertical rotations of the head 2a with respect to the target visual field l, and for measuring the absolute coordinates of the gaze point Q.
A head position that compares the distance between two feature points corresponding to opposite sides of a quadrilateral whose vertices are images p1 to p4 of four feature points.
A posture calculation means 35 is provided.

(Effect) When the subject gazes at a certain point Q in the target visual field 1, the head 2
The coordinates of images p1 to p4 of four feature points detected from the image signal of the field-of-view camera 3a fixed at point a are the feature points P, -P
4 represents the positional relationship between the target field of view l and the subject's head 2a. When measuring the gaze point Q, the feature point P
The distance from the target visual field 1 where I-Pq is installed to the subject's head 2a and the inter-image distance dx of each feature point on the visual field image 4,
The relationship between d and d is measured in advance. Feature point P1~
The movement of the head 2a parallel to the plane where P is installed, the rotation of the head around the normal to the nucleus, and the movement of the head in the cough normal direction are respectively parallel movements of the feature point image p+-p4. , can be detected by measuring the rotation and the change in the distance between the images of the feature points. In addition, the rotation of the subject's head up and down or left and right is determined by the distances d 2.dy
It can be detected by comparing I and dyt.

Therefore, it is possible to convert the relative coordinates of the eyemark into absolute coordinates according to the detected data.

For example, if feature points are placed at the four corners of a rectangle,
When the normal line of the target field of view coincides with the line of sight direction of the field of view camera 3a, the positional relationship between the images pI-p4 of each feature point on the field of view image 4 also becomes non-rectangular (Fig. 4), and from the field of view camera 3a. As long as the distance to the target field of view 1 is constant, the shape and size of the rectangle on the field of view image 4 will remain the same even if the head 2a moves horizontally and vertically in parallel. The amount of parallel movement at this time is 1
It is determined from the amount of movement of the image on the visual field image 4 by focusing on each feature point. When the head 2a moves back and forth, the rectangular shape does not change, but the size changes. That is, the inter-image distance dX, dy of each feature point on the visual field image
changes at the same rate, by measuring this inter-image distance, the distance in the normal direction between the head and the target viewing plane is determined, and the positions of the eye mark image q and the feature point images p1 to p4 are determined. The absolute position of the point of interest is determined from the relationship. In addition, the head 2a
is accompanied by left and right rotational movement, as shown in FIG. 5, the distance between the images of the feature points is d□=dx! , the relationship is dyI≠dy2, and in the case of vertical rotation, d practice≠dX2% dy
The relationship is I = d y2, and from this relationship of image distance, the distance of each feature point P, -P4 from the field of view camera 3a, and the angle between the normal of the target field of view and the direction of the camera's line of sight can be found. The absolute coordinates of the gaze point Q are determined.

(Example) An example of the present invention will be described in detail below with reference to the accompanying drawings.

FIG. 2 is a panoramic view of one implementation system of the present invention, which analyzes the movement of a gaze point Q on a drawing board 1 of a subject 2 who is working in opposition to a drawing board 1 serving as a target field of view. On the diagram,
Nine LEDs (light emitting diodes) P, ~P, serving as feature points, are installed at regular intervals in the X and Y directions, D, respectively. The installation position of the LEDP, -P, is such that at least four LEDs are included in the photographing area of the visual field camera 3a fixed to the subject's head 2a, centering on the area where the subject 2 mainly gazes. .

An eye camera 3 including one visual field camera 3a and two eye mark cameras 3b is fixed to the subject's head 2a. The visual field camera 3a outputs an image signal of the drawing board 1 in the direction in which the face of the subject 2 is facing. eye mark camera 3
b detects the direction of the eyeballs of the subject 2 to the left and right sides and outputs the coordinate signals thereof. The output of the eye camera 3 is given to the recording device 20 and the LED controller 10, and the LED controller 10 controls the lighting timing of the LEDs P+ to P. The measurement information recorded in the recording device 20 is analyzed by the analysis device 30, and the position and posture of the subject's head 2a and the absolute coordinates of the gaze point Q are calculated.

The means for controlling the lighting timing of the LEDPs, . The visual field image signal is output from a visual field camera 3a of an eye camera (for example, an eye mark recorder manufactured by Knack Corporation), and the eye mark coordinate signal is output from an eye mark camera 3b. In an actual eye camera, the eye mark coordinate signal is combined with the visual field image signal as an image signal, and simultaneously output as a digital coordinate signal. The recording device 20 is
Both signals are recorded on a recording medium.

The LED controller 10 separates and outputs vertical synchronizing signals at 1/30 second intervals from the visual field image signal using a synchronizing signal separator 11, and distributes the vertical synchronizing signals using an LED lighting signal generator 12, as shown in FIG. An LED lighting signal is output at the timing shown in FIG. Therefore, the lighting interval of the two LEDs is 9730 seconds. As shown in FIG. 4, the visual field image signal includes at least four LED images pI-p4 and an eye mark image q, and is recorded by the recording device 20 on a recording medium such as a magnetic tape.

The recorded visual field image signal is transmitted to the regenerator 31 of the analysis device 30.
A tying signal is given to the LED coordinate detector 33 when the LED images p, to p4 are detected in the image signal. LED
For P, ~P, the brightness is set sufficiently high compared to the background.
The LED signal detector 32 binarizes the image signal with an appropriate threshold value, detects the signal of the image of L E D P r -P *, and from the detection timing detects the coordinate value on the visual field image 4 of the LED. The coordinate detector 33 detects. As for the coordinates of the LED image plNP9 on the visual field image, the Y coordinate is the number of scanning lines, and the X coordinate is detected by generating a clock pulse that divides 1 scanning line by 320.

Note that the embodiment shown in the figure is an example in which the recorded image signal is played back and the gaze point is analyzed, but the visual field camera 3 of the eye camera
It is also possible to directly transfer the visual field image signal of a to the LED signal detector 32 and analyze it in real time.

In the inter-LED distance calculation means 34, the LED coordinate detector 33
The inter-image paths Md□, dX□, d, and Idy2 between each 120 are calculated from the coordinates of the image of each LED obtained in , and from this inter-image distance, the head position/posture calculation means 35 calculates each L E D P
The distance between +~P4 and the head 2a is calculated, and the position/posture of the head 2a with respect to the drawing board l is determined. The gaze point absolute coordinate calculation means 36 calculates the absolute position of the gaze point Q from the head position/posture signal and the eye mark coordinate signal provided from the regenerator 31. In the example, the process up to the coordinate detection of the LED image p+"l)m is processed by hardware, and each detected coordinate value is transferred to the computer every 1/30 seconds, and the head is kept stationary for 9/30 seconds. Assuming that, the processing after calculation of the distance between images of the LED is performed by software.The details of the processing by the software are as follows.

When the face of the subject 2 is directly facing the drawing board 1, the visual field camera 3a
The field of view camera 3a is aligned so that the direction of
is fixed to the head 2a, the distance from the visual field camera 3a to the drawing board 1 in this standard posture is to, and the visual field camera 3a is fixed to the head 2a.
If the field of view image 4 output from the field of view image 4 is shown in FIG. %d
71. The conversion coefficients between dyt and the absolute distance on drawing board 1, D, are dxa= dxt= dxz dyo= d'y+ in the X direction and y direction, respectively.
= dyz (2).

The absolute position of the gaze point is calculated by using the conversion coefficients α8, α9, the coordinates of the eye mark image q on the visual field image 4 as (xL, yi),
If the coordinates of the feature point image pl are (XLI, 7LI) and the coordinates on the drawing board are (XLI, Y t, r ), then the absolute coordinates (X, Y) are: X=XL++ (XI XLI) °Ctx
(3) Y-YLI+ ()'I Vt+)
・αy (4). Here, α8=α8°, α,=α,. It is.

When the head 2a moves horizontally and vertically in parallel with the illustration 1 while maintaining the distance t0 from the illustration, the positional relationship between the feature points on the image does not change, so the absolute coordinates are calculated using the formulas (1) to ( 4
) is calculated.

When the head 2a moves back and forth and the distance t from the drawing board 1 changes from to, the distance between the images of the LEDs on the image changes in the X direction y.
Changes at the same rate in direction. The conversion coefficients at this time are given by dX and d, respectively, for the distance between images of the LED, and the absolute coordinates are similarly calculated using equations (3) and (4>).

Generally, the relationship between the distance t and the conversion coefficient is αX 1αxo' 0 α, = α.・(6)0, but correction is required depending on each field of view camera, and if the coefficients are η(dx) and η(dy), then the distance t
is expressed by the following equation from the image distance dx, d of the LED.

When the head 2a rotates, the image changes as shown in FIG. 5 (when rotating around the vertical axis).

In other words, the distance in the X direction between feature point images 9+ 9g and ps 91 is the same, but the y distance between p+ 92 and pz pa is the same.
The directional distance changes depending on the rotation angle of the head. Feature point p1
Assuming that the distance in the camera line-of-sight direction from the field-of-view camera 3a of and p is aspz, and the distance of p4 is b, the rotation angle of the head is as shown in FIG.

, and the distance of the gaze point Q from the field of view camera 3a is Sv = d,, + (dyz dl+) ・
λ (9) where λ= Sx /dx, dx = dx+-dxz
(10) is obtained from equation (7), and if this is C, the absolute coordinate of the image q of the point of gaze in the X direction is the absolute coordinate of the image q of the point of gaze in the Y direction: Y=YLI+(yI 3'+2)・αaS.

It is.

When the head 2a rotates in the vertical direction, the positional relationship between x and y is reversed and the calculation method is the same as described above.

(Effects of the Invention) As explained above, according to the present invention, by installing four or more feature points that indicate the absolute position on the object that the subject is gazing at, the eye camera can perform gaze point analysis on the left and right sides of the subject's head. Alternatively, it can be performed automatically even when rotation around a vertical axis is involved. In addition, not only can the absolute position of the gaze point be measured and the analysis of the gaze point automatically performed by tracking it, but the position and direction of the head can also be determined at any time, resulting in a device that is useful for analyzing various tasks and movements. be able to.

[BRIEF DESCRIPTION OF THE DRAWINGS] Fig. 1 is a prong diagram showing the configuration of an embodiment of the present invention, Fig. 2 is a panoramic view of the system, Fig. 3 is a diagram showing the lighting timing of nine LEDs, and Fig. 4 is a diagram showing the lighting timing of nine LEDs. The figure shows the positional relationship of the LEDs on the visual field image, Figure 5 shows the positional relationship of the LEDs on the visual field image when the head rotates, and Figure 6 shows the positional relationship of the LEDs on the visual field image when the head rotates. FIG. 7 is a diagram schematically showing the relationship between the target field of view and the field of view image at the reference position. In the figure: 1: Drawing board 3a: Field of view camera 4: Field of view image 33: LED coordinate detector 2a: Head 3b: Eye mark camera 32: LED signal detector 34: Inter-LED distance calculation means 35: Head position/posture Calculation method

Claims (2)

[Claims] (1) Field of view image (4) taken with field of view camera (3a)
In a gaze posture measurement device that measures the posture of the head (2a) with respect to the target field of view (1) from image information of the target field of view (1), four or more feature points (P_1 to P_9) installed at fixed positions within the target field of view (1) are used. and feature point signal detection means (
32), a feature point coordinate detection means (33) for determining the coordinates of the signal of the detected feature point on the visual field image (4), and a distance between adjacent feature points on the visual field image (4). Inter-feature point distance calculation means (34) and 4 on the visual field image (4)
head position/posture calculation means (35) including means for comparing distances between two feature points corresponding to opposite sides of a quadrilateral having the feature points as vertices; A gaze point posture measuring device capable of detecting posture changes including horizontal and vertical rotations of the portion (2a). (2) Visual field image (4) output from the eye camera (3)
absolute position of the gaze point, which measures the absolute coordinates of the image (q) of the gaze point on the visual field image (4) from the image information of and the eye mark coordinate signal indicating the position of the gaze point (Q) on the visual field image; In the measuring device, four or more feature points (P_1 to P_9) installed at fixed positions within the target field of view (1) and a feature point signal that detects the signal of the feature point from the image information of the field of view camera (3a). Detection means (
32), feature point coordinate detection means (33) for determining the coordinates of the image of the detected feature point on the visual field image (4), and determining the distance between adjacent feature points on the visual field image (4). A head including a feature point distance calculation means (34) and a means for comparing distances between two feature points corresponding to opposite sides of a quadrilateral having four feature points as vertices on the visual field image (4). head position/orientation calculation means (35), the head (
2a) A device for measuring the absolute position of the point of gaze that is capable of measuring the absolute coordinates of the point of gaze (Q) during posture changes including horizontal and vertical rotations.