JPH04288123A - Line-of-sight display device - Google Patents

Abstract

PURPOSE:To determine motions of the line of sight of a driver himself accurately by detecting and displaying the moves of the line of sight and motions of an automobile body being separated clearly based on the will of the driver. CONSTITUTION:A line-of-sight display device which displays motions of the line of sight of a driver of an automobile is disclosed. An acceleration sensor mounted on the body of the automobile detects motions such as inclination of the body generated with the running. The motions of the body detected is detected and displayed being superimposed on a background image on a screen of the display device 10 together with the motions of the line of sight as obtained by the computation of an eyeball motion data and a head motion data.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight display device, and more particularly to a line-of-sight display device that displays the movement of the line of sight of an observer placed on a moving object, taking into account the movement of the moving object.

0002

[Prior Art and Problems to be Solved by the Invention] Conventionally, in analyzing the movement of the line of sight of a driver who is driving or piloting a car or aircraft, it has been difficult to analyze the influence of the movement of the driver's head on the movement of the line of sight. Only the movement of the driver's line of sight due to the movement of the driver's eyeballs was analyzed without taking into account the Therefore, it is necessary to accurately detect movement of the line of sight by taking into account not only eye movement but also head movement, and the applicant has already proposed this method.

[0003] Generally, the body of a vehicle such as an automobile undergoes movements such as tilting and vibration as the vehicle moves. For example, when a car turns, the centrifugal force causes the car body to tilt.
The vehicle body also tilts due to the rise or fall of the road. Since the driver is affected by the inclination of the vehicle body, the movement of his/her line of sight is also affected by the inclination. Conventionally, when analyzing the driver's line of sight movement under such conditions, external factors such as the movement of the vehicle body are not taken into account, so the driver's line of sight movement based on the driver's own will is not considered. It became ambiguous as to whether this was due to the presence of the problem or some other external factor, and as a result, it was not possible to perform an accurate line of sight analysis.

[0004] The present invention was made to solve the above-mentioned problems, and is a system that distinguishes between the voluntary movement of the gaze of an observer placed on a moving object and the influence of the movement of the moving object. An object of the present invention is to provide a line of sight display device that can display a line of sight.

[0005]

[Means for Solving the Problems] A line of sight display device according to the present invention includes an eye movement detecting means for detecting the movement of the eyeballs of an observed person, and a head movement detecting means for detecting the movement of the observed person's head. and a line-of-sight calculation means for calculating the movement of the observed person's line of sight by processing the output data from the eye movement detection means and the head movement detection means, and for photographing the situation within the visual field of the observed person. a photographing means; an object movement detecting means attached to a moving object to detect the movement of the moving object; and an object movement detecting means for displaying an image photographed by the photographing means, and for displaying a line of sight obtained by a line of sight calculation means on the displayed image. and display means for identifying and displaying the motion and the motion caused by the moving object detected by the object motion detection means.

[0006]

[Operation] In the line-of-sight display device according to the present invention, the object movement detecting means detects the movement of the moving object, and the display means detects the movement of the moving object detected on the image taken by the photographing means. Identification is displayed along with the movement of the Therefore, it is possible to distinguish and observe the movement of the line of sight based on the intention of the person to be observed and the influence of the movement of a moving object.

[0007]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a block diagram of a gaze movement display device showing an embodiment of the present invention. Referring to FIG. 1, this eye movement display device includes a head movement detection device 6 that detects a driver's head movement using a magnetic sensor, and an eye movement detection device 5 that detects a driver's eye movement. , a gaze point processing device 7 that calculates the driver's line of sight data by processing the detected head movement data and eyeball movement data, and an image processing device 8 that performs image processing of the obtained line of sight data.
and camera 1 that photographs the external situation from the driver's seat.
5, and a superimposer 9 that superimposes the image-processed line-of-sight signal on the video signal from the camera 15.
and a display device 10. This gaze movement display device is
Furthermore, an acceleration sensor 11 attached to the vehicle body
an integrator 12 that integrates the acceleration signal obtained by the acceleration sensor 11; a coordinate converter 13 that converts a signal in the coordinate system in which the acceleration sensor 11 is attached to a signal in the coordinate system in the head motion detection device 6; and an image processing device 14 that converts the coordinate-transformed signal into a superimposable signal through image processing.

In operation, eye movement data E detected by the eye movement detection device 5 and head movement data H detected by the head movement detection device 6 are provided to the gaze point processing device 7, where they are processed by arithmetic processing. Gaze data indicating the movement of the driver's gaze can be obtained. The line-of-sight data is provided to the image processing device 8, where it is converted into a line-of-sight signal that can be superimposed. On the other hand, the motion of the vehicle body is detected by an acceleration sensor 11 attached to the vehicle body, and the detected signal is integrated twice by an integrator 12. The integrated signal indicates the movement of the vehicle body in the coordinate system in which the acceleration sensor 11 is placed, and the coordinate converter 13 converts the coordinate system to that in the head motion detection device 6. The coordinate-transformed signal indicating the movement of the vehicle body is given to the image processing device 14, where it is converted into a signal that can be superimposed. Superimposer 9 has camera 1
5, a video signal capturing the external situation from the driver's seat is also provided. The superimposer 9 superimposes the line-of-sight signal indicating the movement of the line of sight and the signal indicating the movement of the vehicle body on the video signal from the camera 15, and the resulting image is displayed on the screen of the display device 10. .

FIG. 5 shows the head motion detection device 6 shown in FIG.
FIG. Referring to FIG. 5, this head motion detection device 6 includes an excitation coil 61 that generates a reference magnetic field, a drive circuit 65 for AC driving the excitation coil 61, and a head motion detection device 6 that is attached to the head of an observed person. A sensor coil 62 , a detection circuit 63 that detects the voltage induced in the sensor coil 62 according to the movement of the head of the person to be observed, and a calculation process based on the voltage detected by the detection circuit 63 to detect the person being observed. and a CPU 64 that calculates head movement. The CPU 64 also controls a drive circuit 65. The excitation coil 61 has 3
It is composed of three coils wound in the axial (X, Y, Z) directions. Each coil is supplied with driving power supply voltages generated from the drive circuit 65 and having different frequencies. Therefore, a magnetic field based on three axes is generated by the excitation coil 61, and the person to be observed is present in the magnetic field. The sensor coil 62 also includes three detection coils wound in the (X, Y, Z) directions. When the head of the person to be observed moves, an electromotive force is induced in each coil in the sensor coil 62 according to the movement. By detecting the electromotive force generated in each coil by the detection circuit 63, a signal corresponding to the movement of the head of the person to be observed is obtained. The CPU 64 performs arithmetic processing on the basis of this signal to obtain head movement data H indicating the movement of the head of the observed person, and outputs it.

FIG. 3 is a schematic diagram showing a coordinate system for head motion used in the head motion detection device 6. As shown in FIG. The aforementioned sensor coil 62 is attached to the forehead of the driver 2. Parameters Hx, Hy, Hz, Hθ, Hφ shown in Figure 3
and Hψ indicate the movement of the head movement in the direction shown in this figure, respectively. That is, Hx indicates horizontal parallel motion of the head, Hy indicates forward-backward motion, and Hz indicates vertical parallel motion. In addition, Hθ indicates the movement in the direction of tilting the neck to the left and right, Hφ indicates the rotational movement of the neck in the vertical direction, and Hψ
indicates the rotational movement of the head in the left-right direction.

FIG. 4 is a schematic diagram showing head motion detection in the driver's seat. Referring to FIG. 4, an excitation coil 61 is mounted at a fixed location near the driver's seat, such as at the bottom of the steering wheel. The sensor coil 62 is attached to the driver's forehead. As the driver's head moves, the positional relationship between the excitation coil 61 and the sensor coil 62 changes relative to each other, and head movement is detected. The acceleration sensor 11 is attached to a part of the vehicle body, as shown in FIG.

FIG. 6 shows the eye movement detection device 5 shown in FIG.
FIG. 2 is a schematic diagram showing the detection principle in FIG. With this device,
An example of this method uses a scleral reflection method that utilizes the difference in reflectance between the white and iris of the corneal surface. As other methods, a corneal reflection method and a search coil method can also be applied. Referring to FIG. 6, sensors 71 and 72 are placed in front of left and right eyes E1 and E2, respectively. Each sensor 7
A light emitting element 73 is arranged at the center of 1 and 72. As the light emitting element 73, a light emitting diode that emits infrared radiation with a relatively wide directivity of about ±21° is used. A light receiving element 74 is provided on both sides of the light emitting element 73. As the light receiving element 74, a photodiode with sharp directivity of approximately ±10° is used. The light projected from the light emitting element 73 toward the eyeball is reflected at the white and iris portions of the eye. The whites of the eyes and the iris have different reflectances, so if you amplify this difference in reflectance and take the difference, you will get the horizontal (left and right) eye movements as an output, and if you add the sum, you will get the vertical (up and down) eye movements. The movement of is obtained as output.

[0013] The positions of the sensors 71 and 72 relative to the eyeball are different in the horizontal and vertical directions, with the horizontal sensor 71 detecting reflected light at the center of the eyeball above and below, and the vertical sensor 72 detecting reflected light downward. and are arranged so as to detect the reflected light. A horizontal sensor 71 is placed on one eye E1, and a vertical sensor 72 is placed on the other eye E2. If these are used simultaneously, two-dimensional eye movement can be detected. Using this detection principle, the moving speed, direction, distance, and
For example, a method for determining gaze time, etc.
It is disclosed in Japanese Patent No. 26140. Therefore, data E indicating the driver's eye movement is obtained from the eye movement detection device 5 shown in FIG. 1 and is provided to the gaze point processing device 7.

FIG. 7 is a schematic diagram for explaining the movement of the line of sight due to head movement. Referring to FIG. 7, generally when a visual object moves, the eyeball moves to follow the visual object. Instead of moving the eyeballs, it is also possible to track the visual object by moving the head. Usually, both movements are used together. Head movements include parallel movements caused by the orientation of the feet and spine, and rotational movements realized by the neck, spine, hips, legs, etc. This head movement is converted into an eyeball rotation angle by a method described later, and a head movement corrected eyeball rotation angle α shown in FIG. 7 is obtained. In the gaze point processing device 7 shown in FIG. 1, line of sight data is expressed by a combination of the head movement corrected eyeball rotation angle α and the eyeball rotation angle θ. Here, the sum of the rotation angles α and θ is collectively defined as the line of sight.

The processing in the gaze point processing device 7 shown in FIG. 1 will be explained below. The gaze point processing device 7 receives the corrected head movement data H and eye movement data E. It is assumed that (Hx, Hy, Hz) is given as the horizontal motion component of data H, and (Hφ, Hθ, Hψ) is given as the rotational motion component. Horizontal component of head movement (Hx, Hy,
The line of sight changes depending on the eyeball rotation angle (E
x, Ey), the following formula is used. Here D
is the distance between the subject and the observation target.

[0016]

[Math 1]

When the head is tilted by Hθ towards the left or right shoulder, the coordinates of the eye movement system rotate. Therefore, H
It is necessary to convert the eye movement coordinate system (Xe, Ye) tilted by θ to a coordinate system (Xe', Ye') orthogonal to the original observation object. Therefore, the following formula is used.

[0018]

[Formula 2] Xe′=Xe・cosHθ+Ye・sinHθ
...(3) Ye'=-Xe・
sinHθ+Ye・cosHθ
(4) The movement of the line of sight (Xh, Yh) realized by head movement is expressed by the following equation from equations (1) and (2).

[0019]

[Math. 3] Xh=Ex+Hψ

...(5) Yh=Ey+Hφ

(6) Therefore, the movement of the line of sight (Xv, Yv) in consideration of the movement of the head is expressed by the following equation from equations (3) to (6).

[0020]

[Formula 4] Xv=Xe′+Xh
…
...(7) Yv=Ye′+Yh

...(8) By using equations (7) and (8), it is possible to reproduce the normal movement of the line of sight that is performed by combining head movement and eyeball movement. The above processing is performed in the gaze point processing device 7 shown in FIG.
given to.

Next, the process of detecting the movement of the vehicle body will be explained. As the acceleration sensor 11 shown in FIG. 1, for example, an ultrasonic gyro using a tuning fork using a ceramic vibrator is used. By attaching such an acceleration sensor 11 to the vehicle body, the acceleration when the frame of the vehicle moves is detected. A coordinate system for detecting acceleration of a vehicle body is shown, for example, as shown in FIG. Therefore, signals (X'', Y'', Z'') indicating the acceleration of the vehicle body are obtained by the acceleration sensor 11. Here, '' indicates a second differential. By integrating this acceleration signal twice by the integrator 12, body movement data (
X, Y, Z) are obtained. This data (X, Y, Z) is the reference coordinate (Xo, Yo, Zo) when the car is stopped.
), it is a combination of rotation and translation of the coordinate system. The coordinate converter 13 calculates the rotation angle and the amount of translation of this acceleration coordinate system, and converts them into coordinate system data for head motion detection. As a result, data indicating the movement of the vehicle body can be obtained in the coordinate system shown in FIG. The image processing device 14 converts the coordinate-transformed data signal into a superimposable signal, and supplies the converted signal to the superimposer 9.

The superimposer 9 is also provided with a video signal taken by a camera 15 and showing the situation seen from the driver's seat. Therefore, in the superimposer 9, a line-of-sight signal indicating the movement of the line of sight and a signal indicating the movement of the vehicle body are superimposed on the video signal from the camera 15. The superimposed video signal is applied to the display device 10, and the superimposed video is displayed on the screen of the display device 10.

FIG. 8 is a screen diagram showing an example of a screen displayed on the display device 10. As shown in FIG. Referring to Figure 8, the movement of the line of sight is displayed on the background image, where a indicates the movement of the vehicle body, and b indicates the movement of the line of sight due to the driver's own will. There is. As shown in Figure 8, the direction of movement of the car body and the direction of movement of the driver's own line of sight may match, but in reality, while driving, the car body moves in the opposite direction to the direction the driver is trying to look. There are many things.

In this way, it is possible to clearly distinguish and display on the screen of the display device 10 the movement of the driver's own will and the movement of the vehicle body. This makes it possible to more accurately analyze the line of sight of people. By using the analysis results, it will be possible to accurately obtain important data regarding ergonomics in order to obtain easier-to-read instruments in automobiles and other vehicles.

[0025]

As described above, according to the present invention, an object movement detecting means for detecting the movement of a moving object placed by an observed person is provided, and the movement of the object is detected based on the movement of the observed person's line of sight. Since a display means for distinguishing and displaying was provided, a gaze display device was obtained that can distinguish and display the movement of the gaze based on the intention of the observed person and the influence due to the movement of a moving object.

[Brief explanation of the drawing]

FIG. 1 is a block diagram of a gaze movement display device showing an embodiment of the present invention.

FIG. 2 is a coordinate diagram showing a coordinate system in the acceleration sensor shown in FIG. 1;

3 is a schematic diagram showing a coordinate system and directions of each motion parameter used in the head motion detection device shown in FIG. 1. FIG.

FIG. 4 is a schematic diagram showing head motion detection in the driver's seat.

FIG. 5 is a block diagram of the head motion detection device shown in FIG. 1.

6 is a schematic diagram showing the detection principle in the eye movement detection device shown in FIG. 1. FIG.

FIG. 7 is a schematic diagram for explaining the movement of the line of sight during head movement.

FIG. 8 is a screen diagram showing a display example of detected eye movement.

[Explanation of symbols]

5 Eye movement detection device 6 Head movement detection device 7 Gaze point processing device 9. Superimposer 10 Display device 11 Acceleration sensor 12 Integrator 13 Coordinate converter

Claims (1)

[Claims] 1. A line-of-sight display device that displays the movement of the line of sight of an observed person placed on a moving object, comprising: an eye movement detecting means for detecting a movement of the eyeball of the observed person; a head movement detection means for detecting movement of the head of the person; and a line of sight calculation for calculating the movement of the line of sight of the observed person by processing output data from the eye movement detection means and the head movement detection means. a photographing means for photographing a situation within the visual field of the observed person; an object movement detecting means attached to the moving object and detecting movement of the moving object; and an image photographed by the photographing means. and display means for identifying and displaying the movement of the line of sight obtained by the line of sight calculation means and the movement of the moving object detected by the object movement detection means on the displayed image. Gaze display device.