JPH0583247B2 - - Google Patents

Description

[Detailed description of the invention]

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight display device, and more particularly to a line-of-sight display device that displays the movement of the line of sight of an observed person placed on a moving object, taking into account the movement of the moving object.

【0002】

[Prior Art and Problems to be Solved by the Invention] Conventionally, in analyzing the movement of the line of sight of a driver who is driving or piloting a car or aircraft, it has been necessary to analyze the influence of the movement of the driver's head on the movement of the line of sight. Only the movement of the driver's line of sight due to the movement of the driver's eyeballs was analyzed without taking into account the Therefore, it is necessary to accurately detect movement of the line of sight by taking into consideration not only eye movement but also head movement, and the applicant has already proposed this method.

[0003] In general, a vehicle body such as an automobile undergoes movements such as tilting and vibration as the vehicle travels. For example, when a car turns, the centrifugal force causes the car body to lean, and also when the road goes up or down. Since the driver is affected by the inclination of the vehicle body, the movement of his/her line of sight is also affected by the inclination. Conventionally,
When analyzing the driver's line of sight movement under such conditions, external factors such as the movement of the vehicle body are not taken into account, so it is difficult to imagine whether the driver's line of sight movement is based on the driver's own will. It became unclear whether this was caused by other external factors or not, and as a result, it was not possible to perform accurate gaze analysis.

[0004] This invention was made in order to solve the above-mentioned problems, and it distinguishes and displays the influence of the movement of a moving object based on the intention of an observed person placed on a moving object. The purpose of the present invention is to provide a line of sight display device that can

[0005]

[Means for Solving the Problems] A visual line display device according to the present invention includes an eyeball device detection means for detecting the movement of the eyeball of an observed person, and a head movement detection means for detecting the movement of the observed person's head. and a line-of-sight calculation means for calculating the movement of the observed person's line of sight by processing the output data from the eye movement detection means and the head movement detection means, and for photographing the situation within the visual field of the observed person. a photographing means, an object movement detecting means attached to a moving object and detecting the movement of the moving object, displaying an image photographed by the photographing means, and a line of sight calculation means on the displayed image; The apparatus includes display means for distinguishing and displaying the obtained line-of-sight movement and the movement caused by the moving object detected by the object movement detection means.

[0006]

[Operation] The line of sight display device in this invention has the following features:
The object movement detecting means detects the movement of the moving object, and the display means identifies and displays the movement of the detected moving object on the image photographed by the photographing means together with the movement of the observer's line of sight. Therefore, it is possible to distinguish and observe the movement of the line of sight based on the intention of the observed person and the influence of the movement of a moving object.

[0007]

Embodiment FIG. 1 is a block diagram of a gaze movement display device showing an embodiment of the present invention. Referring to FIG. 1, this eye movement display device includes a head movement detection device 6 that detects the driver's head movement using a magnetic sensor, and an eyeball movement detection device 5 that detects the driver's eye movement. , a gaze point processing device 7 that calculates the driver's line of sight data by processing the detected head movement data and eyeball movement data, an image processing device 8 that performs image processing of the obtained line of sight data, The camera 15 includes a camera 15 that photographs an external situation taken from a seat, a superimposer 9 that superimposes a line-of-sight signal subjected to image processing on a video signal from the camera 15, and a display device 10. This gaze movement display device further includes an acceleration sensor 11 attached to the body of the automobile, an integrator 12 that integrates the acceleration signal obtained by the acceleration sensor 11, and a signal of the coordinate system in which the acceleration sensor 11 is attached. It includes a coordinate converter 13 that converts signals in the coordinate system in the head motion detection device 6, and an image processing device 14 that converts the coordinate-exchanged signals into signals that can be superimposed by image processing.

[0008] In operation, eye movement data E detected by the eye movement detection device 5 and head movement data H detected by the head movement detection device 6 are provided to the gaze point processing device 7, Then, through arithmetic processing, line-of-sight data indicating the movement of the driver's line of sight is obtained. The line-of-sight data is provided to the image processing device 8, where it is converted into a line-of-sight signal that can be superimposed. On the other hand, the movement of the vehicle body is detected by an acceleration sensor 11 attached to the vehicle body, and the detected signal is integrated twice by an integrator 12. The integrated signal indicates the movement of the vehicle body in the coordinate system in which the acceleration sensor 11 is placed, and the coordinate converter 13 converts the coordinate system to that in the head motion detection device 6. A signal indicating the coordinate-converted movement of the vehicle body is sent to the image processing device 14.
where it is converted into a superimposable signal. The superimposer 9 is also given a video signal captured by a camera 15 of the external situation from the driver's seat. super imposer 9
superimposes a line-of-sight signal indicating the movement of the line of sight and a signal indicating the movement of the vehicle body on the video signal from the camera 15, and the resulting image is displayed on the screen of the display device 10.

[0009] Fifth is a block diagram of the head movement detection device 6 shown in FIG. Referring to FIG. 5, this head motion detection device 6 includes an excitation coil 61 that generates a reference magnetic field, a drive circuit 65 for AC driving the excitation coil 61, and a head motion detection device 6 that is attached to the head of an observed person. A sensor coil 62 , a detection circuit 63 that detects the voltage induced in the sensor coil 62 according to the movement of the head of the person to be observed, and an arithmetic process based on the voltage detected by the detection circuit 63 . and a CPU 64 for determining the head movement of the person. CPU
64 also controls a drive circuit 65. The excitation coil 61 has three coils wound in three axes (X, Y, Z) directions.
It consists of two coils. Each coil is supplied with driving power supply voltages generated from the drive circuit 65 and having different frequencies. Therefore, a magnetic field based on three axes is generated by the excitation coil 61, and the person to be observed is present in this magnetic field. The sensor coil 62 also has (X, Y, Z)
It is equipped with three detection coils wound in each direction. When the head of the person to be observed moves, an electromotive force is induced in each coil in the sensor coil 62 according to the movement. By detecting the electromotive force generated in each coil by the detection circuit 62, a signal corresponding to the movement of the head of the person to be observed is obtained. The CPU64 is
By performing arithmetic processing on the basis of this signal, head motion data H indicating the movement of the observed person's head is obtained and output.

[0010] FIG. 3 is a schematic diagram showing a coordinate system for head movement used in the head movement detection device 6. The aforementioned sensor coil 6 is placed on the forehead of the driver 2.
2 is installed. The parameter Hx shown in Figure 3,
Hy, Hz, Hθ, Hφ and Hψ each indicate the movement of the head movement in the direction shown in this figure. That is, Hx indicates horizontal parallel movement of the head, Hy indicates movement in the front-back direction, and Hz indicates vertical parallel movement. In addition, Hθ indicates the movement in the direction of tilting the neck left and right, Hφ indicates the rotational movement of the neck in the vertical direction,
Hψ indicates the rotational movement of the head in the left-right direction.

[0011] FIG. 4 is a schematic diagram showing how head motion is detected in the driver's seat. Referring to FIG. 4, an excitation coil 61 is mounted at a fixed location near the driver's seat, such as at the bottom of the steering wheel. The sensor coil 62 is attached to the driver's forehead.
When the driver's head moves, the excitation coil 61
The positional relationship between the sensor coil 62 and the sensor coil 62 changes relatively, and head motion detection is performed. The acceleration sensor 11 is attached to a part of the vehicle body, as shown in FIG.

[0012] FIG. 6 is a schematic diagram showing the detection principle in the eye movement detection device 5 shown in FIG. As an example, this device uses a scleral reflection method that utilizes the difference in reflectance between the white and iris of the corneal surface. As other methods, a corneal reflection method and a search coil method can also be applied. Referring to FIG.
Sensors 71 and 72 are arranged in front of the left and right eyes E1 and E2, respectively. A light emitting element 73 is arranged at the center of each sensor 71 and 72. As the light emitting element 73, ±
A light emitting diode with an infrared projection of about 21° is used. A light receiving element 74 is provided on both sides of the light emitting element 73. As the light receiving element 74, a photodiode with sharp directivity of approximately ±10° is used. The light projected from the light emitting element 73 toward the eyeball is reflected at the white and iris portions of the eye. The whites of the eyes and the iris have different reflectances, so if you amplify this difference in reflectance and take the difference, you will get the horizontal (left and right) eye movements as an output, and if you add the sum, you will get the vertical (up and down) eye movements. The movement is obtained as the output.

[0013] The positions of the sensors 71 and 72 with respect to the eyeball are different in the horizontal direction and the vertical direction, and the horizontal sensor 71 detects the reflected light in the center with respect to the top and bottom of the eyeball, and the vertical sensor 72 detects the reflected light downward. and are arranged so as to detect the reflected light. A horizontal sensor 71 is placed on one eye E1, and a vertical sensor 72 is placed on the other eye E2. If these are used simultaneously, two-dimensional eye movement can be detected. A method of determining the moving speed, moving direction, moving distance, gaze time, etc. of the eyeball using such a detection principle is disclosed in, for example, Japanese Patent Laid-Open No. 126140/1983. Therefore, data E indicating the driver's eye movement is obtained from the eye movement detection device 5 shown in FIG. 1 and is provided to the gaze point processing device 7.

[0014] FIG. 7 is a schematic diagram for explaining the movement of the line of sight due to head movement. Referring to FIG.
Generally, when a visual object moves, the eyeball moves to follow the visual object. Instead of moving the eyeballs, it is also possible to track the visual object by moving the head. Usually, both movements are used together. Head movements include parallel movements caused by the orientation of the feet and spine, and rotational movements realized by the neck, spine, hips, legs, and the like. This head movement is converted into an eyeball rotation angle by a method described later, and a head movement corrected eyeball rotation angle α shown in FIG. 7 is obtained. Figure 1
In the gaze point processing device 7 shown in , line of sight data is expressed by a combination of the head movement corrected eyeball rotation angle α and the eyeball rotation angle θ. Here, the sum of the rotation angles α and θ is collectively defined as the line of sight.

[0015] The processing in the gaze point processing device 7 shown in FIG. 1 will be described below. The gaze point processing device 7 is
Corrected head movement data H and eye movement data E
and received. It is assumed that (Hx, Hy, Hz) are given as horizontal motion components of data H, and (Hφ, Hθ, Hψ) are given as rotational motion components.
The line of sight changes due to the horizontal component of head movement (Hx, Hy, Hz), which can be expressed as the eye rotation angle (Ex, Ey).
The following formula is used to convert to . Here, D is the distance between the subject and the observation target.

[0016]

[Math. 1] Ex=180/π・tan ^-1 Hx/(D+Hy) ...(1) Ey=180/π・tan ^-1 Hz/(D+Hy) ...(2)

[0017] When the head is tilted by Hθ towards the left shoulder or the right shoulder, the coordinates of the eye movement system rotate. Therefore, the eye movement coordinate system (Xe,
Ye) in the coordinate system (Xe′,
Ye′). Therefore, the following formula is used.

[0018]

[Mathematical 2]
Yh) is expressed by the following equation from equations (1) and (2).

【0019】

[Math. 3] Xh=Ex+Hψ...(5) Yh=Ey+Hψ...(6) Therefore, the movement of the line of sight (Xv, Yv) considering the movement of the head can be calculated from equations (3) to (6). It is expressed by the following formula.

【0020】

[Mathematical 4] It is possible to reproduce normal eye movements. The above-described processing is performed in the gaze point processing device 7 shown in FIG. 1, and gaze data indicating the movement of the gaze is provided to the image processing device 8.

[0021] Next, the process of detecting the movement of the vehicle body will be described. As the acceleration sensor 11 shown in FIG. 1, for example, an ultrasonic gyroscope using a tuning fork using a ceramic vibrator is used. By attaching such an acceleration sensor 11 to the vehicle body, the acceleration when the frame of the vehicle moves is detected. A coordinate system for detecting acceleration of a vehicle body is shown, for example, as shown in FIG. Therefore, signals (X″, Y″, Z″) indicating the acceleration of the vehicle body are obtained by the acceleration sensor 11. Here, ″
indicates second differentiation. This acceleration signal is transferred to the integrator 12
By integrating twice, vehicle body movement data (X, Y, Z) indicating the movement of the vehicle body is obtained. This data (X, Y, Z) is the sum of the rotation and parallel movement of the coordinate system with respect to the reference coordinates (Xo, Yo, Zo) when the car is stopped. The coordinate converter 13 calculates the rotation angle and the amount of translation of this acceleration coordinate system, and converts them into coordinate system data for head motion detection. As a result, data indicating the movement of the vehicle body can be obtained in the coordinate system shown in FIG. Image processing device 1
4 converts the coordinate-transformed data signal into a superimposable signal, and supplies the converted signal to the superimposer 9.

[0022] The superimposer 9 has a camera 15.
A video signal showing the situation seen from the driver's seat was also provided. Therefore, in the superimposer 9, a line-of-sight signal indicating the movement of the line of sight and a signal indicating the movement of the vehicle body are superimposed on the video signal from the camera 15. The superimposed video signal is applied to the display device 10, and the superimposed video is displayed on the screen of the display device 10.

[0023] FIG. 8 is a screen diagram showing an example of a screen displayed on the display device 10. Referring to FIG. 8, the movement of the line of sight is displayed on the background image, and a indicates the movement of the vehicle body;
b shows the movement of the driver's line of sight according to his own will. As shown in Figure 8, the direction of movement of the car body and the direction of movement of the driver's own line of sight may match, but in reality, while driving, the car body moves in the opposite direction to the direction the driver is trying to look. There are many things.

[0024] In this way, on the screen of the display device 10, it is possible to clearly distinguish and display the eye movement caused by the driver's own will and the movement of the vehicle body. It becomes possible to analyze line of sight more accurately. By using the analysis results, it will be possible to accurately obtain important data regarding ergonomics in order to obtain easier-to-read instruments in automobiles and other vehicles.

【0025】

Effects of the Invention As described above, according to the present invention, an object movement detection means for detecting the movement of a moving object placed on an observed person is provided, and the movement of the object is detected as a movement of the observed person's line of sight. Since a display means for distinguishing and displaying is provided, a gaze display device is obtained that can distinguish and display the movement of the gaze based on the intention of the observed person and the influence due to the movement of a moving object.

[Brief explanation of drawings]

FIG. 1 is a block diagram of a gaze movement display device showing an embodiment of the present invention.

2 is a coordinate diagram showing a coordinate system in the acceleration sensor shown in FIG. 1. FIG.

3 is a schematic diagram showing the coordinate system and directions of each motion parameter used in the head motion detection device shown in FIG. 1. FIG.

FIG. 4 is a schematic diagram showing how head motion is detected in the driver's seat.

5 is a block diagram of the head motion detection device shown in FIG. 1. FIG.

6 is a schematic diagram showing the detection principle in the eye movement detection device shown in FIG. 1. FIG.

FIG. 7 is a schematic diagram for explaining the movement of the line of sight during head movement.

FIG. 8 is a screen diagram showing a display example of detected eye movement.

[Explanation of symbols]

5 Eye movement detection device 6 Head movement detection device 7 Gaze point processing device 9 Superimposer 10 Display device 11 Acceleration sensor 12 Integrator 13 Coordinate converter.

Claims (1)

[Claims] 1. A line-of-sight display device that displays the movement of the line of sight of an observed person placed on a moving object, comprising: eye movement detection means for detecting the movement of the eyeballs of the observed person; a head movement detection means for detecting movement of the head of the person; and a line-of-sight calculation for calculating the movement of the line of sight of the observed person by processing output data from the eye movement detection means and the head movement detection means. means for photographing a situation within the line of sight of the observed person; object movement detection means attached to the moving object to detect movement of the moving object; displaying an image of the moving object, and identifying and displaying the movement of the line of sight obtained by the line of sight calculation means and the movement of the moving object detected by the object movement detection means on the displayed image. A line-of-sight display device including display means.