JP2507913B2 - Eye movement tracking type visual presentation device by projection method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye movement tracking type visual presentation device for presenting a highly realistic image to a person.

[0002]

2. Description of the Related Art In order to visually display a scene photographed by a TV camera to a person with a realistic sensation, or to display a virtual world created in a computer with a realistic sensation to a person. , Head mounted displays have been developed. This head-mounted display is a device in which a small TV monitor is independently arranged in front of both the left and right eyes of a person and it is possible to present a person with a stereoscopic image.

Generally, a graphics image generated on a TV camera or a computer display is operated according to the movement of the head measured by a goniometer or a magnetic method, and the image thus generated is displayed on both the left and right sides. By presenting it on the eye display, it is as if you are looking at the outside world at the position where the TV camera is placed, or whether you are actually seeing the world in the image world presented by computer graphics. Such a feeling can be generated. Head-mounted displays have been attracting attention as visual display devices for so-called artificial reality systems, but on the other hand, since a small display is used to present images to the left and right eyes, the outer frame of the display is always in the field of view, providing a sense of realism. It has also been pointed out that there is a drawback in that

[0004]

It is said that a person usually has a visual field with a visual angle of ± 40 ° about the center of the eyeball. Therefore, in a head-mounted display, unless a TV monitor having a screen size that covers this area is placed in front of both eyes, a person can see the frame of the TV monitor, and the resulting image is displayed through the frame. You will feel like you are looking into. One way to solve these problems is to introduce a TV monitor with a screen that covers the maximum field of view of a person, but generally there is a limit to the number of scan lines that can display an image on the TV monitor (for example, a normal In the NTSC system, there are 525 scanning lines and a high-definition TV monitor has 2,000 scanning lines.) If a TV monitor with a large screen is used, the resolution of the image decreases. This decrease in resolution presents a blurred image to people, which is one of the causes of impairing the sense of presence.

A solution to this problem is to develop a TV monitor with a large number of scanning lines capable of displaying a high-resolution image on the screen without impairing the human visual acuity. Developing a high resolution TV monitor is technically difficult. The present invention has been made to provide a method of expanding the field of view of such a current head mounted display having a narrow field of view without deteriorating the resolution. A person can see only the gazing point (the center axis of the eyeball, that is, the position of the object looking toward the line of sight) and its narrow area (usually within ± 5 ° of the line of sight), and other areas It is known that the image looks blurry. The present invention relates to a device capable of presenting to an operator an image equivalent to that obtained by using a display with a wide field of view and a high resolution by an optomechatronic method by paying attention to such characteristics of a living body. In order to display only such a gazing point of a person and its narrow surroundings with high resolution, first, the gazing point of a person, that is, the line of sight of a person is detected, and a high-resolution image is tracked to the line of sight of the person. A good solution is to give it to others. The applicant of this patent application has previously proposed a technique of using a spheroidal mirror to make a high-resolution image follow a human line of sight. The line-of-sight tracking type head mount display using this spheroidal mirror can easily detect the line of sight of a person, can perform good image tracking, and can achieve good virtual reality. However, there is a problem that the configuration of the device becomes complicated because it is necessary to match the human eyeball with the focus of the spheroidal mirror. Furthermore, the image is reflected by a spheroidal mirror, and the image is projected from the back of the person through the side of the face in order to project it on the eyeball of the person. There is a problem of being limited. The present invention has been made in view of the above circumstances, and is capable of presenting a wide-field image as well as a high-resolution image at the gazing point, and detecting the line of sight of a person. It is an object of the present invention to provide an eye movement tracking type visual presentation device that is easy to perform, can perform good image tracking, can achieve good virtual reality, and has a simple device configuration. It is what

[0006]

To achieve this object, an eye movement tracking type visual presentation device according to the projection method of the present invention creates an image corresponding to the line of sight of the wearer and an image creating device. An image presenting device that presents the image to the wearer, and an eye position measuring device that measures the position of the eyeball of the wearer, and the image presenting device is a display device that displays the image created by the drawing device, and the wearer. An image transmission element that is disposed in front of the eyeballs and that allows an image displayed on the display device to enter, transmit, and output, and a variable attitude that is disposed in the optical path between the display device and the image transmission device. A mirror and a mirror controller for controlling the posture of the mirror are provided, and the configuration is such that the imaging device is controlled and the mirror driving device is controlled by a signal from the eye position measuring device.

[0007]

[Operation] The optical system is operated in response to the movement of the eyeball so that the line of sight always looks at one point in the center of the monitor, and at the same time, in the center area of the monitor, when the monitor is placed directly in front of the eyeball, Is a device that presents a high-resolution image of the area that should be gazed at, and presents a low-resolution image of the surrounding area of the area.The image is attached from the front of the wearer using an image transmissive element that transmits the image. Given to the person.

First, when the image transmissive element is arranged immediately in front of the eyeball of the wearer and one point at the image transmissive element is gazed at, a line of sight thereof is transmitted through the image transmissive element and further taken out by the half mirror. When the line-of-sight direction is changed to face the center of the monitor, the bidirectionality of the optical system allows the wearer to see the center of the monitor through the image transmissive element. On the other hand, when the eye moves and the direction of the line of sight changes, the sensor placed on the axis passing through the center of the monitor detects and detects the movement of the eye, and the direction of the mirror is controlled accordingly, and Make sure the line of sight is fixed in the center of the monitor. At the same time, the drawing device is controlled according to the movement of the eyeball so that an image in the line-of-sight direction according to the movement of the eyeball is displayed at the center of the monitor. The control of the mirror and the control of the drawing device are performed by the respective control devices based on the eye position signal from the line-of-sight signal transmission device.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The basic idea of the present invention is shown in FIGS. 1 (a) and 1 (b). For example, consider a case where a person looks at the object 10 with the naked eye. First, it is assumed that the eyeball is at the position indicated by the reference numeral 15 and the line of sight thereof faces the direction of 50, and the point P ₁ on the object 10 is focused. Next, change the direction of the line of sight and move the eyeball to 15 '.
To the point P ₂ with the line of sight directed in the direction of 50 ′.
Imagine a situation of gazing at. At this time, due to the characteristics of the human visual system, the person can obtain a high-resolution image near the gazing points P ₁ and P ₂ at each eyeball position. On the other hand, since the state of the outer position deviated from the gazing point cannot be clearly discriminated, an image with a wide angle of view and low resolution can be obtained. Considering that such a visual characteristic is utilized, a high-resolution image is always presented only for the gazing point in the line-of-sight direction, so that an equivalently wide-angle and high-resolution image is presented.

Therefore, consider a system as shown in FIG. 1 (b) as a display system equivalent to that shown in FIG. 1 (a). That is, first, as shown in (a) of FIG. 1 (b), a display system in which the position of the monitor moves in the direction of the line of sight according to the movement of the eyeball, and as shown in (b) of (b) of FIG. Assume an imaging system placed so that the position O ′ and the optical axis coincide with the position O and the line of sight of the eyeball in FIG. Here, when the eyeball is at the position 15 in FIG. 1B corresponding to the position 15 in FIG. 1A, the camera device 5 is directed toward the line of sight 50, the point P ₁ is imaged, and the gazing point image is obtained. Line of sight 5 with high resolution
The image is displayed on the monitor 122 in the 0 direction, and when the eyeball moves to 15 ′, the movement is detected and the monitor 122 is moved to the position of 122 ′ in the direction of the line of sight 50 ′ so as to follow the movement, and at the same time, the camera is also moved. Further, it is moved in a direction corresponding to the line of sight 50 ', the point P ₂ is imaged, and the gazing point image is captured with high resolution on the monitor 122 at the position 122'. Here, if the size of the monitor is set to a size that covers the narrow visual field of the eye with high visual acuity, a person can receive an image similar to that of FIG. 1A even in the system of FIG. 1B. You can Here, as shown in (a) of FIG.
When a monitor with the same number of scanning lines is used, a similar image presentation effect can be obtained on a monitor with a small screen, as compared with the case where a fixed monitor that presents the entire surface of the object 10 at a time is used. , It becomes possible to present a higher resolution image. Next, as in the case of creating a narrow-field image, a wide-field image is created so as to match the line of sight of the wearer, and is presented on a monitor large enough to cover the wide field of view of the eye where a person has low vision. The view image is superimposed on the wide view image and given to the wearer. However, since it is difficult to move the monitor 122 according to the high-speed movement of the eyeball as shown in (a) of FIG. 1B, in the present invention, an effect equivalent to this is realized by using an optical system. Think about it.

The details will be described below with reference to the drawings showing an embodiment. In FIG. 2, reference numeral 1 is an eye movement tracking type visual presentation device. The eye movement tracking type visual presentation device 1 includes a device 100a for the right eye and a device 100b for the left eye.
The device for right eye 100a is a device for handling an image given to the right eye 15a of the wearer 14, and the device for left eye 100b is a device handling an image given to the left eye 15b of the wearer 14,
Since the right-eye device 100a and the left-eye device 100b have symmetrical configurations, the right-eye device 100a will be described here. The device 100a for the right eye includes a drawing device 2, an image presenting device 3, and an eyeball position measuring device 4. The drawing device 2 is a device mainly for creating an image to be presented on a monitor. The image drawing device 2 is a computer graphics device or an image pickup device 2 that takes an image of the object 10 and creates an image.
However, in any case, the created image has contents corresponding to the line of sight 50 of the wearer 14, and therefore the contents of the image also change if the line of sight 50 of the wearer 14 changes. It changes. In the embodiment described below, a case will be described in which an imaging device 2a that captures an image of the object 10 and creates an image is used as the drawing device 2.
The image presentation device 3 is a device for presenting the image created by the drawing device 2 to the wearer 14. The eye position measuring device 4 measures the eye movement of the wearer 14 to measure the eye movement.
This is a device for controlling the (imaging device 2 a) and the image presentation device 3.

The image pickup device 2a comprises a camera device 5 for taking an image of the object 10 and an image pickup direction control device 6 for controlling the image pickup direction of the camera device 5. In this embodiment, the camera device 5 includes a pair of wide-view TV camera 5A and narrow-view TV camera 5a, and the imaging direction control device 6 includes a half mirror 60, a scanner 61, a scanner drive circuit 62, and a scanner controller 63. Has become.

The wide-view TV camera 5A and the narrow-view TV camera 5a have an autofocus function,
The focal points of A and 5a coincide with the same point P on the surface of the object 10. The wide-field image 40 captured by the wide-field TV camera 5A is displayed on the wide-field monitor 24A after being image-processed by the image processing device 12A. The narrow-field image 41 captured by the narrow-field TV camera 5a is the image processing device 1
The image is processed at 2a and then displayed on the narrow-field monitor 24a. The point to be noted here is that the wide-field image 40 and the narrow-field image 41 are later superimposed by superimposing, but the portion of the wide-field image 40 where the narrow-field image 41 should be superimposed is the center. The image is missing and does not exist. The image processing of the wide-field image 40 is performed by the image processing device 12A. The wide-field image 40 and the narrow-field image 41 are sent to the image transmission element 13 of the image presentation device 3.

The image presentation device 3 includes a pair of image transmission elements 13.
And a wide-field monitor 24A, a narrow-field monitor 24a, a half mirror 26, and a half mirror 27. As shown in FIGS. 3 and 4, the image transmission element 13 has a concave lens 10
1, an optical fiber 102 and an eyepiece lens 103. The concave lens 101 is the light incident surface 1 of the optical fiber 102.
04, the eyepiece 103 is attached to the wearer 1
Reference numeral 4 is for use by mounting it on the eyeball 15. The optical fiber 102 has, for example, a diameter of 50 mm, and the diameter is selected according to the size of the eyes of the wearer 14 and the size of the face. Both the light incident surface 104 and the light emitting surface 105 of the optical fiber 102 are formed in a spherical concave shape.

The half mirror 27 is attached to the drive shaft of the scanner 29 and is rotatable about two axes. The scanner 29 is driven by the scanner drive circuit 30. As such a half mirror 27, for example, a galvanometer mirror can be used.

On the reflection side of the half mirror 26, the eyeball position measuring device 4 is arranged. The eyeball position measuring device 4 includes a lens 32, a split photodetector (APD), a photodetector 33 such as an infrared CCD camera, and an eyeball illuminating optical device 34. The eyeball illumination optical device 34 includes an infrared LED 35, a lens 36, and a half mirror 37. Photo detector 3
3 is connected to an eyeball position calculation circuit 39 via an amplifier circuit 38, and the output of the eyeball position calculation circuit 39 is input to the scanner controller 42, the scanner controller 63, and the image processing apparatus 12A.

The eyeball position calculation circuit 39 can be composed of, for example, a computer and software, calculates the position of the eyeball 15 based on the output signal of the photodetector 33, and the scanner controller 42, the scanner controller 63, the image processing device. Input to 12A. The scanner controller 42 can be configured by, for example, a computer and software, and the eyeball position calculation circuit 39
The drive amount of the scanner 29 is calculated by the signal from the control signal and the operation of the scanner drive circuit 30 is controlled.

The scanner controller 63 can be composed of, for example, a computer and software, calculates the drive amount of the scanner 61 based on a signal from the eyeball position calculation circuit 39, and controls the operation of the scanner drive circuit 62. In addition, in FIG. 2, the solid line indicates the electrical connection and the broken line indicates the optical axis.

The image presenting operation in the eye movement tracking type visual presenting apparatus 1 configured as described above is as follows.
The object 10 in the remote work environment is a pair of wide-view TVs.
The image is taken by the camera 5A, and the signal is the image processing device 12A.
Is displayed on the wide-field monitor 24A after being processed.
The content of the processing in the image processing device 12A includes, for example, processing such as erasing an image in the central portion of the screen where the narrow-field image is to be displayed or erasing noise. On the other hand, similarly, the object 10 is photographed by the pair of narrow-field TV cameras 5a, and the signal thereof is processed by the image processing device 12a and then displayed on the narrow-field monitor 24a. Image processing device 12a
The contents of the processing in 1) include processing such as image reduction or noise elimination.

The wide-field monitor 24A inputs an image into the optical path 44. In the optical path 44, the light from the wide-field monitor 24A passes through the half mirror 27, and then enters the eyeball 15 of the wearer 14 through the image transmission element 13. On the other hand, the image on the narrow-field monitor 24a is displayed from the optical path 45 to the half mirror 26.
Is transmitted, is reflected by the half mirror 27, and enters the optical path 44. In the optical path 44, the wide-field image 40 and the narrow-field image 4
1 becomes a composite image in which the narrow-field image 41 is fitted in the image cutout portion of the wide-field image 40.
This composite image is given to the eyeball 15 of the wearer 14 and forms an image on the retina of the eyeball 15. As a result, the gazing point image and the peripheral image of the point P on the object 10 are formed in front of the eyeball 15 of the wearer 14 by the composite image. At this time, the position of the eyeball 15, that is, the line of sight is measured by the eyeball position measuring device 4. In the eye position measuring device 4, infrared light is emitted from the infrared LED 35, passes through the lens 36, and the half mirror 37,
The light is reflected by the half mirror 26 and the half mirror 27, is input to the optical path 44, passes through the same optical path as the image, and illuminates the eyeball 15 through the image transmission element 13. The infrared light reflected from the eyeball 15 traces the same optical path in reverse, reaches the half mirror 37, passes through the half mirror 37, reaches the eyeball position measuring device 4, and enters the photodetector 33 through the lens 32. To do. In the photodetector 33, the output changes due to the change in the position of the eyeball 15, and the signal is input to the eyeball position calculation circuit 39 via the amplifier circuit 38, and the eyeball position calculation circuit 3
9 calculates the eyeball 15.

Next, when the wearer 14 moves his or her line of sight, that is, the eyeball 15, the image illuminated by the infrared light of the eyeball 15 is transmitted from the light emitting surface 105 to the image transmission element 13 by the bidirectionality of light. It enters, passes through and is emitted from the light incident surface 104, is reflected by the half mirrors 27 and 26, then passes through the half mirror 37, and enters the eyeball position measuring device 4. In the eyeball position measuring device 4, the image of the eyeball 15 is obtained by the lens 3
The image signal is detected by the photodetector 33 through 2, and the signal of the image is amplified by the amplifier circuit 38 and input to the eyeball position calculation circuit 39, where the movement amount and position of the eyeball 15 are detected. Then, the eyeball position signal is input to the imaging direction control device 6,
The imaging direction control device 6 is a TV for narrow field of view in the direction of the line of sight of the eyeball.
The imaging direction of the camera 5a is changed.

As a result, the line of sight of the wearer 14 and the line of sight of the narrow-field TV camera 5a coincide with each other. In this way, only the image information in the direction in which the wearer's 14 line of sight is directed is the narrow-view TV.
The image is taken by the camera 5a and displayed on the narrow-field monitor 24a.

On the other hand, when the eyeball 15 of the wearer 14 moves, the optical path 44 from the eyeball 15 to the monitor 24a changes, so that the eyeball 15 does not see the center of the monitor 24a. Arithmetic circuit 39
1/2 of the rotation angle θ of the eyeball 15 calculated in step 1, that is, (1 /
2) The half mirror 27 is rotated by θ to match the line of sight of the eyeball 15 with the center of the monitor 24a.

The half mirror 27 is controlled by the scanner drive circuit 30 driving the scanner 29 based on the eyeball position signal from the scanner controller 42. When the line of sight of the eyeball 15 coincides with the center of the monitor 24a in this manner, the images of the monitors 24A and 24a are made incident on the eyeball 15 of the wearer 14 and the TV camera 5 in front of the eyeball 5 is inserted.
Composite image information in the line-of-sight direction captured in A and 5a is formed. In this way, the device allows only the image in the direction of the line of sight to be incident on the eye even if the line of sight changes, and enables the construction of a system having an effect equivalent to that of FIG. 1b.

It should be noted that in the present description, a TV for narrow field of view in the direction of the line of sight.
The system for controlling the camera 5a and presenting the image on the narrow-field monitor 24a has been described. However, if the computer graphics image is controlled according to the eyeball position signal, a virtual reality system for displaying a virtual world is obtained. It is also possible.

(Other Embodiments) As the image transmission element, not only the one shown in FIG. 4 but also the one shown in FIGS. 5 and 6 can be used. The image transmission element 13 ′ shown in FIG. 5 does not use the optical fiber 102 and the eyepiece lens 103, but has a plurality of concave lenses 106 and 107 on the incident side and a convex lens 1 on the outgoing side.
08 and 109. The image transmission element 13 ″ shown in FIG. 6 does not use the eyepiece lens 103, but the optical fiber 111, the lens 112, and the convex lens 113 from the incident side.
It consists of.

[0027]

As described above, in the eye movement tracking type visual presentation device of the present invention, even if a wide-field image and a narrow-field image are combined and a monitor having the same number of scanning lines is used, the surrounding background is constituted. Since the wide-field image can be displayed at a low resolution and the narrow-field image of the gazing point can be displayed at a high resolution, a highly realistic image can be given to the wearer. Moreover, as an especially important point, in the eye movement tracking type visual presentation device of the present invention, since the image is incident from the front of the wearer through the image transmission element, the number of refracting the optical path of the image is reduced, and the optical path configuration is reduced. Since it becomes simple, the image is less likely to be distorted, and a good image can be provided. Since the line of sight of the TV camera changes following the line of sight of the wearer and only the image in the direction in which the line of sight of the wearer is facing can be given to the wearer, an image around the line of sight is presented with high resolution. As a result, it is possible to display a highly accurate image without making the frame of the screen conscious even on a relatively small screen, and the wearer can obtain a good visual artificial reality. In addition, it is possible to accurately measure the line of sight, facilitate the tracking of the line of sight, and have the effect of simplifying the configuration of the device because it is not necessary to match the human eyeball with the focus of the spheroid mirror. .

[Brief description of drawings]

1] Eyeball and object, eyeball and display, and TV
Explanatory drawing which shows the relationship between a camera and an object.

FIG. 2 is an explanatory diagram of a configuration of an eye movement tracking type visual presentation device.

FIG. 3 is a perspective view illustrating a schematic configuration of an image presentation device.

FIG. 4 is a vertical cross-sectional explanatory view of an image transmission element.

FIG. 5 is a vertical cross-sectional explanatory view of another image transmission element.

FIG. 6 is a vertical cross-sectional explanatory view of another image transmission element.

[Explanation of symbols]

 1 Eye movement tracking type visual presentation device 2 Drawing device 2a Imaging device 3 Image presentation device 4 Eyeball position measurement device 5 Camera device 5A Wide-field TV camera 5a Narrow-field TV camera 6 Imaging direction control device 10 Object 12A, 12a image Processing device 13, 13 ', 13 "Image transmission element 14 Wearer 15, 15' Eye 15a Right eye 15b Left eye 24A Wide-field monitor 24a Narrow-field monitor 26 Half mirror 27 Half mirror 29 Scanner 30 Scanner drive circuit 32 Lens 33 photodetector 34 optical device for eyeball illumination 35 infrared LED 36 lens 37 half mirror 38 amplification circuit 39 eye position calculation circuit 40 wide-field image 41 narrow-field image 42 scanner controller 44 optical path 45 optical path 50, 50 'line of sight 60 half mirror 61 Scanner 62 Sca Drive circuit 63 Scanner controller 100a Right eye device 100b Left eye device 101 Concave lens 102 Optical fiber 103 Eyepiece 104 Light incident surface 105 Light emission surface 106 Concave lens 107 Concave lens 108 Convex lens 109 Convex lens 111 Optical fiber 112 lens 113 Convex lens 122, 122 ' monitor

Claims (3)

(57) [Claims] 1. An image drawing device for creating an image corresponding to the line of sight of the wearer, an image presenting device for presenting the image created by the image drawing device to the wearer, and an eyeball position for measuring the position of the eyeball of the wearer. And a measuring device, the image presentation device,
A display device that displays an image created by the drawing device; an image transmission element that is disposed in front of the eyeball of the wearer and that allows the image displayed on the display device to enter, transmit, and output. A mirror having a variable attitude arranged in the optical path between the device and the image transmission element, and a mirror control device for controlling the attitude of the mirror, wherein the image is formed by a signal from the eye position measuring device. An eye movement tracking type visual presentation device according to a projection method, characterized in that the device is controlled and the mirror drive device is controlled. 2. The drawing device includes a wide viewing angle image drawing device for creating a wide viewing angle image and a narrow viewing angle image drawing device for creating a narrow viewing angle image having a narrower viewing angle than the wide viewing image, The eyeball according to the projection method according to claim 1, wherein the display device includes a wide viewing angle image display device that displays the wide viewing angle image and a narrow viewing angle image display device that displays the narrow viewing angle image. A motion-following visual presentation device. 3. The eye movement tracking type visual presentation according to claim 1, wherein the image transmission element includes a combination of a lens and an optical fiber, a combination of a plurality of lenses, or a combination of a lens and an optical fiber. apparatus.