JPH09211376A - Head mounted display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To more accurately display an electronic picture to a user by providing a gazing point detection means detecting the gazing point of the user within an observation range and outputting a gazing point detection signal. SOLUTION: After light emitted from a gazing point detecting light source 11 is turned into substantially parallel luminous fluxes by a light projecting lens 12, it is projected to the eyeball through a half mirror 13 and a gazing point detecting half mirror 14. Therefore, the reflected light from the eyeball is brught and condensed on the nearby position of gazing point detecting image sensors 16R and 16L by a light receiving lens 15 through the half mirrors 14 and 13. The gazing point is obtained by arithmetic calculation based on the gazing point detection signal (output signal) from the image sensors 16R and 16L by means of an analysis computer constituting a part of a control part.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a head mount display (hereinafter referred to as "HMD") device,
In particular, it relates to a see-through type HMD device.

[0002]

2. Description of the Related Art An image display means is fixed to a user's head,
An HMD device that allows the user to see a display image in the entire field of view has already been proposed. For example, in Japanese Unexamined Patent Publication No. 5-303053, a display image (hereinafter referred to as "electronic image") of an image display means is superimposed on a visual field image of the user's outside world (hereinafter referred to as "outside world image"). In addition, a see-through type HMD device is described which allows a user and a person other than the user to view the superimposed images at the same time. Further, Japanese Patent Laid-Open No. 6-242395 discloses an HMD device that detects the line of sight of the user and displaces a part of the optical system to change the image forming characteristics of the line of sight of the user. .

[0003]

However, the conventional HMD device cannot change the electronic image according to the line of sight of the user. Therefore, when the user moves his or her line of sight, necessary image information is obtained. There is a problem in that the user is out of the field of vision or the user cannot obtain information according to the line of sight of the user.

The present invention has been made in view of such conventional problems, and an object thereof is to provide a see-through type HMD device capable of displaying a more accurate electronic image to a user. I am trying.

[0005]

In order to achieve the above object, an HMD device according to the present invention is provided with an image display means for displaying an image, and an observation for guiding the image displayed on the image display means to a user's eye. Optical means, an image from the outside world is introduced into the optical path of the observing optical means, and an optical path combining means for enabling the image from the outside world to be observed in an overlapping manner with the image displayed on the image display means, A gaze point detecting means for detecting a gaze point within the observation range of the user and outputting a gaze point detection signal, an image display means, an observation optical means, an optical path synthesizing means and a gaze point detecting means are housed and used. A display unit having a housing configured to be fixed to a person's head, and a control unit controlling the image display means based on the gazing point detection signal are provided.

Preferably, the control unit generates an image information from an analysis computer which performs an operation based on the gazing point detection signal and an image information signal from the analysis computer, and causes the image display means to display the image. And a device.
Further, the gazing point detecting means is preferably an illuminating means for illuminating the eyeball of the user and an image information detecting means for detecting image information based on reflected light from the eyeball of the user and outputting a gazing point detection signal. It has and.

The HMD device according to the present invention further comprises an image forming lens capable of synthesizing the image from the image display means and the image from the outside, and an image pickup image sensor for picking up the image formed by the image forming lens. Since it is provided, the image obtained by the image sensor for imaging can be displayed to a person other than the user. Further, the HMD device according to the present invention is
Preferably, the distance information is obtained from the gazing point detecting means, and the distance information is displayed on the image display means. In this case, it is preferable that the sense of distance of the image displayed on the image display unit can be corrected by changing the position of the image displayed on the image display unit according to the distance information obtained by the gazing point detection unit.

[0008]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 2 shows the appearance of the display unit of the HMD device according to the present invention. As shown in FIG. 2, the display unit 21 of the HMD device includes a housing 22, and the optical elements and electric elements of an optical system described later are housed in the housing 22. A protective permeable membrane 23 is arranged on each of the outer peripheral surface and the inner peripheral surface of the housing 22, and speakers 24 for providing voice information to the user are arranged at both ends thereof. Also, the housing 22
Has a band 25 for fixing it to the user's head.
Is attached. Further, the display unit 21 has a cable 26 for inputting image information, audio information, and control information from a control unit 31 (FIG. 4), which will be described later, to optical elements and electric elements housed in the housing 22, and a cable 26. A cable 27 for outputting signals from those housed in the housing 22 to the control unit 31 is provided. In addition, the transmission and reception of signals between the control unit 31 and the display unit 21 are
It may be performed wirelessly.

FIG. 1 shows a first embodiment of an optical system housed in a housing 22. User's eyes 1R, 1L
In order to provide the image information of the electronic image to the left and right (two-dimensional) liquid crystal display devices 2R and 2L, and the light sources for the backlights of the liquid crystal display devices 2R and 2L, which are arranged on the left and right corresponding to both eyes which are image display means. 3 and a collimator lens 4 for converting the light from the light source 3 into a parallel light flux, a condenser lens 5 as an observing optical means, an optical path bending member 6, and an optical path synthesizing concave half mirror 7 as an optical path synthesizing means. ing. A metal mirror, a prism, or a field lens can be used as the optical path bending member 6. The image information input into the housing 22 as a signal is displayed on the liquid crystal display devices 2R and 2L, respectively, and the audio information is output via the speaker 24 (FIG. 2).

Further, the gazing point detecting means for detecting the gazing point of the user is a light source 1 for gazing point detection which is an illuminating means.
1, a light projecting lens 12, a half mirror 13, a gazing point detecting half mirror 14, a light receiving lens 15 which is an image information detecting means, and a gazing point detecting image sensor 16.
It is composed of R and 16L.

The operation of the above configuration will be described below.
In FIG. 1, the light emitted from the backlight light source 3 becomes a substantially parallel light flux at the collimator lens 4,
The liquid crystal display devices 2R and 2L are illuminated from behind. The light transmitted through the liquid crystal display devices 2R and 2L is condensed by the condenser lens 5 at a position near the optical path bending member 6 to form an image of the light source 3. Further, this light is reflected by the concave half mirror 7 for optical path synthesis, then passes through the half mirror 14 for gazing point detection, and forms a secondary image of the light source 3 at positions near the pupils of the eyes 1R, 1L. In FIG. 1, the optical path of the light emitted from the light source 3 is shown by a solid line.

The light from the image information displayed on the liquid crystal display devices 2R and 2L is converted into a substantially parallel light flux through the condenser lens 5, the optical path bending member 6 and the concave half mirror 7, and this light flux. Is a half mirror for gazing point detection
4, and forms an electronic image on the retina through the crystalline lenses (not shown) of the eyes 1R and 1L.

On the other hand, a part of the light entering the concave half mirror 7 for synthesizing the optical path from the outside world is concave half mirror 7.
Of the liquid crystal display devices 2R and 2L to be incident on the eyes 1R and 1L to form an external image. In FIG. 1, the optical path of light from the outside world between the outside world and the concave half mirror 7 is indicated by a broken line.

Therefore, the user can see the image in which the electronic image and the external image overlap each other. Also,
Since the image of the light source 3 is formed near the pupils of the eyes 1R and 1L, the light from the light source 3 can be used without waste, and thus,
You can get a bright image.

Next, the operation of the gazing point detecting means for detecting the gazing point of the user, that is, the line of sight will be described. In FIG. 1, the light emitted from the gazing point detection light source 11 is converted into a substantially parallel light flux by the light projecting lens 12, and then the half mirror 13 and the gazing point detection half mirror 1
It is projected to the eyeball through 4. As a result, the reflected light from the eyeball is brought, and the reflected light is received by the light-receiving lens 15 via the gazing point detecting half mirror 14 and the half mirror 13 and is positioned in the vicinity of the gazing point detecting image sensors 16R and 16L. Collected. The gazing point is the control unit 31.
Is calculated by an analysis computer 33 (FIG. 4) forming a part of the above, based on gazing point detection signals (hereinafter referred to as “output signals”) from gazing point detection image sensors 16R and 16L. To be

FIG. 3 shows a second embodiment of the optical system housed in the housing 22. The optical system of the second embodiment is different from the optical system of the first embodiment shown in FIG. 1 in that the light reflected by the gazing point detection half mirror 14 causes the images of the liquid crystal display devices 2R and 2L, that is, It further includes an imaging lens 17 that forms an electronic image and an external image, and imaging image sensors 18R and 18L that are arranged on the left and right corresponding to both eyes. Here, the light reflected by the gaze point detecting half mirror 14 enters the image sensors 18R and 18L for imaging through the imaging lens 17 to form an image. In addition, the images captured by the image sensors 18R and 18L for imaging are controlled by the control unit 3 via the cable 27 (FIG. 2).
The signal is output to 1. This allows people other than the user to
It becomes possible to observe and record a composite image of the external image and the electronic image similar to the image seen by the user.

Next, the control unit of the HMD device will be described with reference to FIG. FIG. 4 shows a system configuration example of the HMD device according to the present invention. In this configuration, the optical system of the second embodiment shown in FIG. 3 is applied as the display unit 21, and The control unit 31 includes an image generation device 32, an analysis computer 33, and an external input device 3 that enables appropriate input of information to be provided to the user.
4 and a monitor 35 for allowing a person other than the user to observe the composite image viewed by the user.

In the above configuration, the analysis computer 3
Reference numeral 3 denotes a gazing point by performing a calculation based on the output signals of the gazing point detecting image sensors 16R and 16L for each eye, and the gazing point thus obtained and the image sensors 18R and 18 for imaging.
The information is analyzed based on the image signal from L, an information signal corresponding to the user's gaze point, that is, the line of sight is generated, and the generated information signal is transmitted to the image generation device 32. Image generation device 3
A liquid crystal display device 2 in the display unit 21 generates image information to be provided to the user based on the information signal from the analysis computer 33, and generates an image signal corresponding to the generated image information.
Send to R, 2L. As a result, the image information corresponding to the line of sight of the user is displayed on the liquid crystal display devices 2R and 2L. It should be noted that signal transmission / reception is performed by wire or wirelessly, and although not shown, a voice signal is also transmitted to the display unit 21 together with the image signal so that voice information is provided to the user. .

Further, more specifically, an embodiment actually applied to a driver (user) of an automobile will be described with reference to FIG. In FIG. 5, the character "15m" is the HMD.
The display by the device is shown, and the scenery other than the character "15 m" is actually shown by the driver (user). The HMD device of the present embodiment displays the distance from the user to what the user is looking at. In the case shown in FIG. 5, the user is looking at the traffic light.

In the case of this embodiment, the line-of-sight directions of both eyes are obtained by the control unit as described above. From the obtained gaze direction, the distance to the gazing point is obtained using the triangulation method. This makes it possible to represent the distance from the user to the traffic light. In the conventional HMD devices, when the head of the user vibrates, the display of the HMD device also vibrates accordingly. However, in the HMD device according to the present embodiment, since the HMD device displays at a fixed position below the gazing point, even if the head of the user vibrates, the HMD device does not change as long as the gazing point does not change. The display can always be displayed in a fixed place with respect to the actual landscape.

The details will be described below. The gazing point is determined by the user looking at the traffic light. When the user's gazing point is determined, the analysis computer 33 of the control unit first detects the line-of-sight direction. Next, from the line-of-sight direction of both eyes,
Calculate the distance to the gazing point using the triangulation method.
The distance information to the gazing point obtained by the analysis computer 33 becomes an information signal and is transmitted to the image generation device 32. At this time, at the same time, the analysis computer 3
3 analyzes where the gazing point is, and the information signal regarding the fixed position between the gazing point position and the display position is also
It is transmitted to the image generation device 32. As a result, the distance to the gazing point is displayed at a predetermined position on the liquid crystal display devices 2R and 2L via the image generation device 32.

Thus, even if the user's head vibrates, the display position of the distance to the gazing point on the liquid crystal display devices 2R, 2L changes according to the vibration. Therefore, as long as the user keeps on looking at the traffic light, it is as if the display of the HMD device is present in the landscape that the user is seeing.

In this embodiment, the sense of distance to the display of the HMD device is corrected by subtly changing the positions of the images displayed on the liquid crystal display devices 2R and 2L.
If the display of the HMD device is to be exactly aligned with the position of the gazing point in the depth direction, the condenser lens 5 may be moved back and forth on the optical axis.

The present invention is not limited to the above-mentioned embodiments, and various modifications can be carried out without departing from the spirit of the present invention. For example, the optical path synthesizing concave half mirror 7 may be a combination of a half mirror having a plane parallel plate and a refraction lens, and the arrangement is limited to the horizontal direction as shown in FIG. Instead, they can be arranged vertically or in other directions.

[0025]

As described above, according to the present invention, it is possible to provide the user with an optimum electronic image according to the line of sight of the user. Further, by using the image pickup device, it becomes possible for a person other than the user to observe and record a composite image according to the line of sight of the user.

[Brief description of drawings]

FIG. 1 is a first optical system in an HMD device according to the present invention.
It is the schematic which shows an Example.

FIG. 2 is a schematic perspective view showing an appearance of a display unit in a use state of the HMD device according to the present invention.

FIG. 3 is a second optical system in the HMD device according to the present invention.
It is the schematic which shows an Example.

FIG. 4 is a block diagram showing a system configuration example of a display unit and a control unit of the HMD device according to the present invention.

FIG. 5 is a schematic diagram showing an example of a composite image displayed on the display unit of a driver (user) who is using the HMD device according to the present invention.

[Explanation of symbols] 2 Two-dimensional liquid crystal display device 3 Light source for backlight 7 Concave half mirror for optical path synthesis 11 Light source for gazing point detection Half mirror for gazing point detection 16 Image sensor for gazing point detection 18 Image sensor for imaging 21 Display Part 22 Housing 31 Control part

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical indication location G03B 35/18 G03B 3/02 H04N 5/64 511 (72) Inventor Gen Uchida Marunouchi, Chiyoda-ku, Tokyo 3 2-3 No. 3 In stock company Nikon

Claims (6)

[Claims] 1. An image display means for displaying an image, an observation optical means for guiding the image displayed on the image display means to a user's eye, and an image from the outside world in an optical path of the observation optical means. Introduced,
Optical path synthesizing means for enabling observation of an image from the outside on the image displayed on the image display means, and a gazing point detection signal for detecting a gazing point within the observation range of the user. And a housing configured to be fixed to the user's head, which houses the image display means, the observation optical means, the optical path synthesizing means, and the gazing point detection means. A head mount display device comprising: a display unit having the display unit; and a control unit that controls the image display unit based on the gazing point detection signal. 2. The control unit generates an image information from an analysis computer that performs an operation based on the gazing point detection signal and an image information signal from the analysis computer, and displays an image on the image display means. An image generating device for enabling the head mount display device according to claim 1. 3. The gazing point detection means detects illumination information for illuminating an eyeball of the user and image information based on reflected light from the eyeball of the user,
3. The head mount display device according to claim 1, further comprising image information detection means for outputting a gaze point detection signal. 4. An imaging lens comprising an imaging lens capable of synthesizing an image from the image display means and an image from the outside, and an imaging image sensor for imaging the image formed by the imaging lens. The head mount display device according to claim 1, wherein an image obtained by the image sensor for display is displayed to a person other than the user. 5. The head mount display device according to claim 1, wherein the distance information is obtained from the gazing point detecting means and the distance information is displayed on the image display means. 6. The sense of distance of the image displayed on the image display means can be corrected by changing the position of the image displayed on the image display means according to the distance information obtained by the gazing point detection means. The head mount display device according to any one of claims 1 to 5.