JP2014022937A - Head-mounted information input/output device and head-mounted information input/output method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head-mounted information input/output device and a head-mounted information input/output method that are capable of reducing power consumption.SOLUTION: A head-mounted information input/output device comprises: a display unit that displays an image; a holding unit that holds the display unit in the vicinity of a user's eye; a plurality of illumination units that are provided on the display unit, and illuminate the vicinity of the user's eye; and a control unit that controls lighting states of the plurality of illumination units in accordance with a positional relationship between the user and the display unit.

Description

  The present invention relates to a head-mounted information input / output device and a head-mounted information input / output method.

  In recent years, a technique for detecting a user's line of sight or a gazing point has been proposed. Such a technique may be used for a camera finder. In the line-of-sight detection device disclosed in Patent Literature 1, an illumination light source for detecting a photographer's eyeball image is arranged around an eyepiece lens in a camera finder. The light emitted from this illumination light source is reflected by the photographer's eyeball, and this reflected light is received by the image sensor arranged in the viewfinder, thereby detecting the photographer's eyeball image, that is, the line of sight. Is done.

JP-A-6-138375

  However, such an apparatus has a problem that the power consumption of the illumination light source for detecting the eyeball image of the photographer, that is, the line of sight is large.

  The present invention has been made in view of the above circumstances, and aims to provide a head-mounted information input / output device and a head-mounted information input / output method that can reduce power consumption. Yes.

  To achieve the above object, a head-mounted information input / output device according to an aspect of the present invention includes a display unit that displays an image, a holding unit that holds the display unit in the vicinity of a user's eye, and the display. A plurality of illumination units that illuminate the vicinity of the user's eyes, and a control unit that controls lighting states of the plurality of illumination units according to a positional relationship between the user and the display unit. It is characterized by providing.

  In order to achieve the above object, a head-mounted information input / output method according to an aspect of the present invention is a head-mounted information input / output method in a head-mounted information input / output device, which displays an image. A plurality of illumination units provided in the unit illuminate the vicinity of the user's eyes, and the control unit controls the lighting state of the plurality of illumination units based on the positional relationship between the user and the display unit And a control procedure to be performed.

  According to the present invention, the head-mounted information input / output device can reduce power consumption.

It is a perspective view of the head mounted display of a 1st embodiment. It is the perspective view which looked at the head mounted display concerning a 1st embodiment from the back side. It is a horizontal sectional view of the display main part concerning a 1st embodiment. It is operation | movement explanatory drawing of the headband which concerns on 1st Embodiment. It is a figure which shows the mounting | wearing form (1st display mode) of the head mounted display which concerns on 1st Embodiment. It is a perspective view which shows the stereo earphone attached to the head mounted display which concerns on 1st Embodiment. It is a figure explaining lighting and extinction of the eyeball illumination light concerning a 1st embodiment. It is sectional drawing of the display part for demonstrating the structure and optical system of a display part which concern on 1st Embodiment. It is a figure explaining the difference of the display area in each of the 1st-3rd observation state. It is a figure explaining the observation range and imaging range in a 1st observation state. It is a figure explaining an example of the captured image in the 1st observation state. It is a figure explaining an example of the captured image in the 2nd observation state. It is a figure explaining the observation range and imaging range in a 2nd observation state. It is a figure explaining an example of the captured image in the 3rd observation state. It is a figure explaining the observation range and imaging range in a 3rd observation state. It is a figure explaining the example which arrange | positions a display part in front of eyes and images the surrounding image containing eyes. It is a figure explaining the example which arrange | positions a display part to the lower side of the eye which concerns on 1st Embodiment, and images an eye from the bottom. It is a figure explaining the angle of view and selection range of the image sensor concerning a 1st embodiment. It is a figure explaining the imaging area and selection area concerning a 1st embodiment. It is a figure explaining the mechanism which performs selection of the selection range concerning a 1st embodiment mechanically. It is a functional block diagram of the head mounted display concerning a 1st embodiment. It is a functional block diagram of the process part which concerns on 1st Embodiment. It is a flowchart of the operation | movement performed after the power supply state of HMD which concerns on 1st Embodiment. It is a flowchart of the detection process procedure of the right and left eyes which concerns on 1st Embodiment. It is a figure explaining an example of the picked-up image of the imaging range S in the case of a right eye observation form. It is a figure explaining an example of the captured image of the imaging range S in the case of a left eye observation form. It is a figure explaining an example of the captured image in a left-eye observation form. It is a flowchart of the process sequence of the mounting guide of HMD which concerns on 1st Embodiment. It is a flowchart of the selection processing procedure of the HMD eyeball illumination light according to the first embodiment. It is a figure explaining the example of calibration as a comparative example with "Calibration for gaze point detection" concerning a 1st embodiment. It is a flowchart of the procedure of the calibration for the gaze point detection which concerns on 1st Embodiment. It is a flowchart of the procedure of the calibration for the gaze point detection which concerns on 1st Embodiment. It is a figure explaining lighting and extinguishing of the eyeball illumination light in the right eye observation mode according to the first embodiment. It is a figure explaining the relationship between the position of the display part which concerns on 1st Embodiment, and an eyeball illumination light, and is the figure which looked at the user's face from the horizontal front. It is an example of the password input screen which concerns on 1st Embodiment. It is a figure which shows an example of the menu screen which concerns on 1st Embodiment. It is an example of the screen which prompts the user who concerns on 1st Embodiment. It is a figure explaining the screen of the application of the eyes | visual_axis input which concerns on 1st Embodiment. It is a figure which shows the mounting form (2nd display mode, retreat mode) of the head mounted display which concerns on 2nd Embodiment. It is a figure explaining the relationship between the position of the display part which concerns on 2nd Embodiment, and an eyeball illumination light, and is the figure which looked at the user's face from the horizontal front. It is a flowchart of the selection processing procedure of the eyeball illumination light of HMD which concerns on 2nd Embodiment. It is a figure explaining arrangement | positioning of the eyeball illumination light which concerns on 3rd Embodiment. It is a figure explaining the relationship between the position of the display part which concerns on 3rd Embodiment, and an eyeball illumination light, and is the figure which looked at the user's face from the horizontal front. It is sectional drawing of the display part for demonstrating the optical system which concerns on 4th Embodiment. It is a figure explaining the observation range and imaging range which concern on 4th Embodiment.

The head mounted display (head-mounted information input / output device) according to the present embodiment includes a plurality of illumination units for illuminating the user's eyes in the vicinity of the display unit. The head mounted display controls the lighting state of the plurality of illumination units according to the positional relationship between the user and the display unit. That is, the head mounted display selects the illumination unit to be turned on according to the positional relationship between the user and the display unit. Accordingly, the head mounted display can detect the line of sight and the gazing point observed by the user regardless of whether the display unit is disposed on the lower side, the upper side, or the outer side with respect to the user's eyes.
Hereinafter, an example in which the information output apparatus is applied to a head mounted display will be described, but the present invention is not limited to this.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In the following description, an XYZ rectangular coordinate system is set as necessary, and the positional relationship of each part of the head mounted display will be described with reference to the XYZ rectangular coordinate system. A predetermined direction in the horizontal plane is defined as an X-axis direction, a direction orthogonal to the X-axis direction in the horizontal plane is defined as a Y-axis direction, and a direction orthogonal to each of the X-axis direction and the Y-axis direction (that is, a vertical direction) is defined as a Z-axis direction. Further, the rotation (inclination) directions around the X axis, Y axis, and Z axis are the θX, θY, and θZ directions, respectively.

[First Embodiment]
FIG. 1 is a perspective view of a head mounted display (head mounted device, hereinafter also referred to as HMD) 1 according to the present embodiment. FIG. 2 is a perspective view of the head mounted display 1 according to the present embodiment as viewed from the back side. FIG. 3 is a horizontal sectional view of the display main body 20 according to the present embodiment. FIG. 4 is an explanatory diagram of the operation of the headband according to the present embodiment. FIG. 5 is a diagram illustrating a mounting mode (first display mode) of the head mounted display according to the present embodiment.

The HMD 1 is a monocular head mounted display that includes a display main body 20 and a headband 40 that is attached to the user's head and supports the display main body 20. The HMD 1 of the present embodiment can be used with either eye as shown in FIG. FIG. 5A shows a state where the user is viewing the display unit 60 with the right eye (right eye), and FIG. 5B shows a state where the user is viewing with the left eye (left eye).
As shown in FIG. 2, the display main body 20 and the headband 40 are configured to be detachable via a connecting pin 41. 1 and 2, the longitudinal direction of the display main body 20 is defined as the Y-axis direction, and the direction in which the headband 40 sandwiches the user's head is defined as the X-axis direction.

Hereinafter, the configuration of each part of the HMD 1 will be described in detail.
The display main body 20 includes a device main body 21 that includes a main circuit and includes an operation unit and various interfaces, and a display unit 60 that is connected to the tip of the device main body 21.

  The apparatus main body 21 has a substantially plate-shaped casing 21A (see FIG. 3). In the present embodiment, an end portion (+ Y side end portion) on the side where the connection portion with the headband 40 is provided in the apparatus main body portion 21 is a base end portion, and an end portion on the opposite side of the base end portion ( The end on the -Y side) is the tip. Further, in a state where the apparatus main body 21 is attached to the headband 40, the headband 40 side (+ X side) of the apparatus main body 21 is the inside, and the opposite side (−X side) from the headband 40 is the outside.

As shown in FIG. 1, a main switch 28, a touch switch 34, and a sound collecting microphone 24 are arranged on the outer surface of the apparatus main body 21 along the longitudinal direction of the housing 21A.
The main switch 28 is a switch for turning on / off the power of the display body 20. The touch switch 34 is a touch panel that can perform various operations of the HMD 1 by touching the surface with a finger or the like. The sound collection microphone 24 is an external microphone that collects environmental sounds.

  As shown in FIG. 2, an ear speaker 23, an audio connector 26, and a headband hinge 32 having a connection hole 31 are provided on the base end side of the inner surface of the apparatus main body 21. A heart rate sensor 137 is provided at the center of the inner surface of the apparatus main body 21. A call microphone 37 is provided at the tip of the inner side surface of the apparatus main body 21.

  The ear speaker 23 is disposed in the vicinity of the user's ear. Audio information is transmitted from the ear speaker 23 to the user. The audio connector 26 is, for example, an audio input / output terminal to which the earphone shown in FIG. 6 is connected. The headband hinge 32 is a joint part with the headband 40. The voice of the user is input to the call microphone 37.

  The heart rate sensor 137 is a sensor that measures a user's heart rate by contacting the surface of the user's face. The heart rate sensor 137 includes a light emitting unit including a light emitting diode and the like, and a light receiving unit that detects light reflected inside the user's skin. The heart rate sensor 137 counts the heart rate by detecting a change in the amount of reflected light due to a change in blood flow. The heart rate sensor 137 is disposed near the user's eyes, but if the infrared light is emitted from the light emitting unit, the user will not feel glare.

A USB connector 25, an operation switch 30, and a video connector 27 are provided on the side end surface of the apparatus main body 21 on the base end side.
The USB connector 25 is a connection terminal of a USB (Universal Serial Bus) device. In this embodiment, for example, a remote controller (not shown) is connected.
The operation switch 30 is a pointing device such as a trackball or a stick. The operation switch 30 is provided to face the screen displayed on the display unit 60. As a result, the left-right direction of the operation on the operation switch 30 matches the left-right direction of the screen, so that the user can intuitively operate the operation switch 30 while looking at the screen.
The video connector 27 is a video input / output terminal.

  As shown in FIG. 3, the apparatus main body 21 incorporates a plate-like circuit board 29 extending along the longitudinal direction of the housing 21 </ b> A and a battery 33. A control circuit, a power supply circuit, and the like (not shown) are mounted on the circuit board 29, and are electrically connected to each part of the display body 20 via wiring (not shown).

  A display panel 36 made of a liquid crystal panel and a backlight 35 are arranged inside the touch switch 34 exposed on the outer surface of the apparatus main body 21. In the present embodiment, the display image on the display panel 36 is displayed through the touch switch 34. The display panel 36 and the backlight 35 may be an organic EL panel or an electrophoresis panel.

  The headband hinge 32 is a ball joint composed of a concavely curved housing portion 32a provided in the housing 21A and a spherical portion 32b fitted to the housing portion 32a. The spherical portion 32b has a spherical side surface portion and two plane portions formed in parallel to each other so as to sandwich the side surface portion. A connecting hole 31 is formed so as to penetrate the two flat portions vertically. The connection hole 31 is formed in a hexagonal shape when viewed in the axial direction. The display main body 20 and the headband 40 are connected by inserting the connection pin 41 of the headband 40 into the connection hole 31.

  By providing the headband hinge 32, the display main body 20 can be rotated in the A direction (around the X axis centering on the headband hinge 32) shown in FIG. In the present embodiment, the rotatable range of the display body 20 is about 270 °. By the rotation operation around the X axis, a switching function between a form in which an image is observed with the right eye shown in FIG. 5A and a form in which an image is observed with the left eye shown in FIG. 5B is realized.

  Further, since the headband hinge 32 is a ball joint, the display main body 20 can also be swung in the B direction (around the Z axis centering on the headband hinge 32) shown in FIG. By this swinging operation, the position of the display main body 20 relative to the user's eyes and ears can be adjusted.

  The heart rate sensor 137 in the vicinity of the headband hinge 32 is provided so as to protrude from the inner surface of the apparatus main body 21, and can be brought into contact with the surface of the user's face when the HMD 1 is worn. The battery 33 may be either a primary battery or a secondary battery.

As shown in FIGS. 1 and 2, the display unit 60 is connected to the distal end portion of the apparatus main body unit 21. In the present embodiment, the headband 40 side of the display unit 60 is the inside, and the side opposite to the headband 40 is the outside. The display unit 60 is an arm member that has a curved shape in a top view (Z-axis view), and has a shape that curves inward from the connecting portion with the apparatus main body 21 toward the distal end side. A finder opening 67 is provided on the inner surface of the display unit 60. A camera 64 is provided on the outer surface of the display unit 60. Moreover, as shown in FIG. 2, the shape of the finder opening 67 is a rectangle.
On the housing surface of the display unit 60, eyeball illumination lights 166 a to 166 d are provided in the vicinity of the finder opening 67. In FIG. 2, the eyeball illumination lights 166 a and 166 b are provided at positions along one of the long sides of the finder opening 67 and in the vicinity of the corner of the finder opening 67. The eyeball illumination lights 166 c and 166 d are provided along the other of the long sides of the finder opening 67 and near the corner of the finder opening 67.

  The eyeball illumination lights 166a to 166d are, for example, LEDs (light emitting diodes) that emit light in the infrared wavelength region. The eyeball illumination lights 166a to 166d are turned on or off according to the positional relationship between the user and the display unit 60. Hereinafter, the eyeball illumination lights 166a to 166d are collectively referred to as an eyeball illumination light 166.

  As shown in FIG. 3, the display unit 60 is connected to the apparatus main body unit 21 via a display hinge 61. The display hinge 61 is a ball joint composed of a concave curved housing 61a formed in the housing 60A of the display 60 and a spherical portion 61b formed in the apparatus main body 21 and fitted in the housing 61a. .

  The spherical portion 61b of the apparatus main body 21 has a normal direction of the inclined surface 21a (in FIG. 3) to an inclined surface 21a extending obliquely with respect to the longitudinal direction of the casing 21A formed at the outer surface tip of the apparatus main body 21. It is provided so as to protrude in the Y′-axis direction).

  The display unit 60 can be freely rotated around the Y ′ axis with respect to the spherical portion 61 b of the display hinge 61. In the present embodiment, the display hinge 61 is provided on the inclined surface 21a of the apparatus main body 21, and the display unit 60 has a shape curved inward.

  Further, since the display hinge 61 is a ball joint, the display unit 60 can swing in a direction orthogonal to the rotation plane around the Y ′ axis. This swinging operation facilitates position adjustment of the finder opening 67 by the user. If the interval between the display unit 60 and the apparatus main body unit 21 is increased, the swinging width of the display unit 60 can be increased.

  The spherical portion 61b of the display hinge 61 is formed with a through hole 61c that penetrates the spherical portion 61b in the height direction (Y′-axis direction). The inside of the display unit 60 and the inside of the apparatus main body unit 21 are communicated with each other through the through hole 61c. A cable (not shown) is inserted through the through hole 61c. The circuit board 29 and each part of the display unit 60 are electrically connected via the inserted cable.

  Inside the display unit 60, there are a backlight 62, a display panel 63, a camera 64, a prism 65, a reflecting mirror 66, a front light 68, a front speaker 70, an imaging lens 71, and an image sensor 72. (Imaging part) is provided. The configuration of the optical system of the display unit 60 will be described later.

  The camera 64 includes, for example, an image sensor with 5 to 10 million pixels, and is configured to be able to perform an autofocus operation. The camera 64 captures an image (optical image) formed on the light receiving surface of the image sensor. For example, the camera 64 can capture the front direction for the user. Note that the captured image may be signal-processed in either a moving image format or a still image format.

The front light 68 is, for example, an LED light. The front light 68 may include red, green, and blue light emitting elements, and may be configured to emit any color at any timing. The front light 68 may be used as a device that displays information to the outside depending on the light emission color and the light emission timing, or may be used as an illumination device when photographing with the camera 64.
The image sensor 72 captures an image (optical image) formed on the light receiving surface. The image sensor 72 can image a user's face, for example. Note that the captured image may be signal-processed in either a moving image format or a still image format.

  Although not shown in FIG. 3, the display unit 60 according to the present embodiment is provided with a laser transmitter. The laser transmitter can be provided in the vicinity of the front light 68, for example. By irradiating, for example, red laser light emitted from a laser transmitter forward, pointing (display of an index) with the laser light is possible.

Next, the headband 40 will be described with reference to FIGS.
As shown in FIG. 2, the headband 40 includes a pair of head pads (mounting members) 46 and 47 that sandwich the user's head, a first headband 43, a second headband 44, and a rotation mechanism 56. , 57.

  The first headband 43 is an elastic member having an arc shape as a whole. A joint portion 43 a for bending the first headband 43 is provided on the top of the first headband 43. At both ends of the first headband 43, bearing portions 43b and 43c constituting the rotation mechanisms 56 and 57 are provided, respectively. Bearing portions 43d and 43e to which the head pads 46 and 47 are connected are provided further on the front end side of the band than the bearing portions 43b and 43c.

  The second headband 44 is an elastic member having an arc shape as a whole. A joint portion 44 a for bending the second headband 44 is provided on the top of the second headband 44. At both ends of the second headband 44, shaft members 44b and 44c constituting rotation mechanisms 56 and 57 are provided.

  In the present embodiment, the second headband 44 has a configuration in which the surface of the spring member 48 made of a metal such as stainless steel is covered with a flexible material such as a resin. The spring member 48 of the second headband 44 generates a spring force that holds the user's head. The second headband 44 is formed wider than the first headband 43. The formation part of the joint part 44a is a mark display part 49 formed wider than the other band parts. A product tag or the like is attached to the mark display section 49 by sticker or printing. The entire second headband 44 may be made of metal.

  The head pad 46 includes a plate-like support plate 46a and an elastic member 46b having an arched cross section provided on one surface side of the support plate 46a. On the surface of the support plate 46a opposite to the elastic member 46b, a substantially hexagonal column-shaped connecting pin 41 is erected in a posture perpendicular to the surface. The connecting pin 41 is pivotally supported by a bearing portion 43 d provided at one end of the first headband 43. As a result, the head pad 46 can rotate around the connecting pin 41.

  The head pad 47 includes a plate-like support plate 47a and an elastic member 47b having an arcuate cross section provided on one surface of the support plate 47a. A shaft member 47c is provided on the surface of the support plate 47a opposite to the elastic member 47b. The shaft member 47 c is pivotally supported by a bearing portion 43 e provided at the tip of the first headband 43. Thereby, the head pad 47 can rotate around the shaft member 47c.

  With this configuration, the second headband 44 is urged to rotate in a direction away from the first headband 43 by the rotation mechanism 56, while being within a predetermined angle with respect to the first headband 43 by the rotation mechanism 57. The rotation is restricted to the range. As a result, the second headband 44 is held at a predetermined angle with respect to the first headband 43. Therefore, the rotation mechanisms 56 and 57 function as an angle holding mechanism in the headband 40.

  Next, the operation of the headband in the HMD 1 having the above configuration will be described with reference to FIGS. Hereinafter, the right side for the user is referred to as “the right side of the user”. Further, the left side for the user is referred to as “the left side of the user”. The upper side for the user is referred to as “the upper side of the user”. Further, the lower side for the user is referred to as “the lower side of the user”. The clockwise direction for the user is referred to as “clockwise direction”. Further, the counterclockwise direction for the user is referred to as “counterclockwise direction”.

  As shown in FIG. 5, the HMD 1 of the present embodiment can be used by switching between the right-eye observation form shown in FIG. 5A and the left-eye observation form shown in FIG. Here, the right-eye observation form is a form in which the user observes the display unit 60 with the right eye. The left-eye observation form is a form in which the user observes the display unit 60 with the left eye. Hereinafter, the form shown in FIGS. 5A and 5B, that is, the form in which the finder opening 67 is arranged horizontally when viewed from the user is referred to as a first display mode.

  For example, to switch from the right-eye observation mode to the left-eye observation mode, first, the display body 20 connected to the headband 40 is moved around the rotation axis of the headband hinge 32 (see FIG. 2) (A in FIG. 5). Direction). Further, in the headband 40, the position of the second headband 44 and the position of the first headband 43 are obtained by swinging the second headband 44 around the axes of the rotation mechanisms 56 and 57 (see FIG. 4). Swap the context. By this operation, as shown in FIG. 5B, the left-eye observation form in which the display main body 20 is arranged on the left eye side of the user and the second headband 44 is arranged on the occipital side of the user is performed. Can be switched.

  In the right-eye observation mode, the eyeball illumination lights 166a and 166b in the vicinity of the right eye are turned on, and the eyeball illumination lights 166c and 166d are turned off. On the other hand, in the left-eye observation mode, the eyeball illumination lights 166c and 166d in the vicinity of the left eye are turned on, and the eyeball illumination lights 166a and 166b are turned off.

In the present embodiment, as shown in FIG. 4, the arc height r <b> 1 of the first headband 43 is smaller than the arc height r <b> 2 of the second headband 44. As a result, even if the second headband 44 is moved so as to intersect the first headband 43, the front and rear can be switched smoothly without interfering with each other.
The arc height r1 shown in FIG. 4 is the maximum value of the distance from the center position C of the rotation axis L of the rotation mechanisms 56 and 57 arranged coaxially to the first headband 43. The arc height r2 is the maximum value of the distance from the center position C of the rotation axis L to the second headband 44.

  The arc height r1 of the first headband 43 may be larger than the arc height r2 of the second headband 44, and the second headband 44 may be swung inside the first headband 43.

FIG. 6 is a perspective view showing a stereo earphone attached to the head mounted display according to the present embodiment.
Stereo earphone 100 includes a connector 101, a cable 102, a first speaker 103, a second speaker 104, a sound collecting microphone 105, and a plurality of clips 106.

  The connector 101 is provided at one end of the cable 102. The connector 101 is a general 4-pole φ3.5 mm mini plug. The breakdown of the four poles is the sound collection microphone 105, the first speaker 103, the second speaker 104, and the ground (GND). The cable 102 is bifurcated in the vicinity of the connector 101, and a first speaker 103 is provided at the end of the branched cable. A second speaker 104 and a sound collecting microphone 105 are provided at the other end of the cable 102. The plurality of clips 106 are arranged on the cable 102 at a predetermined interval.

  The stereo earphone 100 is used by connecting the connector 101 to the audio connector 26 of the display body 20. When the stereo earphone 100 is connected, the ear speaker 23 of the display body 20 and the call microphone 37 of the display unit 60 are invalidated. Further, the front speaker 70 of the display unit 60 is also invalidated as necessary. Then, the first speaker 103, the second speaker 104, and the sound collecting microphone 105 of the stereo earphone 100 are activated.

  The first speaker 103 of the stereo earphone 100 is attached to the ear of the user on the side where the display main body 20 is arranged, and the second speaker 104 is attached to the ear on the opposite side to the first speaker 103. At this time, the cable 102 can be fixed to the second headband 44 by the clip 106.

Further, stereo recording can be performed by the sound collecting microphone 105 of the stereo earphone 100 and the sound collecting microphone 24 provided on the outer surface of the housing 21 </ b> A in the display main body 20. For example, if the display main body 20 is arranged on the right eye side as shown in FIG. 5A, the sound collecting microphone 24 of the display main body 20 collects sounds on the right side of the user and is attached to the left ear. The sound collecting microphone 105 collects the sound on the left side of the user.
In the right-eye observation mode, the right channel sound is output from the first speaker 103 of the stereo earphone 100, and the left channel sound is output from the second speaker 104.

  On the other hand, when the display main body 20 is arranged on the left eye side of the user as shown in FIG. 5B, the sound collecting microphone 24 of the display main body 20 collects the sound of the user's left side and The attached sound collecting microphone 105 collects sound on the right side of the user. In the left-eye observation mode, left channel sound is output from the first speaker 103 of the stereo earphone 100, and right channel sound is output from the second speaker 104.

Next, lighting and extinguishing of the eyeball illumination lights 166a to 166d will be described.
FIG. 7 is a diagram for explaining turning on and off of the eyeball illumination light 166 according to the present embodiment. FIG. 7A is a diagram for explaining turning on and off of the eyeball illumination light 166 in the right-eye observation mode. FIG. 7B is a diagram for explaining turning on and off of the eyeball illumination light 166 in the left-eye observation mode. In FIG. 7A and FIG. 7B, the front direction of the user's face is denoted as HZ.

  In the example shown in FIG. 7A, the display unit 60 is disposed obliquely below the right eye. Further, the front direction HZ and the normal line of the screen of the display unit 60 form an angle α. The HMD 1 selects an eyeball illumination light on the side close to the user's eye among the eyeball illumination lights 166a to 166d, and turns on the selected eyeball illumination light. On the other hand, the HMD 1 turns off the eyeball illumination lights other than the selected one, that is, the eyeball illumination lights far from the user's eyes.

  In FIG. 7A, the eyeball illumination lights 166a and 166b are disposed below the front direction HZ by an angle α1. On the other hand, the eyeball illumination lights 166c and 166d are arranged at an angle α2 below the front direction HZ. Here, the angle α1 is smaller than the angle α, and the angle α2 is larger than the angle α. That is, the eyeball illumination lights 166a and 166b are disposed closer to the user's right eye than the eyeball illumination lights 166c and 166d.

  In this case, the HMD 1 turns on the eyeball illumination lights 166a and 166b and turns off the eyeball illumination lights 166c and 166d. In this lighting state, the image sensor 72 images around the right eye of the user. Hereinafter, an image captured around the right eye of the user is referred to as a “right eye peripheral image”.

  The peripheral image of the right eye may include an image of the right eye, an image of the right eyebrow, and an image of a part of the face (head, hair, etc.). The peripheral image of the left eye may include a left eye image, a left eyebrow image, and a part of the face (head, hair, etc.). Hereinafter, the peripheral image of the right eye and the peripheral image of the left eye are collectively referred to as “eye peripheral image”.

  In the example shown in FIG. 7A, the eyeball illumination lights 166c and 166d far from the user's right eye may not be able to illuminate the user's right eye due to the user's cheek getting in the way. Therefore, the HMD 1 turns off the eyeball illumination lights 166c and 166d far from the user's right eye. Thereby, HMD1 can reduce power consumption.

  On the other hand, in FIG. 7B, the eyeball illumination lights 166c and 166d are arranged on the lower side by an angle α3 from the front direction HZ. On the other hand, the eyeball illumination lights 166c and 166d are disposed at an angle α4 below the front direction HZ. Here, the angle α3 is smaller than the angle α, and the angle α4 is larger than the angle α. That is, the eyeball illumination lights 166a and 166b are disposed closer to the user's left eye than the eyeball illumination lights 166c and 166d.

  In this case, the HMD 1 turns on the eyeball illumination lights 166c and 166d and turns off the eyeball illumination lights 166a and 166b. In this lighting state, the image sensor 72 images around the left eye of the user. Hereinafter, an image in which the periphery of the user's left eye is captured is referred to as a “left eye peripheral image”.

  In the example shown in FIG. 7B, the eyeball illumination lights 166a and 166b far from the user's left eye may not be able to illuminate the user's left eye due to the user's cheek getting in the way. Accordingly, the HMD 1 turns off the eyeball illumination lights 166a and 166b on the side far from the user's left eye. Thereby, HMD1 can reduce power consumption.

  In other words, the HMD 1 turns on an eyeball illumination light that can illuminate the user's eyes from the front, among the eyeball illumination lights 166a to 166d, according to the positional relationship between the user's eyes and the display unit 60. Turn off the eyeball illumination light.

Next, the configuration of the display unit 60 and the optical system will be described.
FIG. 8 is a cross-sectional view of the display unit 60 for explaining the configuration and optical system of the display unit 60 according to the present embodiment. As shown in FIG. 8, the display unit 60 of the present embodiment includes a backlight 62, a display panel 63, a first prism 65a, a second prism 65b, a reflection mirror 66, a viewfinder opening 67, an imaging lens 71, and an image sensor. 72 and a quarter wavelength plate 75. In FIG. 8, symbols H, I, J, K, L, M, and N each represent a light beam.

First, the configuration of the display unit 60 will be described.
The prism 65 has a configuration in which a substantially triangular first prism 65a and a second prism 65b are bonded to each other on a top view (Z-axis view). A display panel 63 made of a liquid crystal panel is provided at a position facing one of the other two surfaces of the first prism 65a other than the bonding surface. A backlight 62 that transmits and illuminates the display panel 63 is disposed on the back surface of the display panel 63. A reflection mirror 66 is disposed at a position facing the other surface of the first prism 65a. The reflection mirror 66 is located substantially in front of the finder opening 67.

  Of the two surfaces other than the bonding surface of the second prism 65 b, one surface is a viewfinder eyepiece disposed in the viewfinder opening 67. An imaging element 72 is disposed opposite to the other surface of the second prism 65b via the imaging lens 71.

Next, the optical system of the display unit 60 will be described.
In the display unit 60, the image displayed on the display panel 63 is emitted from the finder opening 67 through the first prism 65a, the reflection mirror 66, and the second prism 65b, and is observed by the user. Further, the image looking at the viewfinder opening 67 is imaged on the image sensor 72 via the second prism 65 b and the imaging lens 71.
The captured image acquired via the image sensor 72 is used for analysis of the direction of the user's line of sight, blinking, facial expression, and posture of the display unit 60. The captured image includes a peripheral image of the eye. Note that the backlight 62 may also be used as auxiliary illumination when an image around the eye is captured by the image sensor 72.

  The light emitted from the backlight 62 passes through the display panel 63 and enters the first prism 65a. At this time, due to the characteristics of the liquid crystal of the display panel 63, the light irradiated by the backlight 62 is polarized, and the S-polarized light of the polarized light beam H is incident on the first prism 65a.

  The light beam H is totally reflected by the influence of the air layer provided on the back surface 65d of the first prism 65a, and enters the joint surface 65c of the first prism 65a and the second prism 65b as the light beam I. Here, a polarizing reflection film that reflects S-polarized light and transmits P-polarized light is deposited on the joint surface 65c of the two prisms. Here, since the light ray H is S-polarized light, it is reflected by the joint surface 65c to become a light ray J. Since the light beam J is incident on the back surface 65d at an incident angle of 0 degree, it passes through the back surface 65d and enters the quarter-wave plate 75.

The light beam J passes through the quarter-wave plate 75 after passing through the air layer, and becomes circularly polarized light clockwise with respect to the traveling direction. The light beam K reflected by the reflecting mirror 66 becomes counterclockwise circularly polarized light by reflection. Then, the light K passes through the quarter-wave plate 75 again and becomes P-polarized light.
Since the light K has been converted to P-polarized light, it can pass through the joint surface 65c, and thus can pass through the finder opening 67. With such a configuration, the user can observe the image data displayed on the display panel 63 through the viewfinder opening 67.

  On the other hand, the light beam L incident from the finder opening 67 is reflected by the joint surface 65c and converted into an S-polarized light beam M. The light ray M is totally reflected by the air layer on the front surface 65e of the second prism 65b to become a light ray N. The light beam N is collected by the imaging lens 71, and the collected light beam N enters the image sensor 72. With such a configuration, when the finder opening 67 is disposed in front of the user's eye, a peripheral image (optical image) of the user's eye is formed on the image sensor 72.

Next, an image observed by the user through the finder opening 67 and an image captured by the image sensor 72 in the right-eye observation mode will be described.
In FIGS. 10, 13, and 15, the reflecting mirror 66 (see FIG. 8) is replaced with a convex lens so as to be optically equivalent. 10, 13, and 15, the alternate long and short dash line CT represents the optical axis of the imaging lens 71.

Hereinafter, a state in which the display unit 60 is disposed within a predetermined distance from the user's eyes is referred to as a “first observation state”.
FIG. 9 is a diagram for explaining a difference in display area in each of the first to third observation states. In the first observation state, since the display unit 60 is disposed within a predetermined distance from the user's eyes, the entire display area of the display panel 63 can be observed by the user. Therefore, the display panel 63 can display an image in the entire display area. That is, the entire display area of the display panel 63 becomes the display area S.

  Hereinafter, the horizontal length of the display area S is expressed as Sw. Also. The vertical length of the display area S is expressed as Sh. In the following description, as an example, the description will be continued assuming that Sw is 800 pixels and Sh is 600 pixels. In addition, the vertical length of the display area T is expressed as Th. Further, the horizontal length of the display area T is denoted as Tw. Further, the vertical length of the display area U is denoted as Uh. Further, the horizontal length of the display area U is denoted as Uw.

  FIG. 10 is a diagram illustrating the observation range and the imaging range in the first observation state. When the user's eyes are in the observation range 67 a, the user can observe the display area S corresponding to the entire display area of the display panel 63 through the finder opening 67. The observation range 67a is in the range from the finder opening 67 to the distance P on the optical axis of the reflection mirror 66 (convex lens in FIG. 10). On the other hand, the imaging range 67c that can be imaged by the imaging device 72 includes an observation range 67a. That is, even when the user's eyes are not in the observation range 67a, the imaging element 72 can capture the user's eyes including eyebrows if the user has the eyes in the shooting range 67c.

  FIG. 11 is a diagram illustrating an example of a captured image in the first observation state. In the approximate center of FIG. 11, the eye of the user is imaged by the imaging element 72. In this case, as described with reference to FIG. 9A, the eyeball illumination lights 166a and 166b are turned on. In the case of the right eye observation mode, the + Z side in FIG. 11 is the vertically upward direction, the −Z side is the vertically downward direction, the + X side is the left side of the user, and the −X side is the right side of the user. In the case of the left-eye observation form, the display unit 60 is inverted 180 degrees, so that the peripheral image of the eye is inverted horizontally and vertically as compared with the right-eye observation form.

  In FIG. 11, reference numeral 601 </ b> R indicates a right eye image, and reference numeral 602 </ b> R indicates a right eyebrow image. Reference numeral 604A represents a bright spot image by the eyeball illumination light 166a (see FIG. 2), and reference numeral 604B represents a bright spot image by the eyeball illumination light 166b (see FIG. 2). Reference sign L41 is a sign indicating the width of the user's eyes in the X-axis direction. For example, when the resolution of the image sensor 72 is 640 pixels in the horizontal direction (X-axis direction) in the first observation state, the eye width L41 in the image is about 350 pixels.

In FIG. 11, since the distance between the user's eyes and eyebrows is equal to or smaller than the width L41, it can be said that an appropriate range is captured in the captured image. Thus, it is effective to photograph the user's eyes from below.
Thus, in the first observation state, the finder opening 67 does not get in the way when the user looks at the front. As will be described later, in the first observation state, the HMD 1 can detect the user's line-of-sight direction based on the movements of the eyebrows, the eyes, and the eyeballs captured in the image captured by the image sensor 72.

Hereinafter, a state in which the user's eyes are separated from the display unit 60 as compared with the first observation state is referred to as a “second observation state”.
Next, a 2nd observation state is demonstrated using FIG.9, FIG12 and FIG.13.

FIG. 12 is a diagram illustrating an example of a captured image in the second observation state. Reference numeral 601R1 represents a right eye image, and reference numeral 602R1 represents a right eyebrow image. Reference numeral 604C represents a bright spot image by the eyeball illumination light 166a, and reference numeral 604D represents a bright spot image by the eyeball illumination light 166b. Reference sign L71 is a sign indicating the width of the user's eyes in the X-axis direction.
In the second observation state, since the user's eyes are separated from the display unit 60, an image of the user's eyes is captured smaller than in the first observation state. When the width L71 of the user's eye in the captured image is equal to or smaller than the width L41 of the user's eye in the first observation state (for example, 300 pixels or less), the HMD 1 determines that the user is taking his eyes off the display unit 60. To do.

  FIG. 13 is a diagram illustrating an observation range and an imaging range in the second observation state. In FIG. 13, the user's eye is located on the optical axis of the reflection mirror 66 (convex lens in FIG. 13) at a position within the distance P2 from the finder opening 67 and within the distance P2, and the observation range 67b in the first observation state. There is no distance (within the distance P from the finder opening 67 on the optical axis). For this reason, the user can observe only a part of the display area S (see FIG. 9).

  Therefore, in the second observation state, the HMD 1 displays an image reduced at the reduction rate η in the display area T (see FIG. 9). In the second observation state, the reduction rate η (ratio with the first observation state) of the image displayed in the display region T is expressed by the following equation (1).

  η = L71 / L41 (1)

  For example, when the user's eye width L41 (see FIG. 11) is 350 pixels and L71 (see FIG. 12) is 220 pixels, HMD1 is about 0.63 times (= (L71 / L41) = (Th / Sh) = (Tw / Sw)), an image reduced in length and width is displayed in the display area T of the display panel 63. The entire image (optical image) displayed in the display area T reaches the user's eyes without being blocked by the edge of the finder opening 67. In the second observation state, the HMD 1 may increase the luminance of the eyeball illumination light 166 compared to the luminance in the first observation state.

Hereinafter, a state in which the relative position between the display unit 60 and the user's eye is shifted in either the left or right direction compared to the second observation state is referred to as a “third observation state”.
Next, a 3rd observation state is demonstrated using FIG.9, FIG14 and FIG.15.
FIG. 14 is a diagram illustrating an example of a captured image in the third observation state. In the third observation state, it is assumed that, as an example, an image of the user's eye is captured with a vector V1 shifted from the vicinity of the center of the image in the lower right direction as compared with the second observation state.

  Reference numeral 601R2 represents a right eye image, and reference numeral 602R2 represents a right eyebrow image. Reference numeral 604E represents a bright spot image by the eyeball illumination light 166a, and reference numeral 604F represents a bright spot image by the eyeball illumination light 166b. Also, 605A indicated by a one-dot chain line represents an outline image of an eyeball that is captured when the position of the user's eye is not displaced. Reference numeral 605B represents an image of the eyeball when the user's eye is at the center of the optical axis.

  FIG. 15 is a diagram illustrating an observation range and an imaging range in the third observation state. In FIG. 15, the user's eye eye is at a position within the observation range 67e, that is, at a distance P or more from the finder opening 67 and shifted inward from the optical axis CT by the X component amount (= V1x) of V1. exist. Since the user's eye in the observation range 67e is not in the observation range 67b (within the distance P from the finder opening 67 on the optical axis) in the first observation state, one of the display areas S (see FIG. 9). Only the part can be observed.

  Therefore, in the third observation state, the HMD 1 reduces the image at a predetermined reduction rate (= (Uh / Sh) = (Uw / Sw)) so that the entire display image can be observed from the user. Further, the HMD 1 displays the reduced image in the display area U (see FIG. 9). Here, the HMD 1 determines the position of the display area U in the display area S (see FIG. 9) according to the positional deviation amount and direction of the user's eyes. In the third observation state, the display area U is defined at the upper right of the display area S.

  Here, it is assumed that the number of recording pixels of the imaging system including the imaging lens 71 and the imaging element 72 is, for example, 640 pixels wide and 480 pixels vertically. Further, it is assumed that the angle of view of the imaging system including the imaging lens 71 and the imaging element 72 is 60 degrees lateral and 45 degrees longitudinal. In this case, the angle of view per pixel is about 0.094 degrees.

  Therefore, if the amount of horizontal displacement of the user's eye position (X component amount of the vector V1) is, for example, 30 pixels, the user's eye position is 4.7 degrees (= 30 pixels × 0.094). Degree). If the viewing angle in the horizontal direction (Sw: 800 pixels) of the display area S (see FIG. 9) is, for example, 35 degrees, the HMD 1 is 108 pixels (= Sw × 4. The position of the display area U is determined to the right in the horizontal direction by 7 degrees / 35 degrees. Similarly, in the vertical direction, the HMD 1 defines the display area U at a position shifted by a predetermined number of pixels.

  14 and 15, the HMD 1 reduces the image until the horizontal direction Uw of the display area U is 500 pixels and the vertical direction Uh is 375 pixels, 108 pixels to the right in the horizontal direction from the center of the display area S, and A display area U is defined at a position shifted vertically by the predetermined number of pixels corresponding to the z component amount (= V1z) of the vector V1. In the display area U, an image reduced at the same reduction ratio as the display area U is displayed. Thereby, even when the user's eye eye is shifted from the optical axis, the HMD 1 can display the image so that the entire image can be seen.

  The HMD 1 extracts, for example, an eye contour image from the captured image, and detects an eye displacement amount and direction (vector V1) based on the extracted position of the eye contour image. Here, the detection of the eye position deviation amount and direction may be performed by extracting an eyeball image from the captured image and detecting the eye position deviation amount and direction based on the extracted deviation amount of the outline image of the eyeball. Good. Further, the HMD 1 detects the vector V1 by appropriately turning on the eyeball illumination lights LEDs 166a to 166d (see FIG. 2) and measuring spots on the eyeball of the user projected by the eyeball illumination lights. You may do it.

  As described above, even when the user's eyes are separated from the display unit 60 or deviated from the optical axis, the HMD 1 allows the user to select the entire area of the image according to the positional relationship between the user and the display unit 60. An image can be displayed on the display unit 60 so that the image can be observed. Note that the timing for determining the display area may be before or after calibration processing (described later) for detecting a point of interest.

Next, arrangement of the display unit 60 and imaging of the user's eyes will be described.
FIG. 16 is a diagram illustrating an example in which the display unit 60 is arranged in front of the eyes and a peripheral image including the eyes is captured. FIG. 17 is a diagram illustrating an example in which the display unit 60 is arranged on the lower side of the eye according to the present embodiment and the eye is imaged from below.

  In FIG.16 and FIG.17, the code | symbol L11 represents the distance from a user's lower eyelid to eyebrows. Further, in FIGS. 16 and 17, the distance L11 is 40 [mm]. 16 and 17, when the user's face is viewed from the side, the vertical direction is referred to as the Z-axis direction, and the front-back direction is referred to as the Y-axis direction.

  As shown in FIG. 16, when the display unit 60 is arranged in front of the eyes, in order to image from the user's lower eyelid to eyebrows, the display unit 60 is captured until a width L11 or more (for example, width L31) can be imaged. It is necessary to keep the imaging lens 71 away from the user's face. Here, the width L31 is, for example, 70 [mm]. When the range of the width L31 is imaged, the distance L21 needs to be separated from the user's eye to the center of the imaging lens 71. Here, the distance L21 is, for example, 85 [mm].

  The distance L21 also includes the optical path lengths of the first prism 65a, the second prism 65b, etc. in the optical system shown in FIG. Assuming that the total optical path length of the first prism 65a, the second prism 65b, etc. is 20 [mm], the display unit 60 is only 65 (= 85-20) [mm] in the Y-axis direction from the user's face. It is necessary to keep away.

  Therefore, when the display unit 60 is disposed in front of the eyes, the display unit 60 needs to be disposed at a position away from the user's face in order to capture an image from the user's lower eyelid to the eyebrows.

  On the other hand, as shown in FIG. 17, when the display unit 60 is arranged on the lower side of the eye, in order to image the range of the distance L11 from the user's lower eyelid to the eyebrows, the imaging range of the width L61 is imaged. There is a need. Here, since the image is picked up from below, the width L61 is shorter than the width L31 shown in FIG. For example, the width L61 is 20 [mm]. An imaging distance (distance L51) necessary for securing the imaging range 20 [mm] is, for example, 45 [mm].

  Similarly to FIG. 16, when the total optical path length of the first prism 65a, the second prism 65b, etc. is 20 [mm], the display unit 60 is moved away from the user's face by 25 [mm]. When the display unit 60 is inclined by an angle β with respect to the Y-axis direction, the distance in the Y-axis direction from the face to the imaging lens 71 is a value obtained by multiplying cos β by 45 [mm].

  In the HMD 1, the display unit 60 arranged under the eyes can capture a peripheral image of the eye (see FIG. 17), so that the HMD 1 is compared with the case where the display unit 60 is arranged in front of the eyes (see FIG. 16). See), and the distance from the face to the display unit 60 is short. Thereby, HMD1 does not enlarge. Moreover, since the display part 60 is not arrange | positioned in front of a user, HMD1 does not block the front view of a user. Furthermore, since the display part 60 can also be arrange | positioned in the vicinity of a user's face, the display part 60 does not become obstructive, for example in a narrow work space.

  In addition, the HMD 1 lowers the luminance of the eyeball illumination light 166 when the display unit 60 is disposed under the eyes as compared with the case where the display unit 60 is disposed in front of the eyes (see FIG. 16). May be. Thereby, HMD1 can reduce power consumption.

Next, adjustment of the installation angle of the image sensor 72 that accompanies switching between the right eye observation mode and the left eye observation mode will be described.
FIG. 18 is a diagram illustrating the angle of view and the selection range of the image sensor 72 according to the present embodiment. FIG. 18A is a diagram for explaining the angle of view and the selection range of the image sensor 72 in the right-eye observation mode. FIG. 18B is a diagram for explaining the angle of view and the selection range of the image sensor 72 in the left-eye observation mode. In FIGS. 18A and 18B, the first headband 43, the second headband 44, the apparatus main body 21 and the like are not shown.

  FIG. 19 is a diagram illustrating an imaging area and a selection area according to the present embodiment. FIG. 19A is a diagram illustrating an imaging region by the imaging element 72 (see FIG. 3) when the headband hinge 32 (see FIG. 2) is swung. FIG. 19B is a diagram for explaining a selection area selected from the imaging area. Here, the symbol P is a symbol indicating a range in which the image sensor 72 can capture an image. The image sensor 72 has, for example, an aspect ratio of 4 to 3.

  In the case of the right-eye observation mode, in the first display mode (see FIG. 5A), the image sensor 72 is provided in the display unit 60 so that the aspect of the imaging region is vertically long. In this case, as shown in FIGS. 18 (A) and 19 (B), the image sensor 72 is a region indicated by a symbol Q among regions indicated by a symbol P (hereinafter referred to as “imaging region P”). The image data of only the range to be imaged is taken in (hereinafter referred to as “selected area Q”). Thereby, the image sensor 72 can appropriately capture the image data of the right eye including the eyebrows in the right eye observation mode.

  On the other hand, in the case of the left-eye observation mode, in the first display mode (see FIG. 5B), the image sensor 72 is provided in the display unit 60 so that the aspect of the imaging region is vertically long. In this case, as shown in FIGS. 18B and 19B, the imaging element 72 is imaged in an area indicated by a symbol R (hereinafter referred to as “selection area R”) in the imaging area P. Image data of only the range. Thereby, the image sensor 72 can appropriately capture the image data of the left eye including the eyebrows in the left-eye observation mode.

  Further, the HMD 1 appropriately determines whether it is the right-eye observation form or the left-eye observation form based on an image captured by the image sensor 72 instead of a detection value (described later) indicating the acceleration detected by the acceleration sensor. can do.

  FIG. 20 is a diagram illustrating a mechanism for mechanically selecting a selection range according to the present embodiment. A rotation shaft 72 b is fixed to the main body 72 a of the image sensor 72. A dial 72c is provided for the rotating shaft 72b. The user can tilt the image sensor 72 about the rotation shaft 72b by turning the dial 72c within the rotatable range (in the D direction in FIG. 20).

  When the imaging lens 71 is tilted in the upward direction, the selection area Q (see FIG. 19B) is determined as an upper area in the imaging area P. Thereby, the image sensor 72 can image the user's eyes and eyebrows in the imaging region Q. On the other hand, when tilted in the direction toward the bottom of the imaging lens 71, the selection region R (see FIG. 19B) is defined as a lower region of the imaging region P. Thereby, the image sensor 72 can image the user's eyes and eyebrows in the imaging region R.

  The HMD 1 can also finely adjust the position of the imaging area Q or the selection area R with the dial 72c. When a captured image by the image sensor 72 is displayed on the display unit 60, the user observes the captured image displayed on the display unit 60 and determines the image sensor according to the positional relationship between the display unit 60 and the eyes. 72 imaging areas can be selected. As a result, the HMD 1 can increase the accuracy of calibration for HMD 1 alignment (described later) and gaze point detection using a captured image that is captured so that the range is appropriate.

Next, functional blocks of the HMD 1 will be described.
FIG. 21 is a functional block diagram of the head mounted display 1 according to the present embodiment.
The HMD 1 includes various electric circuits centering on the processing unit 123. The processing unit 123 is a CPU (Central Processing Unit) and is connected to various circuits of the HMD 1 and comprehensively controls the HMD 1.

The battery 33 supplies power to the power supply circuit 120.
The power supply circuit 120 supplies the power supplied from the battery 33 to each unit of the HMD 1 under the control of the processing unit 123.

  In the present embodiment, the processing unit 123 includes an encoder 129 and a decoder 121, a power supply circuit 120, an operation switch 30, a flash memory 122, a BL driver 126, a BT communication circuit 130, a WiFi communication circuit 131, an acceleration sensor 132, and a geomagnetic sensor. 133, front light 68, 3G / LTE communication circuit 138, laser transmitter 73, angular velocity sensor 134, GPS sensor 135, temperature / humidity sensor 136, heart rate sensor 137, memory 127, main switch 28, touch switch 34, eyeball illumination light 166a to 166d are connected.

  The encoder 129 encodes (encodes) the audio signal and the image signal into audio data and image data of a predetermined method. Connected to the encoder 129 are a camera 64, an image sensor 72, a sound collection microphone 24, a call microphone 37, an audio connector 26, and a video connector 27.

  The encoder 129 includes an audio signal input from the sound collection microphone 24 and the call microphone 37, an image signal input from the camera 64, an audio signal input from the audio connector 26, an image signal input from the video connector 27, and Then, an image signal of the image sensor 72 that captures a peripheral image of the eye is input. The audio signal and image signal input to the encoder 129 are encoded into audio data and image data, and then input to the processing unit 123. The input audio data and image image data are used for a reproduction operation by the processing unit 123 or recorded in the flash memory 122.

  The decoder 121 decodes (decodes) audio data and image data into audio signals and image signals. An LCD driver 125, a speaker amplifier 162, an audio connector 26, and a video connector 27 are connected to the decoder 121. The LCD driver 125 is a drive control device for a liquid crystal panel, and is connected to the display panel 36 and the display panel 63. The speaker amplifier 162 is a device that amplifies an audio signal and outputs it to the speaker, and is connected to the ear speaker 23 and the front speaker 70.

  When reproducing the image data, the image data recorded in the flash memory 122 or the image data input from the encoder 129 is input to the decoder 121 via the processing unit 123. The image data input to the decoder 121 is decoded into an image signal and then supplied to the display panels 36 and 63 via the LCD driver 125. Then, an image based on the image data is displayed on the display panel 36 or the display panel 63 to which the image signal is input. The image signal output from the decoder 121 to the video connector 27 is output to an external device via the video connector 27.

  When displaying an image, the processing unit 123 turns on the backlight 35 for the display panel 36 and the backlight 62 for the display panel 63 as necessary. The BL driver 126 is a drive control device for the backlight, and is connected to the backlight 35 and the backlight 62. The processing unit 123 transmits a drive signal to the BL driver 126, and the BL driver 126 lights the backlights 35 and 62 individually.

  When reproducing the audio data, the audio data recorded in the flash memory 122 or the audio data input from the encoder 129 is input to the decoder 121 via the processing unit 123. The audio data input to the decoder 121 is decoded into an audio signal and then output to one or both of the ear speaker 23 and the front speaker 70 via the speaker amplifier 162. Then, sound is output from the ear speaker 23 or the front speaker 70 to which the sound signal is input. The audio signal output from the decoder 121 to the audio connector 26 is output to the stereo earphone 100 via the audio connector 26.

  In the case of this embodiment, the ear speaker 23 and the front speaker 70 are assumed to use monaural sound, and the ear speaker 23 and the front speaker 70 emit a sound in which left and right audio signals are synthesized.

  On the other hand, when an audio signal is output to the stereo earphone 100, the sound of the left channel or the right channel of the first speaker 103 and the second speaker 104 is output. Here, since the HMD 1 of this embodiment is for both left and right, the channel of the audio signal to the stereo earphone 100 is switched according to the mounting position. That is, when the display body 20 is attached to the right eye, the right channel sound is output to the first speaker 103 and the left channel sound is output to the second speaker 104. When the display body 20 is attached to the left eye, the left channel sound is output to the first speaker 103 and the right channel sound is output to the second speaker 104.

The memory 127 stores a control program executed by the processing unit 123.
When the main switch 28 for turning on / off the entire power source, the operation switch 30 for performing a pointing operation on the screen, or the touch switch 34 for performing various operations by a touch operation are operated by the user, the operation can be performed from these switches. A control signal is output to the processing unit 123. The processing unit 123 detects an operation based on the control signal, and executes an operation defined in the control program.

  The BT communication circuit 130 is a communication circuit for performing Bluetooth (registered trademark) communication with other devices. The WiFi communication circuit 131 is a communication circuit for performing wireless LAN communication (IEEE 802.11) with other devices. The 3G / LTE communication circuit 138 is a communication circuit for performing mobile communication with other devices.

  The acceleration sensor 132 is used for detecting the inclination of the HMD 1. The acceleration sensor 132 is a triaxial sensor, for example, and detects gravitational acceleration. The acceleration sensor 132 is attached, for example, in the apparatus main body 21.

  Hereinafter, the coordinates of the acceleration sensor 132 are as follows: when the user standing upright on the ground is wearing the HMD 1 on the head, the vertical direction for the user is the Z-axis direction, the horizontal direction is the X-axis direction, and the front-back direction is the Y-axis direction. To do. The acceleration sensor 132 is attached so that the detected value in the Z-axis direction becomes a negative value in the case of the right-eye observation form, and the detected value in the Z-axis direction becomes a positive value in the case of the left-eye observation form. .

  The geomagnetic sensor 133 is used for detecting the direction of the HMD 1. An angular velocity sensor (gyro sensor) 134 is used to detect rotation of the HMD 1. The GPS sensor 135 is used for positioning detection using GPS (Global Positioning System). The temperature / humidity sensor 136 is used to detect the temperature and humidity of the environment. The heart rate sensor 137 contacts the user's cheek and detects the user's heart rate.

  FIG. 22 is a functional block diagram of the processing unit 123 according to the present embodiment. The processing unit 123 includes an input detection unit 301, a control unit 302, a storage unit 303, an eyeball illumination unit 304, an audio adjustment unit 305, an image switching unit 306, an image display unit 307, and a transmission / reception unit 308. In FIG. 22, the WiFi communication circuit 131 and the 3G / LTE communication circuit 138 (see FIG. 21) are collectively referred to as “communication interfaces (131, 138)”.

  The input detection unit 301 determines the type of operation performed on the touch switch 34, and outputs a control signal based on the determined result to the control unit 302, the image switching unit 306, and the transmission / reception unit 308. The types of operations performed on the touch switch 34 are a transmission instruction, a reception instruction, and the like. When a predetermined area on the touch switch 34 is touched, the input detection unit 301 determines the type of these operations based on the touched area.

  The control unit 302 determines the right-eye observation form or the right-eye observation form based on at least one of the detection value detected by the acceleration sensor 132 and the image captured by the image sensor 72. Here, with respect to the image captured by the image sensor 72, the control unit 302, for example, the relative position between the eyebrows and the eye, the behavior of eyelids in blinking, the eyebrow contour, the eye contour, and the lowering of the left and right corners of the eye Based on the above, it is determined whether the user's eye shown in the captured image is an image of the left or right eye (right eye observation form or right eye observation form).

  The control unit 302 controls lighting and extinguishing of the eyeball illumination light 166 based on the determined result. Further, the control unit 302 controls the sound output to the stereo earphone 100 connected to the audio connector 26 based on the determined result. Further, the control unit 302 controls an image projected from the display unit 60 based on the determined result.

  The control unit 302 performs calibration (correction) for gazing point detection based on the control signal output from the input detection unit 301 and the captured image. The control unit 302 causes the storage unit 303 to store the relational expression and information generated by the calibration for detecting the gazing point. Hereinafter, the generated relational expression and information are referred to as “parameters for operating based on the line of sight”. Note that calibration for detecting a point of interest will be described later.

  The control unit 302 determines whether the arrangement of the display unit 60 is appropriate based on the captured image. When the relative position of the display unit 60 with respect to the user's face is not appropriate, the control unit 302 causes the ear speaker 23 to output a sound to be announced so as to shift the display unit 60 in an appropriate direction. Such alignment processing will be described later.

  The storage unit 303 stores parameters for operation based on the line of sight generated by the control unit 302.

  Based on the control of the control unit 302, the eyeball illumination unit 304 drives the eyeball illumination lights 166a to 166d to be turned on or off.

  The voice adjustment unit 305 receives voice data output from the encoder 129 (voice data of the user of the HMD 1) and a reception signal output from the transmission / reception unit 308. The voice adjustment unit 305 extracts the voice data of the user of the other terminal (hereinafter also referred to as voice data of the other terminal) from the reception signal output from the transmission / reception unit 308. The voice adjustment unit 305 is based on the voice data output from the encoder 129 (HMD1 user voice data) or the voice of the other terminal user voice data based on the information indicating the mounting state output from the control unit 302. Is output to the left or right speaker of the stereo earphone 100, and the volume of each of the left and right speakers is adjusted.

  The image switching unit 306 acquires the image data read from the flash memory 122 or the image data output from the encoder 129 based on the control signal output from the input detection unit 301. The image switching unit 306 switches the display mode of the acquired image data based on the information indicating the mounting state output by the control unit 302, and outputs the image data to the image display unit 307 in the switched display mode. Here, the display mode is, for example, a first display mode or a second display mode.

The image switching unit 306 extracts an image switching signal from the reception signal output from the transmission / reception unit 308. When the image switching signal can be extracted, the image switching unit 306 outputs the image data read from the flash memory 122 or the image data output by the encoder 129 to the image display unit 307 based on the extracted image switching signal. The image switching unit 306 converts the resolution of the image data output from the encoder 129 or the image data read from the flash memory 122 based on the control signal output from the input detection unit 301, and converts the converted image data into an image. The data is output to the display unit 307.
The image switching unit 306 displays a password setting screen, a menu screen, and the like on the display panel 63 according to instructions from the control unit 302.

  The image data output from the image switching unit 306 is input to the image display unit 307. The image display unit 307 generates image data to be displayed on the display panel 63 based on these input image data. The image display unit 307 outputs the generated image data to the transmission / reception unit 308, the video connector 27, and the LCD driver 125.

  The transmission / reception unit 308 transmits the image data output from the image display unit 307 via the communication interface (131, 138) based on the control signal output from the input detection unit 301. The transmission / reception unit 308 outputs a reception signal received via the communication interfaces (131, 138) to the image switching unit 306.

As described above, the head-mounted information input / output device (HMD1) of the present embodiment includes a display unit 60 (display panel 63) that displays an image, a holding unit that holds the display unit near the user's eyes, A plurality of illumination units (eyeball illumination lights 166) that are provided on the display unit 60 and illuminate the vicinity of the user's eyes (within a predetermined distance), and a plurality of illumination units (eyeball illumination lights 166) depending on the positional relationship between the user and the display unit 60 A control unit (control unit 302) that controls the lighting state of the illumination unit.
With such a configuration, the display unit 60 is arranged outside the user's right eye, left eye, and left and right eyes in the case of parameter acquisition for operation based on the user's line of sight, or in the case of operation based on the user's line of sight. Even if it is done, it can irradiate the user's eyes with light. As a result, even when the display unit is arranged on the left side, the right side, the lower side, or the upper side of the user's eyes, the position observed by the user can be detected.
The holding unit that holds the display unit 60 in the vicinity of the user's eyes includes a head pad 46, a headband 40, a connection pin 41, the display body 20, the connection hole 31 of the display body 20, and the like.

Next, the operation of the HMD 1 will be described. FIG. 23 is a flowchart of operations performed after the HMD 1 according to the present embodiment is powered on.
(Step S1) The processing unit 123 waits for completion of activation of the system of the HMD1, and proceeds to step S2 when the activation of the system is completed. The activation means that the main switch 28 is operated to cold start the HMD 1 and the HMD 1 is not used for a predetermined time to reach a sleep state, and then the touch switch 34 or the like is operated. For example, when it starts.

(Step S <b> 2) The processing unit 123 performs a process of detecting whether the user is observing the display unit 60 of the HMD 1 with the left or right eye (hereinafter referred to as a left / right eye detection process). Detailed processing of the “left and right eye detection processing” will be described later. The processing unit 123 switches the orientation of the image displayed on the display unit 60 based on the detected result.

(Step S <b> 3) The processing unit 123 performs a process of performing a guide (hereinafter referred to as a mounting guide process) so that the display unit 60 of the HMD 1 is mounted at the correct mounting position. The detailed processing of “mounting guide processing” will be described later.

(Step S4) The processing unit 123 performs a process of selecting an eyeball illumination light 166 to be turned on and an eyeball illumination light to be turned off (hereinafter referred to as an eyeball illumination light selection process). Details of the “eyeball illumination light selection process” will be described later.

(Step S5) The processing unit 123 performs calibration for gazing point detection. Details of the calibration process for detecting the point of interest will be described later.

Next, a procedure for “left and right eye detection processing” will be described.
FIG. 24 is a flowchart of the left and right eye detection processing procedure according to this embodiment.

(Step S101) The control unit 302 of the processing unit 123 determines whether or not the detection value detected by the acceleration sensor 132 is stronger in the Z-axis direction than in the X-axis direction and the Y-axis direction. When it is determined that the detected value in the Z-axis direction is stronger than the X-axis direction and the Y-axis direction (step S101; Yes), the process proceeds to step S102. When it is determined that the detected value in the Z-axis direction is not stronger than the X-axis direction and the Y-axis direction (step S101; No), the process proceeds to step S103.

(Step S102) When it is determined that the detected value in the Z-axis direction is stronger than the X-axis direction and the Y-axis direction, the display unit 60 displays the first value as shown in FIG. 5 (A) or FIG. 5 (B). This is a display mode wearing form. Therefore, the control unit 302 determines whether the observation mode is the right eye observation mode or the left eye observation mode based on the detection value of the acceleration sensor 132. Next, the control unit 302 outputs information indicating the determined result to the image switching unit 306.

(Step S103) When it is determined that the detected value in the Z-axis direction is not stronger than the X-axis direction and the Y-axis direction, the control unit 302 drives the image sensor 72 because the user is sleeping or lying on the back. To do.
FIG. 25 is a diagram illustrating an example of a captured image in the imaging range P in the case of the right eye observation form. FIG. 26 is a diagram illustrating an example of a captured image in the imaging range P in the case of the left-eye observation form.
In order to discriminate the observation form, the control unit 302 captures a peripheral image of the eye with the image sensor 72 using the imaging region P described in FIG. For example, the captured image is based on the captured image (hereinafter referred to as the first right-eye image) when used in the first display mode and the right-eye observation mode illustrated in FIG. Therefore, as shown in FIG. 25, the first right eye image 610R is an erect image. As shown in FIG. 25, in the first right eye image 610R, the right eyebrow image 602R3 is in the + Z direction with respect to the right eye image 601R3.
On the other hand, as shown in FIG. 26, an eye peripheral image (hereinafter referred to as a first left eye image) 610L when used in the first display mode and the left eye observation form is an inverted image. As shown in FIG. 26, in the first left eye image 610L, the right eyebrow image 602L is in the −Z direction with respect to the left eye image 601L.

(Step S104) Returning to FIG. 24, the control unit 302 determines whether or not an eyebrow image can be extracted from the captured image. If it is determined that the image of the eyebrows can be extracted (step S104; Yes), the process proceeds to step S105. For example, when it is determined that the eyebrow image cannot be extracted, such as when the display unit 60 is close to the eyes or when it is difficult to determine the eyebrows by makeup or the like (step S104; No), the process proceeds to step S106. Note that the control unit 302 extracts an image of eyebrows using, for example, a pattern matching technique.
For example, when the user is using the right-eye observation mode, the image captured in the selection area Q (see FIG. 19A) is an image as shown in FIG. On the other hand, when the user is using the left-eye observation mode, the image captured in the selection region R (see FIG. 20B) is an image as shown in FIG. FIG. 27 is a diagram illustrating an example of a captured image in the left-eye observation mode. In FIG. 27, the image of the selection region R is shown rotated by 180 degrees. Therefore, in FIG. 27, the −Z side is the upper side of the user, the + Z side is the lower side of the user, the −X side is the left side of the user, and the + X side is the right side of the user. In FIG. 27, reference numeral 601L1 is a left eye image, and reference numeral 602L1 is a left eyebrow image.
Note that during the processing in steps S101 to S108, the control unit 302 may turn on or turn off the eyeball illumination light 166. During the processes in steps S101 to S108, the control unit 302 may turn on the backlight 62.

(Step S105) When the image of the eyebrows can be extracted from the captured image, the control unit 302 uses, for example, a pattern matching technique or an image feature amount from the position of the eyebrow image and the eye image or the shape of the eyebrows. Using the image comparison technique, it is determined whether the captured image is a right-eye image or a left-eye image of the user. Next, the control unit 302 outputs information indicating the determined result to the image switching unit 306.
For example, as illustrated in FIG. 25, when it is determined that the image of the eyebrows is in the + Z direction with respect to the image of the eyeball, the control unit 302 causes the right eye that the user observes the display unit 60 with the right eye. It is determined that it is an observation form. On the other hand, as illustrated in FIG. 26, when it is determined that the image of the eyebrows is in the −Z direction with respect to the image of the eyeball, the control unit 302 displays the left where the user observes the display unit 60 with the left eye. It is determined that it is an eye observation form.

(Step S106) When the eyebrow image cannot be extracted from the captured image, the control unit 302 uses the pattern matching technique or the image comparison technique based on the image feature amount, for example, based on the behavior of eyelids in blinking. It is discriminated whether it is the right eye or the left eye. Next, the control unit 302 outputs information indicating the determined result to the image switching unit 306.
For example, since the upper eyelid moves during blinking, when the eyelid moves from the + Z direction to the −Z direction in FIG. 11 and immediately returns from the −Z direction to the + Z direction, the control unit 302 displays the right eye. It is determined that this is the right-eye observation mode in which the unit 60 is observed. Further, when the movement start direction and the return direction of the eyelid are reversed, the control unit 302 determines that the left-eye observation mode is that the user observes the display unit 60 with the left eye.
As described above, according to the present embodiment, even when the user is lying in the horizontal direction, it is possible to detect the movement of the eyebrows and the eyelashes without depending on the direction of gravity detected by the acceleration sensor 132. The direction of the eye that the user is observing can be determined.

(Step S107) When it is determined that the user is using the right-eye observation mode based on the determination results up to Step S106, the control unit 302 selects the selection region Q from the imaging region P captured by the imaging element 72. Select and proceed to step S108. On the other hand, when it is determined that the user is using the left-eye observation form, the control unit 302 selects the selection region R from the imaging region P, and proceeds to step S108.

(Step S <b> 108) If it is determined that the right-eye observation mode is set based on the determination results up to step S <b> 106, the image switching unit 306 sets the display direction of the display unit 60 for the right eye. When the stereo earphone 100 is connected, the audio adjustment unit 305 sets the left and right stereo audio output so that the first speaker 103 is for the right ear and the second speaker 104 is for the left ear.
On the other hand, when the left-eye observation mode is determined, the image switching unit 306 sets the display direction of the display unit 60 for the left eye. The display direction for the left eye is a direction in which the display face for the right eye is rotated 180 degrees and turned upside down. When the stereo earphone 100 is connected, the audio adjustment unit 305 sets the left and right stereo audio output so that the first speaker 103 is for the left ear and the second speaker 104 is for the right ear.

In addition, although the example which discriminate | determines whether it is a right-eye observation form in step S101-S106 based on the detected value of the acceleration sensor 132 and a captured image was demonstrated, determination of whether it is a right-eye observation form The method is not limited to this.
For example, the HMD 1 may be detected by a switch or the like (not shown) attached to the display main body 20. Of course, the user may manually set the right-eye observation form or the left-eye observation form.

Next, the “mounting guide processing” procedure will be described.
FIG. 28 is a flowchart of the processing procedure of the mounting guide for the HMD 1 according to the present embodiment. In the following description, the display panel 63 is referred to as an LCD 63. In this process, when the user uses the HMD 1 and the LCD 63 is being displayed, this process may be appropriately performed to prompt the user to correct the deviation of the LCD 63. Alternatively, it may be performed when the user instructs to perform alignment by operating the touch switch 34 or the like.

  In the present embodiment, as described with reference to FIGS. 18A and 18B, the display unit 60 including the image sensor 72 is disposed obliquely below the user's eye. For this reason, as shown in FIG. 17, the image sensor 72 captures an image around the user's eye obliquely from below. However, when the distance between the display unit 60 and the eyes is not appropriate, and when the screen center of the display unit 60 and the center of the eyes are shifted, the user cannot properly observe the image displayed on the LCD 63. For this reason, in the present embodiment, the control unit 302 detects that the position of the display unit 60 is deviated from the position of the user's eye based on the image captured by the image sensor 72 and announces to the user. To do.

(Step S201) The control unit 302 extracts an eye image using, for example, a pattern matching technique. When it is determined that the user is using the right-eye observation form based on the determination result of “left and right eye detection process” (step S201; Yes), the control unit 302 proceeds to step S202, and the user When it is determined that the left-eye observation mode is used (step S201; No), the process proceeds to step S214.

(Step S202) The control unit 302 determines whether or not the position of the user's eyes is shifted in the + X direction based on the captured image (see, for example, FIG. 11). When it is determined that the position of the user's eye is shifted in the + X direction (step S202; Yes), the process proceeds to step S203. When it is determined that the position of the user's eye is not shifted in the + X direction (step S202; No), the process proceeds to step S204.

(Step S203) When it is determined that the position of the user's eye is shifted in the + X direction, the display unit 60 is shifted to the right side of the user with respect to the user's eye. For this reason, in such a case, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the left side of the user.

(Step S <b> 204) The control unit 302 determines whether or not the position of the user's eye is shifted in the −X direction based on the captured image. When it is determined that the position of the user's eye is shifted in the −X direction (step S204; Yes), the process proceeds to step S205. When it is determined that the position of the user's eye is not shifted in the −X direction (step S204; No), the process proceeds to step S206.

(Step S205) When it is determined that the position of the user's eyes is shifted in the -X direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the user's right side. .

(Step S206) The control unit 302 determines whether or not the position of the user's eye is shifted in the + Z direction based on the captured image. When it is determined that the user's eyes are displaced in the + Z direction (step S206; Yes), the process proceeds to step S207. When it is determined that the position of the user's eye is not shifted in the + Z direction (step S206; No), the process proceeds to step S208.

(Step S207) When it is determined that the position of the user's eye is shifted in the + Z direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the user.

(Step S208) The control unit 302 determines whether or not the position of the user's eye is shifted in the −Z direction based on the captured image. When it is determined that the position of the user's eye is shifted in the −Z direction (step S208; Yes), the process proceeds to step S209. When it is determined that the position of the user's eye is not shifted in the + Z direction (step S208; No), the process proceeds to step S210.

(Step S209) When it is determined that the position of the user's eyes is shifted in the -Z direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the user's lower side. Do.

(Step S210) The control unit 302 determines whether the image of the user's eye is larger than a predetermined size based on the captured image. When it is determined that the image of the user's eye is larger than the predetermined size (step S210; Yes), the process proceeds to step S211. When it is determined that the image of the user's eye is not larger than the predetermined size (step S210; No), the process proceeds to step S212. For example, the control unit 302 extracts an eye image, and extracts the eye width based on the extracted image. The control unit 302 determines whether or not the extracted eye width is larger than a predetermined size.

(Step S211) If it is determined that the image of the user's eyes is larger than the predetermined size, it is estimated that the user's eyes are too close to the display unit 60, so that the processing unit 123 moves the display unit 60 away from the eyes. Announcement is made from the ear speaker 23.

(Step S212) The control unit 302 determines whether the image of the user's eye is smaller than a predetermined size based on the captured image. When it is determined that the image of the user's eye is smaller than the predetermined size (step S212; Yes), the process proceeds to step S213. When it is determined that the image of the user's eye is not smaller than the predetermined size (step S212; No), the wearing state determination process is terminated.

(Step S213) If it is determined that the image of the user's eyes is smaller than the predetermined size, it is estimated that the user's eyes are too far from the display unit 60, so that the processing unit 123 brings the display unit 60 closer to the eyes. Announcement is made from the ear speaker 23.

(Step S214) When it is determined that the user is using the left-eye observation form, the control unit 302 shifts the position of the user's eyes in the + X direction based on the captured image (for example, see FIG. 27). It is determined whether or not. When it is determined that the position of the user's eye is shifted in the + X direction (step S214; Yes), the process proceeds to step S215. When it is determined that the position of the user's eye is not shifted in the + X direction (step S214; No), the process proceeds to step S216.

(Step S215) When it is determined that the position of the user's eye is shifted in the + X direction, the display unit 60 is shifted to the left side of the user with respect to the user's eye. For this reason, in such a case, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the right side of the user.

(Step S216) The control unit 302 determines whether the position of the user's eye is shifted in the −X direction based on the captured image. When it is determined that the position of the user's eye is shifted in the −X direction (step S216; Yes), the process proceeds to step S217. When it is determined that the position of the user's eye is not shifted in the −X direction (step S216; No), the process proceeds to step S218.

(Step S217) When it is determined that the position of the user's eyes is shifted in the −X direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the left side of the user. .

(Step S218) The control unit 302 determines whether or not the position of the user's eye is shifted in the + Z direction based on the captured image. When it is determined that the position of the user's eye is shifted in the + Z direction (step S218; Yes), the process proceeds to step S219. When it is determined that the position of the user's eye is not shifted in the + Z direction (step S218; No), the process proceeds to step S220.

(Step S219) When it is determined that the position of the user's eyes is shifted in the + Z direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the user's lower side. .

(Step S <b> 220) The control unit 302 determines whether the position of the user's eye is shifted in the −Z direction based on the captured image. When it is determined that the position of the user's eye is shifted in the −Z direction (step S220; Yes), the process proceeds to step S221. When it is determined that the position of the user's eye is not shifted in the + Z direction (step S220; No), the process proceeds to step S210.

(Step S221) When it is determined that the position of the user's eye is shifted in the -Z direction, the processing unit 123 makes an announcement from the ear speaker 23, for example, so as to shift the display unit 60 to the user. .
After the processing in step S221, the processing unit 123 performs steps S211 to S213, and ends the processing for mounting the HMD1.

  Next, the eyeball illumination light selection process will be described. FIG. 29 is a flowchart of the selection processing procedure of the eyeball illumination light of the HMD 1 according to the present embodiment.

(Step S401) When the control unit 302 determines that the user is using the right-eye observation mode based on the determination result of the “left and right eye detection process” (step S401; Yes), the process proceeds to step S402. If it is determined that the user is using the left-eye observation form (step S401; No), the process proceeds to step S403.

(Step S402) When it is determined that the user is using the right-eye observation mode, the control unit 302 selects the eyeball illumination lights 166a and 166b and stores the selected result in the storage unit 303 as the eyeball illumination light selection setting. Remember. The control unit 302 turns off the eyeball illumination lights 166c and 166d and ends the eyeball illumination light selection process.

(Step S403) When it is determined that the user is using the left-eye observation mode, the control unit 302 selects the eyeball illumination lights 166c and 166d and stores the selected result in the storage unit 303 as the eyeball illumination light selection setting. Remember. The control unit 302 turns off the eyeball illumination lights 166c and 166d and ends the eyeball illumination light selection process.

  As described above, the HMD 1 of the present embodiment selects and turns on the eyeball illumination lights 166a to 166d according to the positional relationship between the display unit 60 and the user. As a result, the HMD 1 of the present embodiment can appropriately illuminate the user's eyes by appropriately selecting a light for illuminating the eyeball, and can reduce power consumption.

Here, a comparative example with the first embodiment will be described.
FIG. 30 is a diagram for explaining an actual example of calibration (hereinafter referred to as “calibration comparison example”) as a comparison example with “calibration for gazing point detection” according to the first embodiment. Hereinafter, a case where the user performs a calibration comparative example using the HMD 1 will be described.
In FIG. 30, an image 400 </ b> A is an image displayed on the display panel 63. When performing an operation on the display unit 60 by a user's line-of-sight input, the HMD 1 detects the line-of-sight coordinate and the coordinate on the display panel 63 each time it is attached, and the line-of-sight coordinate and the coordinate on the display panel 63 are detected. It is necessary to perform calibration. The operation by line of sight input is, for example, an operation of displaying a software keyboard on the display panel 63 and inputting the key based on the line of sight.
In such a line-of-sight operation, the user's line of sight and the position on the display panel must match. For this reason, as shown in FIG. 30, the HMD 1 displays white circles 410a to 410e sequentially at the four corners and the center of the image 400A. The user aligns his / her line of sight with the position of the displayed white circle. The HMD 1 calibrates the camera coordinate system by the image sensor 72 that captures the line of sight and the coordinate system on the display panel 63 so as to eliminate the deviation between the display position and the line of sight. With this process, the position of the user's line of sight and the information displayed on the display panel 63 is calibrated.
In the calibration comparison example, it is necessary to perform such calibration every time the user selects the calibration setting and wears it on the head, for example. Such calibration is troublesome for the user and is not convenient.
On the other hand, in the HMD 1 according to the present embodiment, the user does not set the calibration as in the calibration comparative example, but operates the information displayed on the display panel 63 as will be described later. The parameter for operating based on (gaze point) is acquired, and calibration for gaze point detection is performed.

Next, the details of the calibration for gazing point detection will be described.
FIG. 31 and FIG. 32 are flowcharts of a calibration procedure for gazing point detection according to the present embodiment. FIG. 33 is a diagram for explaining turning on and off of the eyeball illumination light 166 in the right-eye observation mode according to this embodiment. FIG. 34 is a diagram for explaining the relationship between the position of the display unit 60 and the eyeball illumination light 166 according to the present embodiment, and is a diagram of the user's face viewed from the horizontal front.
In the following, as shown in FIG. 7A, FIG. 7B, and FIG. 34, a state in which the display unit 60 is arranged in the right-eye observation form and under the eye, or the left-eye observation form and An example will be described in which a state in which the display unit 60 is placed under the eyes is detected and calibration for detecting a point of interest is performed.

  In FIG. 34, positions 600a to 600b indicated by alternate long and short dash lines are examples of positions where the display unit 60 is disposed. As will be described in the following processing, when the display unit 60 is at the position 600a, the HMD 1 turns on the eyeball illumination lights 166a and 166b in the vicinity of the eye and turns off the others. On the other hand, when the display unit 60 is at the position 600b, the HMD 1 turns on the eyeball illumination lights 166c and 166d in the vicinity of the eyes and turns off the others.

(Step S <b> 501) The control unit 302 determines whether or not the user is using the right-eye observation mode based on the determination result of “left and right eye detection processing”. When it is determined that the user is using the right-eye observation form (step S501; Yes), the control unit 302 proceeds to step S502, and when it is determined that the user is not using the right-eye observation form (step S501). Step S501; No), the process proceeds to Step S503.

(Step S <b> 502) When the right eye observation mode is determined, the control unit 302 reads the selection setting of the eyeball illumination light stored in the storage unit 303. Next, the control unit 302 turns on the eyeball illumination lights 166a and 166b via the eyeball illumination unit 304 based on the selection setting of the eyeball illumination light to be read.
As shown in FIG. 33, when the display unit 60 is placed under the user's right eye for observation, the eyeball illumination lights 166c and 166d cannot be illuminated because they are arranged very close to the cheek. Therefore, in this embodiment, as shown at a position 600a in FIG. 34, the eyeball illumination unit 304 turns off the eyeball illumination lights 166c and 166d in order to reduce power consumption.

(Step S <b> 503) When it is determined that the left-eye observation mode is selected, the control unit 302 reads the selection setting of the eyeball illumination light stored in the storage unit 303. Next, the control unit 302 turns on the eyeball illumination lights 166c and 166d via the eyeball illumination unit 304 based on the read setting of the eyeball illumination light to be read, as indicated by a position 600b in FIG. The eyeball illumination unit 304 turns off the eyeball illumination lights 166a and 166b in order to reduce power consumption.
With the above processing, the HMD 1 of the present embodiment can appropriately select a light that illuminates the eyeball according to the positional relationship between the user's eye and the display unit 60. Furthermore, the HMD 1 of the present embodiment can efficiently take measures for power saving by turning off the eyeball illumination light 166 that is not used.

(Step S504) After step S502 or S503 ends, the image switching unit 306 displays a password input screen as shown in FIG. This is a method for inputting a password set by a user, which is used in PCs and mobile phones that are widely used for mobile purposes, and is intended to prevent others from using the password.
FIG. 35 is an example of a password input screen 400e according to the present embodiment. In FIG. 35, the longitudinal direction of the password input screen 400e is defined as the X-axis direction, and the short direction is defined as the Z-axis direction. As shown in FIG. 35, on the LCD 63, alphabet keys 461 from A to Z, a symbol key 462, a shift key 463, a 10-key switching key 464, a space key 465, a decision (Done) key 466, and input characters A region 467 for displaying “” and a cursor 468 are displayed. In FIG. 35, the symbol Dx represents the coordinate in the X-axis direction, and the symbol Dz represents the coordinate in the Z-axis direction.

(Step S <b> 505) Returning to FIG. 31, the control unit 302 initializes “i” as a parameter suffix (character string) to zero.

(Step S506) The control unit 302 adds 1 to the suffix “i”.

(Step S507) The control unit 302 acquires an image (including video) captured by the image sensor 72. Next, the control unit 302 acquires the coordinates (Ex (i), Ez (i)) of the bright spot by the eyeball illumination lights 166a to 166d irradiated on the eyeball based on the acquired image.

  An image acquired by the control unit 302 is, for example, FIG. As shown in FIG. 11, a bright spot 604A by the eyeball illumination light 166a and a bright spot 604B by the eyeball illumination light 166b are shown in the image, and the bright spot moves in conjunction with the movement of the eyeball. That is, when the eyeball moves to the left side, the bright spots 604A and 604B move to the + X side. Further, when the eyeball moves to the left side, the same bright spot moves to the −X side. When the eyeball moves upward, the bright spot moves to the + Z side, and when the eyeball moves downward, the bright spot moves to the −Z side. As for the movement of the eyeball, the position in the LCD 63 and the position of the bright spot are uniquely determined depending on where the user looks at the display in the LCD 63.

A cursor 468 is displayed on the password input screen 400e of FIG. The cursor 468 can be moved up, down, left, and right by the user operating the operation switch 30 by tilting it up, down, left, or right. As shown in FIG. 35, for example, when “P” is selected as one character of the password, the cursor 468 is moved to the character “P”, and the operation switch 30 is further pressed to change the character “P”. A character is entered.
Immediately before the operation switch 30 is pressed, the user has a very high probability of gazing at the letter “P”, so the coordinates of the bright spot of the eye at this time are captured. Note that the control unit 302 gazes at the first character of the password by using an average value of the coordinates (Xa, Za) of the two bright spots 604A and the coordinates (Xb, Zb) of the bright spots 604B in FIG. The measured values Ex (1) and Ez (1) of the X-axis and Y-axis of the bright spot of the eye are calculated as in the following expressions (2) and (3).

  Ex (1) = (Xa + Xb) / 2 (2)

  Ez (1) = (Za + Zb) / 2 (3)

  The control unit 302 causes the storage unit 303 to store the calculated coordinates (Ex (i), Ez (i)) of the measurement value.

(Step S508) Returning to FIG. 31, the control unit 302 acquires coordinates (Dx (i), Dz (i)) on the LCD 63 on which characters are displayed. For example, when the character “P” is selected in FIG. 35, the control unit 302 acquires the coordinates of the character “P” and acquires (Dx (1), Dz (1)) as the coordinates of the first character of the password. To do.

(Step S509) The control unit 302 determines whether or not the authentication of the password set by the user has been completed. If it is determined that password authentication has been completed (step S509; Yes), the process proceeds to step S510. If it is determined that password authentication has not been completed (S509; No), steps S506 to S509 are repeated.
In steps S507 to S509, a plurality of password characters are selected, so that the measurement coordinates of the bright point of the eye (Ex (i), Ez (i)) and the position coordinates of the characters on the LCD 63 ( The value of Dx (i), Dz (i)) is obtained. When the user determines that the password has been input, the user selects and presses the Done key 466 with the operation switch 30. At this time, the control unit 302 also acquires the coordinates of the Done key 466.

(Step S510) By the password input operation up to Step S312, the measured value of the bright point of the eye and the position information of the character corresponding to this are obtained. Therefore, after this, if the coordinates of the acquired bright spot are measured, it becomes possible to know where the user is gazing on the LCD 63. That is, by these processes, an operation by line-of-sight input becomes possible. Therefore, the control unit 302 generates function expressions for obtaining the display position (Dx, Dz) based on the measured values (Ex, Ez) as the following expressions (4) and (5).

  Dx = F1 (Ex, Ez) (4)

  Dz = F2 (Ex, Ez) (5)

  In the equations (4) and (5), F1 is a function of Dx and (Ex, Ez), and F2 is a function of Dz and (Ex, Ez). Next, the control unit 302 stores the generated expressions (4) and (5) in the storage unit 303.

(Step S511) Upon completion of the password input, the image switching unit 306 displays a menu icon as shown in FIG.
FIG. 36 is a diagram showing an example of the menu screen 400f according to the present embodiment. In the example shown in FIG. 36, nine icons 471 to 479 and a cursor 480 are displayed on the menu screen 400f. For example, the icon 471 is an icon that indicates an instruction to capture a moving image. Icons 471 to 479 are referred to as applications. Of these, the icon 471 is an application having a function of inputting with a line of sight, and is particularly referred to as a line-of-sight input application.
In addition, in the application for taking a moving image represented by the icon 471, it is possible to perform moving image shooting and recording with the camera 64, and to move a marker displayed on the screen and to operate a button by a line-of-sight input as described later. It has become.
In the initial state, a cursor 480 is displayed at the position of the icon 471 as shown in FIG.

Note that the menu screen in FIG. 36 may be a menu screen for performing various settings in, for example, an OS (operation system) that controls the processing unit 123. The menu screen for performing various settings includes, for example, various setting buttons (or icons, keys, etc.) for adjusting the brightness of the display panel 63, adjusting the contrast of the display panel 63, setting the date and time, and setting whether or not to enter the standby state. This is the displayed screen.
Further, an application whose icon is displayed on the menu screen 400f is not limited to an application executed inside the HMD 1. For example, the application in which the icon is displayed may be an application executed on the HMD or PC used by the communication partner that is performing communication. In addition, the application in which the icon is displayed may be, for example, a Web application that is executed on a network such as the Internet.

(Step S512) Returning to FIG. 31, the control unit 302 determines whether or not the icon 471 is manually selected by the operation switch 30 on the menu screen. When it is determined that the icon 471 has not been selected manually by the operation switch 30 (step S512; No), the process proceeds to step S513. When it is determined that the icon has been manually selected by the operation switch 30 (step S512; Yes), the process proceeds to the circle C (FIG. 32, step S519).

(Step S513) The control unit 302 determines whether an application other than the icon 471 has been selected. When it is determined that an application other than the icon 471 has been selected (step S513; Yes), the process proceeds to step S514, and the control unit 302 performs calibration to further improve the accuracy of the line-of-sight input in the menu operation performed by the user. Do. When it is determined that an application other than the icon 471 has not been selected (step S513; No), the process returns to step S512.

(Step S514) When it is determined that an application other than the icon 471 has been selected, the control unit 302 adds 1 to the suffix “i”.

(Step S515) The control unit 302 acquires a captured image. Next, the control unit 302 acquires the coordinates (Ex (i), Ez (i)) of the bright spot by the eyeball illumination lights 166a to 166d that are irradiated to the eyeball based on the acquired captured image.

(Step S516) The control unit 302 acquires coordinates (Dx (i), Dz (i)) on the LCD 63 on which the selected icon is displayed.

(Step S517) The control unit 302 recalculates the relational expressions (4) and (5) stored in the self unit, and stores the relational expression generated by the recalculation in the storage unit 303.

(Step S518) The control unit 302 executes the selected application (or menu). After executing the application, the process returns to step S511.
That is, the control unit 302 obtains the luminance coordinates and recalculates the relational expressions (4) and (5) every time an icon, a check box, or the like is selected by the operation switch 30 or the like, so that the gaze point is obtained. Perform calibration for detection. Note that each time an icon or check box is selected during execution of various applications including the menu screen 400f, calibration for detecting a point of interest is sequentially performed using the icon or check box as the point of interest. You may go.

(Step S519) Moving to FIG. 32, when it is determined that the icon 471 has been manually selected by the operation switch 30, the control unit 302 adds 1 to the suffix “i”.

(Step S520) The control unit 302 acquires (Dx (i), Dz (i)), which are coordinates on the LCD 63 on which the selected icon is displayed.

(Step S <b> 521) The control unit 302 recalculates the relational expressions (4) and (5) stored in the self unit, and stores the relational expression generated by the recalculation in the storage unit 303.

(Step S <b> 522) The control unit 302 determines whether or not calibration for detecting a gazing point has been sufficiently performed before executing the gaze input application. Specifically, for example, if the password registration is one character, there is a possibility that the calibration is not sufficiently performed. For example, if at least two different gazing points are detected on the password input screen 400e or the menu screen 400f, the control unit 302 determines that the calibration has been sufficiently performed. Note that the two different positions are positions arranged at diagonal positions in the password input screen 400e or the menu screen 400f.
When it is determined that the calibration has been sufficiently performed (step S522; Yes), the process proceeds to step S527. When it is determined that the calibration is not sufficiently performed (step S522; No), the process proceeds to step S523.

(Steps S <b> 523 to S <b> 526) The control unit 302 presents text prompting the user to input again as illustrated in FIG. 37 via the image switching unit 306. The control unit 302 additionally presents the screen shown in FIG. 37, and when the user operates this screen, the coordinates on the LCD 63 (Dx (i), Dz (i)) and the eyeball illumination lights 166a to 166d. The coordinates (Ex (i), Ez (i)) of the bright spot are obtained and calibration for gazing point detection is performed.
FIG. 37 is an example of a screen that prompts the user to input according to the present embodiment. As shown in FIG. 37, a character string “Do you use the line-of-sight input mode?” 481, a “Yes” button 482, and a “No” button 483 are displayed on the screen 400g.
For example, in FIG. 37, when the “Yes” button 482 is manually selected by the operation switch 30, the Done key 466 in FIG. 35 and the moving image capturing instruction icon 471 in FIG. 36 are already selected. Three buttons (or keys, icons) that are separated from each other on the screen have been pressed, and calibration with some degree of reliability has been completed. Then, the control unit 302 performs the same processing as steps S519 to S521 by the processing of steps S523 to S526, and derives a function expression for obtaining (Dx, Dz).

  The display for input includes each key (461-466) in FIG. 35, each icon 471-479 in FIG. 36, each button (482, 483) in FIG. Here, when the position where the calibration is insufficient is in the screen, the Done key 466 may be displayed at a position where calibration is necessary by appropriately changing the display position.

(Step S527) Returning to FIG. 32, the control unit 302 displays the image captured by the camera 64 on the LCD 63 via the image switching unit 306 as shown in FIG.
FIG. 38 is a diagram for explaining the screen 400h of the line-of-sight input application according to the present embodiment. In the example shown in FIG. 38, a captured image 491, a marker 492, an “end button” 493, a “record button” 494, and a “menu button” 495 are displayed on the screen 400h. The marker 492 is a marker that indicates a required position in the image 491 by the user. The “end button” 493 is a button for ending the application and returning to the menu screen 400f shown in FIG. The “record button” 494 is a button for starting to record the image signal captured by the camera 64 in the flash memory 122. The “menu button” 495 is a button for displaying a menu (not shown) for performing settings specific to the application.
When the HMD 1 is communicating with another terminal, the captured image 491 is transmitted to the other terminal via the transmission / reception unit 308.
As shown in the screen 400h, the HMD 1 of the present embodiment is specific to the product 496 by transmitting an image indicating the product 496 with a cursor to another terminal in the image that the user is observing and capturing with the camera 64. Can be communicated to the other party.

(Step S528) Returning to FIG. 32, the controller 302 determines whether or not the user has selected the buttons 493 to 495 with the operation switch 30. When it is determined that the user has selected the buttons 493 to 495 with the operation switch 30 (step S528; Yes), the process proceeds to step S535. When it is determined that the user has not selected the buttons 493 to 495 with the operation switch 30 (step S528; No), the process proceeds to step S529.

(Step S529) When it is determined that the user has not selected the buttons 493 to 495 with the operation switch 30, the control unit 302 acquires the coordinates (Ex, Ez) of the bright spot in the user's eye. Next, the control unit 302 calculates where the user is gazing using the acquired coordinates and the functions of Expressions (4) and (5).

(Step S530) The control unit 302 displays a marker 492 at the calculated coordinate position.
Note that the HMD 1 transmits the image on which the image of the marker 492 is superimposed to another terminal. Since the marker 492 moves when the user moves the gazing point, the gazing point can be transmitted to users of other terminals in a hands-free manner.

(Step S531) The control unit 302 determines whether or not the marker 492 is on the buttons 493, 494, and 495. When it is determined that the marker 492 is on the buttons 493, 494, and 495 (step S531; Yes), the process proceeds to step S532. When it is determined that the marker 492 is not on the buttons 493, 494, and 495 (step S531; No), the process returns to step S528.

(Step S532) When it is determined that the marker 492 is on the buttons 493, 494, and 495, the control unit 302 meditates the user's eyelid twice in a short time based on the captured image. It is determined whether or not. Note that the control unit 302 performs determination using a pattern matching technique or an image comparison technique based on image feature amounts. When it is determined that the user's eyelid has been meditated twice in a short time (step S532; Yes), the process proceeds to step S533. When it is determined that the user's eyelid has not been meditated twice in a short time (step S532; No), the process returns to step S528.

(Step S533) When it is determined that the user's eyelid has been meditated twice in a short time, the control unit 302 determines whether or not the “end button” 493 has been selected by blinking twice. When it is determined that the “end button” 493 has been selected by blinking twice (step S533; Yes), the process proceeds to a circle D (FIG. 31, S511). When it is determined that the “end button” 493 is not selected by two blinks, that is, it is determined that another button is selected (step S533; No), the process proceeds to step S534.

(Step S534) If it is determined that the “end button” 493 is not selected by two blinks, the control unit 302 performs an operation corresponding to the selected button, and returns to step S528.

(Steps S535 to S538) When it is determined that the user has selected the buttons 493 to 495 with the operation switch 30, the control unit 302 performs step S535 based on the coordinates of the buttons 493 to 495 and the bright spot of the user's eyes. To S538 to perform gazing point detection calibration.

(Step S <b> 539) The control unit 302 determines whether or not the “end button” 493 is selected by the operation switch 30. When it is determined by the operation switch 30 that the “end button” 493 has been selected (step S539; Yes), the process proceeds to a circle D (FIG. 31, S511). If it is determined by the operation switch 30 that the “end button” 493 has not been selected (step S539; No), the process proceeds to step S540.

(Step S540) When the “end button” 493 is not selected by the operation switch 30, that is, when it is determined that no other button is selected, the control unit 302 performs an operation corresponding to the selected button, The process returns to step S331.

  As described above, in the HMD 1 of the present embodiment, the user's eyes can be irradiated with appropriate light by turning on and off the eyeball illumination light 166 set in the “eyeball illumination lamp selection process”. For this reason, it is possible to accurately perform calibration for detecting a gazing point.

  As described with reference to FIGS. 9 to 15, when the size of the image is reduced and the position is moved before the calibration for detecting the gazing point, for example, in the selected display region T. The password input screen 400e, the menu screen 400f, and the like may be reduced and displayed. In addition, after performing calibration for gazing point detection, when a gaze input is performed by reducing the size of the image and moving the position, the memory is stored based on the display position and the reduction ratio of the image. The coordinates may be converted by correcting the value of the relational expression stored in the unit 303.

Note that the processing unit 123 may turn on the eyeball illumination light 166 when detecting the position of the user's eye based on the captured image. In addition, when the user's eyes are separated from the display unit 60 (see FIGS. 12 and 14), the processing unit 123 calculates the interval between the bright spots 604A and the bright spots 604B (see FIG. 12) in order to calculate the reduction ratio of the display area. 11) and the distance between the bright spot 604C and the bright spot 604D (see FIG. 12).
By reducing the display area S shown in FIG. 9 to the display area T by this process, the processing unit 123 can allow the user to look over the entire display. Further, when it is determined that the intermediate position between the bright spot 604E and the bright spot 604F (see FIG. 14) is shifted from the intermediate position between the bright spot 604A and the bright spot 604B, the processing unit 123 It can be seen that the user's eyes are shifted by the vector V1. In this way, the image switching unit 306 (see FIG. 22) of the processing unit 123 can shift the display area T in FIG. 9 to the display area U.

  In this embodiment, the example in which the password input screen shown in FIG. 35 is used for calibration for detecting a point of interest has been described, but the present invention is not limited to this. Keys, buttons, icons, etc. that can be selected by the operation switch 30 may be displayed on the display panel 63. Also in this case, unlike the prior art, keys, buttons, icons, and the like relating to operations that the user is required to perform are displayed, not just for calibration. Since the user has a very high probability of gazing at these keys, buttons, icons, etc. during operation, the user can perform calibration for gazing point detection based on the captured image at the time of operation.

[Second Embodiment]
In the second embodiment, the eyeball illumination light 166 is switched on and off for many positions of the display unit 60 including the position 600a and the position 600b of the display unit 60 (see FIG. 34 of the first embodiment). explain.

  FIG. 39 is a diagram showing a mounting form of the head mounted display (second display mode, save mode) according to the present embodiment. In FIG. 39, the apparatus main body 21 is in a posture inclined downward by an angle α with respect to an XY plane including a line virtually connecting the left and right eyes of the user.

  FIG. 39A is a diagram illustrating a mounting form in which the display unit 60 is rotated 90 degrees in the clockwise direction from the first display mode of the right-eye observation form. This mounting form is referred to as a second display mode. In the second display mode, the display unit 60 rotates clockwise about the Y ′ axis (see FIG. 3), with the storage unit 61 a and the spherical unit 61 b as rotation axes, so that the tip of the display unit 60 is located on the upper side. Rotated by 90 degrees. Thereby, the display part 60 and the finder opening part 67 are vertically arranged in the Z-axis direction when viewed from the user side. Further, the display panel 63 displays an image having an aspect ratio that is vertically long.

  That is, in the second display mode, the image displayed on the display panel 63 of the display unit 60 is displayed at a position in the diagonally forward right direction with respect to the user's right eye. In the second display mode, the eyeball illumination lights 166c and 166d in the vicinity of the right eye are turned on, and the eyeball illumination lights 166a and 166b are turned off.

  FIG. 39B is a diagram illustrating a mounting configuration in which the display unit 60 is rotated 90 degrees counterclockwise from the first display mode of the right-eye observation mode. This wearing form is referred to as a save mode. In the evacuation mode, the display unit 60 is rotated counterclockwise by 90 degrees around the Y ′ axis (see FIG. 3), with the accommodating portion 61 a and the spherical portion 61 b as rotation axes so that the tip of the display portion 60 is positioned on the lower side. It is rotated in the turning direction. Thereby, the display unit 60 and the viewfinder opening 67 are retracted from the visual field of the user's right eye. In the retreat mode, the eyeball illumination lights 166a to 166d may all be turned off.

  The rotation axis of the display unit 60 is common in the first display mode, the second display mode, and the save mode. For this reason, if the display part 60 is rotated 90 degree | times from 2nd display mode or save mode, it can return to 1st display mode. That is, the user can retract the display unit 60 from the visual field of the right eye of the user or return it to the observation position without removing the HMD 1 or changing the angle of the apparatus main body unit 21.

  In the example shown in FIGS. 39A and 39B, the example in which the display unit 60 is arranged vertically from the first display mode of the right-eye observation mode has been described, but FIG. Similarly, from the first display mode of the left-eye observation mode shown, the display unit 60 can be arranged vertically. That is, from the first display mode of the left-eye observation mode, the display unit 60 is rotated by 90 degrees in the counterclockwise direction on the XZ plane with the accommodating portion 61a and the spherical portion 61b as the central axes, so that the user can The HMD 1 can be switched to the second display mode of the eye observation form. Similarly, from the first display mode of the right-eye observation mode, the display unit 60 is rotated by 90 degrees in the clockwise direction around the Y ′ axis (see FIG. 3), with the accommodating portion 61a and the spherical portion 61b as the central axes. By doing so, the user can switch the HMD 1 to the save mode of the left-eye observation mode.

  Based on the detection value detected by the acceleration sensor 132 and the image captured by the image sensor 72 (see FIG. 22), the control unit 302 performs the right-eye observation mode or the right-eye observation mode, the first display mode, and the second display. The mode or the save mode is determined. Here, with respect to the image captured by the image sensor 72, the control unit 302, for example, the relative position between the eyebrows and the eye, the behavior of eyelids in blinking, the eyebrow contour, the eye contour, and the lowering of the left and right corners of the eye Based on the above, the right-eye observation form or the right-eye observation form and the first display mode, the second display mode, or the save mode are determined. In addition, when the user's eyeball is not captured in the image captured by the image sensor 72, the control unit 302 determines that the mode is the save mode.

  FIG. 40 is a diagram for explaining the relationship between the position of the display unit 60 and the eyeball illumination light 166 according to the present embodiment, and is a diagram when the user's face is viewed horizontally from the front. In FIG. 40, positions 600 a to 600 h indicated by alternate long and short dashed lines are examples of positions where the display unit 60 is arranged.

<First display mode>
The position 600a is a position of the display unit 60 that is disposed sideways below the right eye of the user. In addition, the position 600b is a position of the display unit 60 that is disposed sideways below the left eye of the user. The position 600c is a position of the display unit 60 that is disposed sideways above the right eye of the user. The position 600d is a position of the display unit 60 that is disposed sideways above the left eye of the user.

<Second display mode>
The position 600e is the position of the display unit 60 that is arranged vertically on the right side of the user's right eye. The position 600f is the position of the display unit 60 that is arranged vertically on the left side of the user's left eye.

  When the display unit 60 is located at any of the positions 600a, 600d, and 600f, the control unit 302 (see FIG. 22) of the HMD1 lights the eyeball illumination lights 166a and 166b located closer to the user's eyes at this position. The eyeball illumination lights 166c and 166d are turned off. On the other hand, when the display unit 60 is in any of the positions 600b, 600c, or 600e, the eyeball illumination lights 166c and 166d located closer to the user's eyes are turned on at this position, and the eyeball illumination lights 166a and 166b are turned off. .

  As described above, the control unit 302 irradiates the user's eyes from the front among the eyeball illumination lights 166a to 166d arranged in the vicinity of the corner of the finder opening 67 (within a predetermined distance). The lighting state is controlled according to the positional relationship between the user and the display unit 60 so that the light is turned on.

<Evacuation mode>
The position 600g is the position of the display unit 60 that is arranged vertically on the lower right side of the user's right eye. The position 600h is the position of the display unit 60 that is arranged vertically on the lower left side of the user's left eye.
When the display unit 60 is located at the positions 600g and 600h, the control unit 302 turns off all of the eyeball illumination lights 166a to 166d.

Next, a selection process for selecting an eyeball illumination light to be turned on will be described.
FIG. 41 is a flowchart of the selection processing procedure of the eyeball illumination light of the HMD 1 according to the present embodiment.

(Step S601) The control unit 302 turns on the eyeball illumination lights 166a and 166b via the eyeball illumination unit 304. Next, the control unit 302 drives the image sensor 72 to acquire a captured image, and stores the acquired captured image in the storage unit 303.

(Step S602) The control unit 302 turns off the eyeball illumination lights 166a and 166b and turns on the eyeball illumination lights 166c and 166d via the eyeball illumination unit 304. Next, the control unit 302 drives the image sensor 72 to acquire a captured image, and stores the acquired captured image in the storage unit 303.

(Step S603) The control unit 302 reads the captured image stored in step S601, and extracts bright spot images corresponding to the eyeball illumination lights 166a and 166b from the read captured image. Next, the control unit 302 calculates the luminance of the extracted bright spot image. Next, the control unit 302 reads the captured image stored in step S602, and extracts bright spot images corresponding to the eyeball illumination lights 166c and 166d from the read captured image. Next, the control unit 302 calculates the luminance of the extracted bright spot image.
Next, the control unit 302 determines whether the brightness of the bright spot corresponding to the eyeball illumination lights 166a and 166b is higher than the brightness of the bright spot corresponding to the eyeball illumination lights 166c and 166d. When it is determined that the brightness of the bright spot corresponding to the eyeball illumination lights 166a and 166b is higher than the brightness of the bright spot corresponding to the eyeball illumination lights 166c and 166d (step S603; Yes), the process proceeds to step S604. When it is determined that the brightness of the bright spot corresponding to the eyeball illumination lights 166a and 166b is not higher than the brightness of the bright spot corresponding to the eyeball illumination lights 166c and 166d (step S603; No), the process proceeds to step S605.

(Step S604) When it is determined that the brightness of the bright spot corresponding to the eyeball illumination lights 166a and 166b is higher than the brightness of the bright spot corresponding to the eyeball illumination lights 166c and 166d, the control unit 302 166a and 166b are selected, and the selected result is stored in the storage unit 303. The control unit 302 turns off the eyeball illumination lights 166c and 166d and ends the eyeball illumination light selection process.

(Step S605) When it is determined that the brightness of the bright spot corresponding to the eyeball illumination lights 166a and 166b is not higher than the brightness of the bright spot corresponding to the eyeball illumination lights 166c and 166d, the control unit 302 performs the eyeball illumination. The lights 166c and 166d are selected, and the selected result is stored in the storage unit 303. The control unit 302 turns off the eyeball illumination lights 166c and 166d and ends the eyeball illumination light selection process.

  As described above, since the eyeball illumination light 166 is controlled based on the luminance of the eyeball illumination light 166, the HMD 1 of the present embodiment changes the eyeball according to the positional relationship between the user's eye and the display unit 60. Selection of the light to illuminate can be performed appropriately. Further, the HMD 1 of the present embodiment can efficiently take measures for power saving by turning off the unused eyeball illumination light 166.

  In the present embodiment, the example in which turning on and off of the eyeball illumination light 166 is controlled according to the luminance of the eyeball illumination light 166 and the positional relationship between the user and the display unit 60 has been described. An example of controlling the turning on and off of light is not limited to this. For example, the control unit 302 compares the acquired captured image with the discrimination image stored in the storage unit 303 by pattern matching, and according to the positional relationship between the user and the display unit 60 based on the comparison result. Thus, the turning on and off of the eyeball illumination light 166 may be controlled.

  As described above, the HMD 1 of the present embodiment is in a positional relationship between the user and the display unit 60 even when the display unit 60 is disposed below, above or outside the user with respect to the user's eyes. In response, the turning on and off of the eyeball illumination light 166 is controlled. As a result, the HMD 1 of the present embodiment can detect the point of sight by irradiating the user's eyes with the light of the eyeball illumination light 166 according to the positional relationship between the user and the display unit 60. Further, the HMD 1 of the present embodiment can reduce the power consumption because the eyeball illumination light 166 can be turned off according to the positional relationship between the user and the display unit 60.

  Note that if the display unit 60 is in front of the user's face and all of the eyeball illumination lights 166a to 166d are determined to be at an approximately equidistant position from the user's horizontal line of sight, the HMD 1 is in principle which eyeball. The illumination light 166 may be turned on. However, normally, there are ceiling lighting, sky, sun, and the like above the eyes, and bright spots from these light sources may exist on the eyeball. Since these bright spots and the bright spot of the eyeball illumination light 166 are difficult to discriminate, when it is determined that the display unit 60 is substantially in front of the user's face, the HMD 1 Control may be performed to turn on the eyeball illumination light 166 arranged at.

[Third Embodiment]
In the first and second embodiments, the example in which the eyeball illumination lights 166a to 166d are provided in the vicinity of the long side of the finder opening 67 including the display unit 60 (see FIG. 2), the eyeball illumination lights 166a to 166d are provided. The example provided with is not limited to this. That is, the eyeball illumination light 166 may be disposed in the vicinity of the short side of the finder opening 67.

  FIG. 42 is a diagram illustrating the arrangement of the eyeball illumination lights 166e to 166h according to the present embodiment. As shown in FIG. 42, two eyeball illumination lights 166 e to 166 h are provided on the surface of the display unit 60 along two short sides of the finder opening 67. The eyeball illumination lights 166e to 166h are, for example, LEDs that emit infrared rays.

  FIG. 43 is a diagram for explaining the relationship between the position of the display unit 60 and the eyeball illumination lights 166e to 166h according to the present embodiment, and is a diagram of the user's face viewed from the horizontal front. In FIG. 43, squares 600a to 600h indicated by alternate long and short dash lines are examples of positions where the display unit 60 is arranged, as in FIG.

As shown in FIG. 43, when the display unit 60 is located at any one of the positions 600a, 600d, and 600f, the control unit 302 of the HMD1 lights the eyeball illumination lights 166e and 166f in the vicinity of the eyes, Turn off other eyeball illumination lights that are not nearby.
On the other hand, when the display unit 60 is located at any one of the positions 600b, 600c, or 600e, the control unit 302 turns on the eyeball illumination lights 166g and 166h in the vicinity of the eyes, and other eyeball illuminations that are not in the vicinity of the eyes. Turn off the light.
Furthermore, when the display unit 60 is at the position 600g or 600h, the control unit 302 turns off all the eyeball illumination lights 166.
That is, also in the present embodiment, the control unit 302 performs control so that the eyeball illumination light 166 that irradiates the user's eyes from the front is turned on according to the positional relationship between the user and the display unit 60. In other words, the control unit 302 turns on the illumination units (eyeball illumination lights 166e to 166h) arranged in the vicinity of the two opposing sides of the finder opening 67 according to the positional relationship between the user and the display unit 60. .

  The configuration of the processing unit 123 of this embodiment is the same as that of FIG. 23 of the first embodiment. In addition, the selection process of the eyeball illumination light 166 in the present embodiment is performed in the same manner as steps S601 to S604 illustrated in FIG. Alternatively, the acquired captured image and the discrimination image stored in the storage unit 303 are compared by pattern matching, and based on the comparison result, the eyeball illumination light is selected according to the positional relationship between the user and the display unit 60. The lighting and extinguishing of 166 may be controlled.

  As described above, in this embodiment, two eyeball illumination lights 166e to 166h are provided in the vicinity of the short side of the finder opening 67 of the display unit 60 for each short side. Also in this case, similarly to the second embodiment, the light of the eyeball illumination lights 166e to 166h is switched on and off according to the positional relationship between the user and the display unit 60, so that the light rays are appropriately applied to the region including the user's eyes. Can be irradiated. As a result, it is possible to accurately detect the user's line of sight for calibration of the line of sight input and the line of sight input, and thus it is possible to accurately detect the user's line of sight, that is, the gaze point.

[Fourth Embodiment]
Next, the case where the display unit is a see-through optical system will be described.
The difference from the first to third embodiments is only the configuration of the display unit 60.
FIG. 44 is a cross-sectional view of the display unit 60a for explaining the optical system according to the present embodiment. The members having the same functions as those in FIG.

  As shown in FIG. 44, the display unit 60a of the present embodiment includes a backlight 62, a display panel 63, a first prism 65a, a second prism 65b, a reflection mirror 66, a viewfinder opening 67, an imaging lens 71, and an image sensor. 72, a quarter-wave plate 75, a polarizing plate 90, a concave lens 91, a quarter-wave plate 92, a liquid crystal 93, a polarizing plate 94, and an opening 95. In FIG. 44, symbols H, I, J, K, O, and T each represent a light beam, and symbol S represents a light beam emitted from the user's eye.

As shown in FIG. 44, the scene in front of the user becomes a light ray O and enters the opening 95. Next, the light beam O passes through the opening 95 and passes through the polarizing plate 94. Next, the light beam O transmits only S-polarized light.
Next, the light beam O enters the liquid crystal 93. Note that the polarization direction of the light beam O is changed by the action of the liquid crystal 93. That is, when the liquid crystal 93 is energized (ON state), the light O is incident on the quarter-wave plate 92 with the S-polarized light incident on the liquid crystal 93 becoming P-polarized light. Next, the light beam O becomes counterclockwise polarized light by the action of the quarter-wave plate 92.

In the present embodiment, the reflection mirror 66 is a semi-transmissive type, and transmits, for example, 50% of light and reflects 50% of light. The following description will be made on the assumption that the reflection mirror 66 transmits 50% of light. As a result, 50% of the light beam O enters the quarter-wave plate 75.
The concave lens 91 is a member that is paired with a convex lens constituting the reflection mirror 66 to become a parallel plate with zero power.

Next, the light beam O becomes P-polarized light by the action of the quarter-wave plate 75 and enters the first prism 65a. Next, the light beam O passes through the bonding surface 65 c as it is, passes through the second prism 65 b, and enters the polarizing plate 90. The polarizing plate 90 is configured to transmit P-polarized light.
With the above configuration, an image of the outside world (an image viewed through the opening 95) can be observed together with an image projected from the display panel 63 by the user.

Next, a case where the image of the outside world is erased and only the image projected from the display panel 63 is observed will be described.
In FIG. 44, when the liquid crystal 93 is turned off (off state), the polarization direction of the light beam O is not changed by the liquid crystal 93. For this reason, the light beam O becomes S-polarized light at the polarizing plate 94. Next, the light beam O is incident on the quarter-wave plate 92 as it is. Next, the light beam O becomes clockwise polarized light by the action of the quarter wave plate.

  Next, 50% of the light beam O transmitted through the reflection mirror 66 enters the quarter-wave plate 75. Next, the light beam O becomes S-polarized light by the action of the quarter-wave plate 75. Next, the light beam O that has become S-polarized light enters the first prism 65a. The polarization reflection film provided on the bonding surface 65c reflects S-polarized light. As a result, external light rays do not enter the user's eyes.

  In this way, by switching the liquid crystal 93 between the on state and the off state, it is possible to control whether or not the image of the outside world can be transmitted through the finder and observed by the user. Note that the processing unit 123 performs switching between the ON state and the OFF state of the liquid crystal 93, for example.

44, the action of the polarizing plate 90 is arranged so that the user does not enlarge and observe his / her eyes by the reflection mirror 66. In FIG. Explain why users do not observe their eyes enlarged.
The light beam S emitted from the user's eyes is transmitted through only the P-polarized light by the polarizing plate 90, passes through the second prism 65b, the joint surface 65c, and the first prism 65a, and enters the quarter-wave plate 75. Here, the light beam S becomes counterclockwise circularly polarized light by the action of the quarter-wave plate 75, and is reflected as the light beam T by the reflection mirror 66. Next, the light beam T becomes clockwise circularly polarized light by reflection, and becomes S-polarized light when passing through the quarter-wave plate 75.
Next, the light beam T passes through the first prism 65a and enters the joint surface 65c. Here, the joint surface 65c reflects and does not transmit S-polarized light. Therefore, the user's eyes do not receive the luminous flux from his / her own eyes, and his / her eyes cannot be seen on the display screen.

  Further, the imaging lens 71 and the image sensor 72 for taking a peripheral image of the eye are provided adjacent to the finder opening 67. For this reason, the light rays of the imaging lens 71 and the imaging element 72 system can directly photograph the vicinity of the user's eyes without passing through the second prism 65b.

FIG. 45 is a diagram for explaining an observation range and an imaging range according to the present embodiment. In FIG. 45, reference numeral 67a represents the observation range of the image projected from the display panel 63. Reference numeral 67 b represents an imaging range of the imaging element 72.
FIG. 45 is an equivalent view in which the reflecting mirror 66 is replaced with a convex lens, as in FIG.

In FIG. 45, the focal length of the imaging lens 71 is made shorter than the focal length of the reflection mirror 66. Therefore, when FIG. 44 is cut in a cross section perpendicular to the paper surface, as shown in FIG. 45, the surrounding area including the user's eyes can be photographed.
In FIG. 45, the imaging lens 71 and the image sensor 72 are offset from the optical center of the reflection mirror 66. However, the user can sufficiently separate his / her eyes from the viewfinder opening 67 by lowering the display unit 60a in the direction of the mouth, not in front of the eyes. As shown in FIG. It is at the position of the observation distance L81. For this reason, when the user is observing the display panel 63, according to this embodiment, the image sensor 72 can photograph the user's eyes and the surroundings without any problem.

By providing the above-described configuration, imaging including the user's eyes is performed, and the wearing state of the HMD 1 and the line-of-sight calibration are performed based on the image captured by the imaging device 72 as in the first to third embodiments. It is possible to provide a head mounted display including a display unit of a see-through optical system capable of
In the fourth embodiment, as described in the first and third embodiments, the eyeball illumination light is arranged in the vicinity of two opposite sides of the four sides of the finder opening 67. Also good.

  In the first to fourth embodiments, the example of the head mounted display that can be used by switching between the right eye observation mode and the left eye observation mode has been described as an example of the head mounted display, but is not limited thereto. For example, it may be a head-mounted display dedicated to the right-eye observation form or dedicated to the left-eye observation form. In this case, among the processes described in the present embodiment, only the process relating to the right eye observation form or the left eye observation form may be performed.

  In the first to fourth embodiments, FIGS. 2 and 42 show an example in which the display unit 60 and the finder opening 67 are rectangular. However, the display unit 60 and the finder opening 67 are square and elliptical. It may be circular or the like. Even if the display unit 60 and the viewfinder opening 67 have such a shape, a plurality of eyeball illumination lights are opposed to two of the four sides of the viewfinder opening 67 as shown in FIG. You may make it arrange | position in the vicinity of an edge | side, respectively.

In the first to fourth embodiments, two eyeball illumination lights 166a to 166d or 166e to 166h are provided on each of the upper and lower sides or the left and right sides of the viewfinder opening 67 as shown in FIG. 2 or FIG. Although the example which provides is demonstrated, it is not restricted to this. For example, one each may be provided above and below or right and left of the finder opening 67, or three or more may be provided above and below or right and left of the finder opening 67.
Further, the positions where the eyeball illumination lights 166a to 166d are provided may not be above and below the finder opening 67 as shown in FIG. Any position that irradiates the user's eyeball may be used. For example, the eyeball illumination lights 166 a to 166 d may be provided above and below the viewfinder opening 67. In other words, when the user places the HMD 1 in the vicinity of the user's eyes, the eyeball illumination lights 166a to 166d may be arranged in the vicinity of the user's eyes. For this reason, the eyeball illumination lights 166a to 166d may be arranged in a range of one surface of the display unit 60 where the finder opening 67 is arranged, or a range inside the finder opening 67. Further, the vicinity of the user's eye where the display unit 60 is arranged is, for example, below, above, outside, or the like of the user's eye, such as positions 600a to 600f shown in FIG. 72 may be a range in which a peripheral image of the user's eyes can be captured.

In the first to fourth embodiments, the example in which the detection value of the acceleration sensor 132 or the image captured by the image sensor 72 is used to determine whether the observation mode is the right eye observation mode or the left eye observation mode is described. It is not limited to this. For example, a type including a light receiving unit may be used for the eyeball illumination lights 166a to 166d, and the reflected light from each of the eyeball illumination lights 166a to 166d may be detected. For example, in the case of the right-eye observation mode, as shown in FIG. 33, the eyeball illumination lights 166a and 166b are arranged in front of the user's eye rather than the eyeball illumination lights 166c and 166d. For this reason, the brightness of the reflected light from the eyeball illumination lights 166c and 166d is lower than the brightness of the reflected light from the eyeball illumination lights 166a and 166b.
As described above, the processing unit 123 may detect the reflected light from each of the eyeball illumination lights 166a to 166d and determine the observation form based on the detected result.

The image sensor 72 may capture a captured image at a predetermined time interval. In this case, the processing unit 123 determines whether the user has removed the head-mounted display from the head without turning the main switch 28 based on the image captured by the image sensor 72. Also good. If it is determined that the user has removed the head-mounted display from the head for a predetermined time or longer without turning off the main switch 28, the processing unit 123 stops imaging by the camera 64 and the imaging element 72. Also good. The processing unit 123 may turn off the eyeball illumination lights 166a to 166d. Further, the processing unit 123 may control each communication circuit or the like so as to enter a power saving state. Whether or not to perform these controls may be set by the user by operating the touch switch 34, for example.
Alternatively, the processing unit 123 may determine whether or not the head mounted display is removed from the head based on the detection value of the heart rate sensor 137, for example.
Further, when a detection switch (not shown) detects that the apparatus main body 21 is not attached to the headband 40, the processing unit 123 determines that the apparatus main body 21 is removed from the head, and the eyeball All of the illumination lights 166a to 166d may be turned off.
Thereby, the process part 123 can reduce the power consumption of a head mounted display.

  In the first to fourth embodiments, the monocular type has been described as an example of the head mounted display. However, the head mounted display may be a binocular type. In this case, based on the image captured by the image sensor 72, the processing unit 123 may be in a state where the display unit is placed at the positions 600c and 600d on the eye as illustrated in FIG. It may be determined whether or not the display unit is located at the positions 600a and 600b. The processing unit 123 may control lighting or extinguishing of the eyeball illumination lights 166 provided in the display units corresponding to the left and right eyes, based on the determined result.

  Further, when the head mounted display is a binocular type, the eyeball illumination light 166 and the image sensor 72 may be provided only on one display unit 60 of the left and right display units, or on both display units 60. It may be provided. The reason why the eyeball illumination light 166 may be provided only on one display unit 60 is that, when a person looks at an object, the eyeballs of both eyes generally face the same direction.

  In the first to fourth embodiments, the example in which the HMD 1 includes the functions illustrated in FIG. 21 has been described. However, all the functions may not be included. Depending on the application, for example, the temperature / humidity sensor 136 and the heart rate sensor 137 may not be provided. Conversely, the HMD 1 may include other functional units other than those shown in FIG. 22 as necessary.

  In the first to fourth embodiments, the example in which the display main body 20 is mounted on the headband 40 has been described. However, the present invention is not limited to this. The user may use the display body 20 alone without depending on the headband 40 depending on the application.

  The technical scope of the present invention is not limited to the above embodiment. At least one of the requirements described in the above embodiments may be omitted. The requirements described in the above embodiments can be combined as appropriate.

  The HMD 1 described above has a computer system (for example, the processing unit 123) inside. The operation process of each functional unit is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer system reading and executing the program. Here, the computer system includes a CPU, various memories, an OS, and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, it also includes those that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in that case. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

DESCRIPTION OF SYMBOLS 1 ... Head mounted display, 20 ... Display main body, 26 ... Audio connector, 34 ... Touch switch, 60 ... Display part, 63 ... Display panel (display part), 64 ... Camera, 72 ... Imaging element (imaging part), 100 ... Earphone, 103 ... first speaker, 104 ... second speaker, 121 ... decoder, 123 ... processing unit, 131 ... WiFi communication circuit, 132 ... acceleration sensor, 133 ... geomagnetic sensor, 137 ... heart rate sensor, 138 ... 3G / LTE Communication circuit, 166a to 166h ... eyeball illumination light, 301 ... input detection unit, 302 ... control unit, 303 ... storage unit, 304 ... eyeball illumination unit, 305 ... sound adjustment unit, 306 ... image switching unit, 307 ... image display unit 308: Transmission / reception unit, 600a to 600h: Position

Claims (12)

A display for displaying an image;
A holding unit that holds the display unit in the vicinity of the user's eyes;
A plurality of illumination units that are provided in the display unit and illuminate the vicinity of the eyes of the user;
In accordance with the positional relationship between the user and the display unit, a control unit that controls the lighting state of the plurality of illumination units,
A head-mounted information input / output device comprising: The controller is
The head-mounted information input / output device according to claim 1, wherein a lighting state is controlled so that an illumination unit that illuminates the user's eyes from the front is turned on among the plurality of illumination units. The plurality of illumination units are:
The head-mounted information input / output device according to claim 1, wherein the head-mounted information input / output device is disposed near a finder opening of the display unit. The plurality of illumination units are:
The head-mounted information input / output device according to any one of claims 1 to 3, wherein the head-mounted information input / output device is disposed in the vicinity of a corner of the finder opening. The controller is
The head mounted according to any one of claims 1 to 4, wherein an image is rotated or inverted and displayed on the display unit according to a positional relationship between the user and the display unit. Type information input / output device. The control unit turns on the plurality of illumination units according to whether the display unit is held on the left side, the right side, the upper side, or the lower side for the user as a positional relationship between the user and the display unit. The head-mounted information input / output device according to any one of claims 1 to 5, wherein the state is controlled. It has an acceleration sensor that detects the direction of gravity applied to its own device,
The controller is
The head-mounted information according to any one of claims 1 to 6, wherein a positional relationship between the user and the display unit is determined based on a gravitational direction detected by the acceleration sensor. I / O device. It has an imaging unit that images the area including the user's eyes,
The controller is
The head-mounted information input according to any one of claims 1 to 7, wherein a positional relationship between the user and the display unit is determined based on an image obtained by imaging the area. Output device. It has an imaging unit that images the area including the user's eyes,
The controller is
The illumination unit detects reflected light of light irradiated to the user's eye based on an image obtained by imaging the region, and detects a line of sight of the user's eye based on the detection result. The head-mounted information input / output device according to claim 1. It has an acceleration sensor that detects the direction of gravity applied to its own device,
The controller is
The positional relationship between the user and the display unit is determined based on at least one of an image obtained by capturing the region, the gravity direction, and information set by the user. Head-mounted information input / output device. The controller is
The control unit is configured to turn on the plurality of illumination units when acquiring parameters for operating information displayed on the display unit based on the line of sight of the user. The head-mounted information input / output device according to any one of the above. A head-mounted information input / output method in a head-mounted information input / output device,
A plurality of illumination units provided in a display unit for displaying an image illuminates the vicinity of the user's eyes;
A control procedure for controlling a lighting state of the plurality of illumination units based on a positional relationship between the user and the display unit;
A head-mounted information input / output method comprising:

Images