JP6303274B2 - Head-mounted display device and method for controlling head-mounted display device - Google Patents

Description

  The present invention relates to a head-mounted display device.

  A head-mounted display device (Head Mounted Display, HMD) that is a display device mounted on the head is known. The head-mounted display device, for example, generates image light representing an image using a liquid crystal display and a light source, and guides the generated image light to a user's eye using a projection optical system or a light guide plate This causes the user to visually recognize the virtual image. There are two types of head-mounted display devices: a transmission type in which the user can visually recognize the outside scene in addition to a virtual image, and a non-transmission type in which the user cannot visually recognize the outside scene. The transmissive head-mounted display device includes an optical transmissive type and a video transmissive type. As an input device for operating the head-mounted display device, for example, a keyboard, a mouse, a track pad, and the like are known.

  In Patent Document 1, in a head-mounted display device in which an image of content or the like is visually recognized by one eye of a user, the eye movement of one eye is imaged and the eyelid movement of the other eye is imaged. A technique for controlling an image to be visually recognized by the user based on the captured eye movement and eyelid movement is disclosed. Further, in Patent Document 2, in a head-mounted display device, an image of a user interface such as a keyboard is visually recognized by a user on a plane within a predetermined distance from the user, and a user input operation to the user interface is performed. A technique for operating a head-mounted display device by detecting the above is disclosed. Japanese Patent Application Laid-Open No. 2003-228561 discloses a technique for allowing a user to visually recognize a keyboard, which is an input device for a head-mounted display device, and keyboard keys touched by the user as images.

JP 2010-152443 A JP 2002-318652 A JP 2003-91353 A

  However, in the head-mounted display device described in Patent Document 1, an image to be visually recognized by the user is controlled based on the eye movement of the user's eye. There has been a desire to perform operations other than image control in the display device. Further, there is a demand for operating the head-mounted display device not only by eye movement of one eye but also by various movements including eye movement of both eyes. Moreover, in the technique described in Patent Document 2, if there is no plane within a predetermined distance from the user, the user interface image may not be visible to the user, and there is room for improvement in convenience. was there. Further, in the technique described in Patent Document 3, the user can visually recognize a key touching a keyboard that is an input device, but the user does not need an input device such as a keyboard without using the hand. There was a desire to operate a head-mounted device. In addition, the above-mentioned subject was a subject common to the display apparatus which can make a user visually recognize not only a head-mounted display apparatus but an input device.

SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms. As one aspect of the present invention, a head-mounted display device is provided. The head-mounted display device includes a gaze direction detection unit that detects the gaze direction of the user, and allows a plurality of selectable objects to be detected while transmitting the outside scene while being mounted on the user's head. An image display unit that displays a control image including a pointer indicating a position corresponding to the line-of-sight direction, which is a part of an area where the image can be displayed and overlaps the viewing direction of the user; And a control unit that controls contents according to an object that can be selected by a positional relationship with the pointer among the plurality of objects.

(1) According to one aspect of the present invention, a transmissive head-mounted display device is provided. The head-mounted display device includes a line-of-sight detection unit that detects a user's line-of-sight direction; a plurality of selectable objects in a state of being mounted on the user's head; and the detected line-of-sight direction A pointer that indicates a corresponding position; and an image display unit that allows a user to visually recognize a control image and that transmits an outside scene; a plurality of the object and the pointer with respect to the head-mounted display device; A line-of-sight input control unit that controls contents according to the positional relationship. According to the head-mounted display device of this aspect, various controls of the head-mounted display device can be performed based on the user's line-of-sight direction without the user using the hand. Improves. Further, even if there is no user interface such as a keyboard having a shape as an object, the user can control the head-mounted display device, and the portability of the head-mounted display device is improved.

(2) According to the head-mounted display device of the above aspect, further comprising: a detection unit that is located at a predetermined distance from the user's eyes and detects a shielding object in the line-of-sight direction; the line-of-sight input control unit As the control, one object may be selected based on the positional relationship, and the selected object may be determined based on the detection result of the shielding object. According to the head-mounted display device of this form, the head-mounted display device is controlled based on the user's line-of-sight direction and the shielding object present in the line-of-sight direction, so the user does not use the hand at all. In addition, the head-mounted display device can be controlled, improving the operability for the user.

(3) According to the head-mounted display device of the above aspect, the detection unit detects the state of the user's eyelid as the shield; the line-of-sight input control unit detects the state of the eyelid detected The contents may be controlled according to the positional relationship. According to the head-mounted display device of this embodiment, the head-mounted display device is controlled based on the user's line-of-sight direction and the open / closed state of both eyelids, so that the user can use his / her head without using any hand. The wearable display device can be controlled, improving the operability for the user.

(4) According to the head-mounted display device of the above aspect, the detection unit detects the open / closed state of the right eyelid and the open / closed state of the left eyelid of the user; The content may be controlled according to the detected combination of the opened / closed state of the right eyelid and the opened / closed state of the left eyelid. According to the head-mounted display device of this embodiment, since the head-mounted display device is controlled based on the combination of the open / closed states of both eyelids, many combinations of decisions corresponding to the type of operation are used. The user operability is improved.

(5) According to the head-mounted display device of the above aspect, the detection unit is a right that indicates at least one of color and illuminance in a range from each of the user's right eye and left eye to the predetermined distance. An eye-front index value and a left-eye front index value are detected; and the line-of-sight input control unit performs control of contents according to the positional relationship and the detected right-eye front index value and the left-eye front index value. Also good. According to the head-mounted display device of this form, when the user is tired from the operation due to the state of the eyelid, the user can perform a sensory operation using a part of the body such as the palm, Improved convenience.

(6) According to the head-mounted display device of the above embodiment, the image display section, an area for generating the control image image, based on a detection result of said shield and said detected visual axis direction It may be set. According to the head-mounted display device of this embodiment, the position where the control image image based on the user's intention is generated, since the user can set without using hands, the user's operability and Convenience is improved.

(7) According to the head-mounted display device of the above aspect, an imaging unit that images an outside scene; a specific target that is included in the imaged outside scene and that is located at a position separated from the predetermined distance recognizing the image recognition unit and the provided; the image display unit, it said recognized based on a particular target, may be set region for generating the control images. According to the head-mounted display device of this embodiment, since the area where the image recognition unit control picture image based on the particular target recognized is generated is set, the user can intuitively control image The size of the image can be set, and the convenience for the user is improved. Moreover, the generation presence of the control picture image, since the user need only place a particular subject in a predetermined distance, thereby improving convenience for the user.

(8) According to the head-mounted display device of the above aspect, further comprising: a direction detection unit that detects a direction of the user's head; and the image display unit is based on the detected direction of the head. Te may be set an area for generating the control images. According to this form of the head-mounted display device, it is detected that the user has changed the orientation of the head, and the target that the user wants to visually recognize has been changed. since the control for image image so is generated, thereby improving convenience for the user.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with another new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including one or more or all of the three elements of the gaze direction detection unit, the image display unit, and the gaze input control unit. That is, this apparatus may or may not have a line-of-sight direction detection unit. Further, the apparatus may or may not have an image display unit. The device may or may not have a line-of-sight input control unit. The line-of-sight direction detection unit may detect the line-of-sight direction of the user, for example. The image display unit, for example, a control image including a plurality of selectable objects and a pointer indicating a position corresponding to the detected line-of-sight direction in a state of being mounted on the user's head. It is also possible to make the outside view transparent. The line-of-sight input control unit may control the content corresponding to the positional relationship between the plurality of objects and the pointer, for example, with respect to the head-mounted display device. Such a device can be realized as, for example, a head-mounted display device, but can also be realized as a device other than the head-mounted display device. According to such a form, it is possible to solve at least one of various problems such as improvement and simplification of the operability of the device, integration of the device, and improvement of convenience of the user who uses the device. it can. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device.

  The present invention can also be realized in various forms other than the head-mounted display device. For example, display device, head-mounted display device and display device control method, head-mounted display system, display device, head-mounted display system, and computer program for realizing the functions of the display device, and the computer program Can be realized in the form of a data signal or the like embodied in a carrier wave including the computer program.

2 is an explanatory diagram showing an external configuration of a head mounted display device 100. FIG. 3 is a block diagram functionally showing the configuration of the head-mounted display device 100. FIG. It is explanatory drawing which shows a mode that image light is inject | emitted by the image light production | generation part. It is explanatory drawing which shows the flow of a gaze operation input process. It is explanatory drawing which shows the flow of a gaze operation input process. It is explanatory drawing which shows an example of the control image CI before a magnitude | size is adjusted. It is explanatory drawing which shows an example of the control image CI after the magnitude | size was adjusted. 6 is an explanatory diagram illustrating an example of a control image CIa displayed on the image display unit 20. FIG. 6 is an explanatory diagram illustrating an example of a control image CIb displayed on the image display unit 20. FIG.

Next, embodiments of the present invention will be described in the following order based on the embodiments.
A. Embodiment:
A-1. Configuration of head mounted display device:
A-2. Line-of-sight operation input processing:
B. Variations:

A-1. Configuration of head mounted display device:
FIG. 1 is an explanatory diagram showing an external configuration of the head-mounted display device 100. The head-mounted display device 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The head-mounted display device 100 of the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. In this specification, a virtual image visually recognized by the user with the head-mounted display device 100 is also referred to as a “display image” for convenience. Moreover, emitting image light generated based on image data is also referred to as “displaying an image”.

  The head-mounted display device 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when mounted on the user's head, and a control unit 10 (controller 10) that controls the image display unit 20. I have.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and an outside scene imaging. A camera 61, a distance sensor 63, a right eye imaging camera 37, a left eye imaging camera 38, a right eye illuminance sensor 45, and a left eye illuminance sensor 47 are included. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

  The display drive units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 (see FIG. 3). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 2) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display driving units 22 and 24 to the eyes of the user. The light control plate is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. In addition, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The outside scene imaging camera 61 is disposed at a position corresponding to the eyebrow of the user when the user wears the image display unit 20. The outside scene imaging camera 61 captures an outside scene, which is an external scenery, and acquires an outside scene image. The outside scene imaging camera 61 in the present embodiment is a monocular camera, but may be a stereo camera. The outside scene imaging camera 61 corresponds to the imaging unit in the claims.

  The distance sensor 63 is disposed at a position corresponding to the user's eyebrow when the user wears the image display unit 20. The distance sensor 63 is an infrared distance sensor that emits infrared light, receives infrared light reflected by the object, and measures the distance between the distance sensor 63 and the object based on the light receiving position of the reflected light. The distance sensor 63 used in the present embodiment detects an object at a predetermined distance from the distance sensor 63 by measuring the distance between the distance sensor 63 and the object at a specific position. In another embodiment, the distance sensor 63 may be an infrared distance sensor that includes a plurality of light emitting units and measures the size and position of an object. The distance sensor 63 may be an optical sensor or an ultrasonic sensor instead of an infrared sensor. The distance sensor 63 corresponds to the image recognition unit in the claims.

  The right-eye imaging camera 37 and the left-eye imaging camera 38 (hereinafter also referred to as “eye imaging cameras 37 and 38”) are small CCD cameras that respectively capture the right eye and the left eye of the user. The right eye illuminance sensor 45 and the left eye illuminance sensor 47 (hereinafter also referred to as “illuminance sensors 45, 47”) are sensors that detect the illuminance of the outside scene visually recognized by the right eye and the left eye of the user. The eye imaging cameras 37 and 38 correspond to a detection unit and a line-of-sight direction detection unit in claims. The illuminance sensors 45 and 47 correspond to detection units in the claims, and the illuminance detected by the right eye illuminance sensor 45 and the illuminance detected by the left eye illuminance sensor 47 are respectively the right-eye front index value and the left Corresponds to the anterior index value.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 that are connected to the control unit 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24. The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. A connector (not shown) that fits each other is provided at each of the end of the main body cord 48 opposite to the connecting member 46 and the control unit 10. By fitting / releasing the connector of the main body cord 48 and the connector of the control unit 10, the control unit 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for controlling the head-mounted display device 100. The control unit 10 includes a determination key 11, a lighting unit 12, a display switching key 13, a track pad 14, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. Yes. The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control unit 10. The lighting unit 12 notifies the operation state of the head-mounted display device 100 by its light emission state. Examples of the operating state of the head-mounted display device 100 include power ON / OFF. For example, an LED (Light Emitting Diode) is used as the lighting unit 12. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content video between 3D and 2D, for example. The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the head-mounted display device 100 by detecting a slide operation of the switch.

  FIG. 2 is a block diagram functionally showing the configuration of the head-mounted display device 100. As shown in FIG. 2, the control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power source 130, a wireless communication unit 132, an operation unit 135, a CPU 140, an interface 180, and a transmission unit 51 ( Tx51) and a transmission unit 52 (Tx52). The operation unit 135 receives an operation by the user and includes an enter key 11, a display switch key 13, a track pad 14, a luminance switch key 15, a direction key 16, a menu key 17, and a power switch 18.

  The input information acquisition unit 110 acquires a signal corresponding to an operation input by the user. As a signal corresponding to the operation input, for example, there is an operation input to the track pad 14, the direction key 16, and the power switch 18 arranged in the operation unit 135. The power supply 130 supplies power to each part of the head-mounted display device 100. As the power supply 130, for example, a secondary battery can be used. The storage unit 120 stores various computer programs. The storage unit 120 is configured by a ROM, a RAM, or the like. The wireless communication unit 132 performs wireless communication with other devices such as a content server, a television, and a personal computer in accordance with a predetermined wireless communication standard such as a wireless LAN or Bluetooth. The CPU 140 reads and executes the computer program stored in the storage unit 120, thereby operating the operating system 150 (OS 150), the display control unit 190, the sound processing unit 170, the direction determination unit 161, the illuminance processing unit 145, and the image determination. Unit 142, line-of-sight input unit 175, and image processing unit 160.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 controls driving of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD control unit according to control signals. The left LCD 242 driving ON / OFF by 212, the left backlight 222 driving ON / OFF by the left backlight control unit 202, and the like are individually controlled. Thus, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light.

  The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The direction determination unit 161 is an angle formed by the inclination of the user's head estimated based on the orientation of the image display unit 20 detected by the 9-axis sensor 66 described later, and the horizontal direction perpendicular to the gravity direction and the gravity direction. Is calculated. The direction determination unit 161 and the 9-axis sensor 66 correspond to a direction detection unit in the claims.

  The illuminance processing unit 145 performs various calculations on the illuminance of the outside scene visually recognized by the right and left eyes of the user detected by the illuminance sensors 45 and 47. The illuminance processing unit 145 according to the present embodiment calculates the difference between the illuminance of the right-eye outside scene and the illuminance of the left-eye outside scene, and the change in illuminance detected over a certain period. The illuminance processing unit 145 determines whether the calculated illuminance difference and the illuminance difference before and after the illuminance change are equal to or greater than a preset threshold value. For example, when the user covers only one of the right optical image display unit 26 or the left optical image display unit 28 with his / her hand or the like under a fine daytime, the illuminance processing unit 145 displays the outside scene of the right eye. It is determined that there is a difference equal to or greater than a threshold value between the illuminance of the left eye and the illuminance of the left-eye outside scene. The illuminance processing unit 145 corresponds to the detection unit in the claims.

  The image determination unit 142 determines the opening / closing state of the eyelids in each of the right eye and the left eye by analyzing the images of the right eye and the left eye of the user captured by each of the eye imaging cameras 37 and 38. Detecting the direction of gaze. The image determination unit 142 performs pattern matching of the open / closed state of the eyelid on each of the right eye and the left eye of the user imaged by the eye imaging cameras 37 and 38, so that the eyelid is closed or the eyelid is closed. A thin state that is half closed is determined. Further, the image determination unit 142 performs pattern matching on the object detected by the distance sensor 63 as to whether or not the object is the same as the set object registered in advance. The image determination unit 142 corresponds to the gaze direction detection unit and the image recognition unit in the claims.

The line-of-sight input unit 175 receives the image display unit 20 from the user based on the difference in illuminance calculated by the illuminance processing unit 145 and the change in illuminance, the open / closed state of the eyelid determined by the image determination unit 142, and the like. A control image that accepts input is displayed. The control image displayed on the image display unit 20 includes a plurality of selectable buttons and a position image representing a position corresponding to the detected line-of-sight direction. The line-of-sight input unit 175 controls the head-mounted display device 100 and various devices connected to the head-mounted display device 100 based on the open / closed state of the eyelid, the line-of-sight direction, and the like. The device connected to the head-mounted display device 100 is not limited to a device directly connected by a cord or the like, but also includes a device that communicates via the wireless communication unit 132. Line-of-sight input unit 175 is equivalent to that control section put in a claim.

  The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). Thereafter, the image processing unit 160 stores the converted digital image signal as image data Data (RGB data) of the target image in the DRAM in the storage unit 120 for each frame. Note that the image processing unit 160 may execute image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data as necessary. .

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control unit 10 and the image display unit 20.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker (not shown) connected to the connecting member 46 ( (Not shown). For example, when the Dolby (registered trademark) system is adopted, processing on the audio signal is performed, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, for example.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and an outside scene imaging. The camera 61, the distance sensor 63, the 9-axis sensor 66, the right eye imaging camera 37, the left eye imaging camera 38, the right eye illuminance sensor 45, and the left eye illuminance sensor 47 are provided.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of the user's head is detected. The orientation of the image display unit 20 is specified from the detected movement of the user's head, and the direction determination unit 161 estimates the orientation of the user's head.

  The right display driving unit 22 includes a receiving unit 53 (Rx53), a right backlight control unit 201 (right BL control unit 201) and a right backlight 221 (right BL221) that function as a light source, and a right LCD that functions as a display element. A control unit 211, a right LCD 241 and a right projection optical system 251 are included. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit 54 (Rx54), a left backlight control unit 202 (left BL control unit 202) and a left backlight 222 (left BL202) that function as a light source, and a left LCD that functions as a display element. A control unit 212 and a left LCD 242 and a left projection optical system 252 are included. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as “image light generation unit”. The left projection optical system 252 is configured by a collimating lens that converts the image light emitted from the left LCD 242 into a light beam in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

  FIG. 3 is an explanatory diagram illustrating a state in which image light is emitted by the image light generation unit. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby changing the illumination light IL emitted from the right backlight 221 into an image. Is modulated into an effective image light PL representing The same applies to the left side. As shown in FIG. 3, the backlight system is adopted in the present embodiment, but the image light may be emitted using a front light system or a reflection system.

A-2. Line-of-sight operation input processing:
4 and 5 are explanatory diagrams showing the flow of the line-of-sight operation input process. In the line-of-sight operation input process, when a predetermined state of the eyelid of the user is detected, the display mode of the image display unit 20 shifts to the line-of-sight input mode, and the user opens and closes the eyelid without using the hand. Control of the head-mounted display device 100 and the like can be performed based on the state. First, the eye imaging cameras 37 and 38 monitor whether a predetermined state of the user's eyelid is detected during a predetermined period (step S310). In the present embodiment, the predetermined eyelid state is a combination of a state in which the right eye, which is the dominant eye of the user, is opened, and a state in which the eyelid of the left eye, which is not the dominant eye, is closed. In general, the dominant eye can easily see the outside scene compared to the other non-dominant eye, and the other non-dominant eye can easily close the eyelid compared to the dominant eye. For this reason, in the present embodiment, a predetermined combination of eyelid states is convenient for the user. The other eye, which is dominant and non-dominant, may be preset by the user, and in other embodiments, the combination of eyelid states is not limited to the dominant eye, and the right eyelid and the left eyelid It may be a combination of states.

  In the process of step S310, when a predetermined eyelid state is not detected for a predetermined period (step S310: NO), the eye imaging cameras 37 and 38 continue to monitor the eyelid state (step S310). . When a predetermined eyelid state is detected in a predetermined period (step S310: YES), the image display unit 20 shifts the display mode to the line-of-sight input mode. (Step S320). In the line-of-sight input mode, the image display unit 20 displays a control image. The line-of-sight input unit 175 includes the head-mounted display device 100 and the head-mounted display based on the control image including the position image indicating the position corresponding to the user's line-of-sight direction and the open / closed state of the user's eyelid. Various devices connected to the mold display device 100 are controlled. In addition, in the display modes of the image display unit 20 described below, including the line-of-sight input mode, after changing to each display mode, the user's eyelids may be changed to a normal state that is open. The display mode is not changed. The image display unit 20 shifts to another display mode only when a specific eyelid state that changes the display mode is detected.

  Next, the distance sensor 63 monitors the detection of a specific object at a predetermined distance from the distance sensor 63 (step S330). When the specific object is not detected (step S330: NO), the distance sensor 63 continues to monitor the detection of the specific object (step S330). When the specific object is detected (step S330: YES), the image determination unit 142 determines whether the specific object is a preset setting object by pattern matching (step S340). In this embodiment, the palm of a person is set in advance as the setting object. However, in other embodiments, the object may be an object other than the palm, and the user can arbitrarily change the setting object. An aspect may be sufficient.

  In the process of step S340, when it is determined that the detected specific object is not the setting object (step S340: NO), the distance sensor 63 is again an object different from the detected specific object. Object detection is monitored (step S330). When it is determined that the specific object is the setting object (step S340: YES), the image determination unit 142 determines whether the size of the specific object is equal to or larger than a predetermined size. It is determined whether or not (step S350). The image display unit 20 displays a control image set in accordance with the size of the captured specific object.

  When it is determined that the size of the specific object is not equal to or greater than the predetermined size (step S350: NO), the image determination unit 142 adjusts the control image according to the detected specific object. The size is adjusted (step S360). When the size of the control image is adjusted (step S360), or when it is determined in the process of step S340 that the size of the control image is equal to or larger than the predetermined size (step S340: YES) The image display unit 20 displays a control image including a position image indicating the detected user's line-of-sight direction (step S370).

  FIG. 6 is an explanatory diagram illustrating an example of the control image CI before the size is adjusted. FIG. 6 shows a visual field VR visually recognized by the user when the control image CI is displayed on the image display unit 20 before the size of the control image CI is adjusted. The control image CI is not displayed on the image display unit 20 when it is determined that the control image CI is not larger than a predetermined size. Therefore, the control image CI before the size is adjusted is not visually recognized by the user. As shown in FIG. 6, the transmitted outside scene SC is visually recognized in the user's visual field VR, and the palm HD of the user is included in the outside scene SC. The maximum image display area PN indicates the maximum area where the user can visually recognize the image light generated by the image display unit 20. The outer frame of the maximum image display area PN is not visually recognized by the user, but is shown by a solid line in FIG. 6 for convenience.

  When the image display unit 20 displays the control image CI, first, the image determination unit 142 first sets the tip end SP1 of the thumb, the tip end SP2 of the index finger, the little finger on the palm HD of the user detected by pattern matching. The position of the tip SP3 is detected. The image display unit 20 has at least one of the tip of the thumb SP1, the index finger SP2, and the little finger SP3 as a vertex and passes through the remaining two tips, and the horizontal axis and the vertical axis of the maximum image display area PN. A rectangular control image CI having a line parallel to the axis as an outer frame is displayed. FIG. 6 shows a rectangular control image CI having the tip of the little finger SP3 as a vertex and passing through the tip of the thumb SP1 and the tip of the index finger SP2. The image determination unit 142 determines whether each of the lengths of the vertical and horizontal sides of the outer frame of the generated control image CI is equal to or greater than a predetermined length, whereby the control image CI is determined to be predetermined. It is determined whether or not the size is greater than or equal to. In FIG. 6, since the vertical outer frame in the control image CI before the size adjustment is less than the predetermined length, it is determined that the control image CI is not greater than the predetermined size, and the image display unit 20 Adjusts the size of the control image CI and displays it.

  FIG. 7 is an explanatory diagram illustrating an example of the control image CI whose size has been adjusted. In FIG. 7, the control image CI whose size is adjusted by the image display unit 20 is displayed, and the visual field VR visually recognized by the user is shown. As shown in FIG. 7, the image display unit 20 displays the control image CI that has been adjusted to lengthen the vertical outer frame that is determined to be less than the predetermined length. In FIG. 6, when the control image CI before the size adjustment is temporarily displayed, the control image CI is generated in the upper part of the maximum image display area PN. Therefore, after the size of the control image CI is adjusted, as shown in FIG. 7, the image display unit 20 displays the control image CI with the vertical outer frame extended downward. In the present embodiment, the vertical length of the extended outer frame is a length that is insufficient with respect to the length set in advance when the control image CI is displayed.

  As shown in FIG. 7, the displayed control image CI includes a pointer P1, a keyboard KB, input spaces IB1 and IB2, a search button B1, a switching key B2, and a menu switching key B3. It is. The pointer P1 is a position image representing a position corresponding to the user's line-of-sight direction in the control image CI. The keyboard KB is a keyboard on which alphabets are arranged, and characters input using the keyboard KB and the pointer P1 are displayed in the input spaces IB1 and IB2. The keyboard KB includes a switching key B2 for switching between uppercase and lowercase letters of the alphabet. The search button B1 is a button for executing a search on the Internet based on the characters input in the input spaces IB1 and IB2. The menu switching key B3 is a button for switching the control image CI to a different control image, for example, displaying a list of files stored in a specific folder. The keys corresponding to the alphabets included in the keyboard KB, the search button B1, the switching key B2, and the menu switching key B3 correspond to objects in the claims.

  When the control image CI is displayed (step 370 in FIG. 4), the line-of-sight input unit 175 next monitors input detection based on the user's line-of-sight direction and the eyelid state (step S380 in FIG. 5). ). In the present embodiment, the image determining unit 142 determines the gaze direction of the right eye of the user to be imaged, and the image display unit 20 displays the pointer P1 at a position corresponding to the gaze direction of the right eye. Therefore, the pointer P1 changes the position in the control image CI according to the change in the user's line-of-sight direction. When the eyelid state indicating the predetermined determination operation is not detected (step S380: NO), the line-of-sight input unit 175 continues to monitor the detection of the eyelid state indicating the determination operation (step S380). .

  In the process of step S380, with the pointer P1 overlapping the selectable button of the control image CI, the blinking of the right eyelid with the eyelid indicating the determination operation closed for a predetermined period If detected (step S380: YES), the line-of-sight input unit 175 performs an input operation. The line-of-sight input unit 175 determines the key on which the pointer P1 overlaps and performs an input operation, and the image display unit 20 displays the input result in the input spaces IB1 and IB2 (step S390). As shown in FIG. 7, “A”, “B”, and “C” are input and displayed in the input spaces IB1 and IB2 in uppercase letters. In this embodiment, when the alphabet “a” is determined after the switching key B2 is determined, the image display unit 20 displays the uppercase “A” in the input spaces IB1 and IB2. Therefore, the input result of the input spaces IB1 and IB2 in FIG. 7 is a state in which the switching key B2, “a”, the switching key B2, “b”, the switching key B2, and “c” are sequentially selected. In another embodiment, when the switching key B2 is determined, the subsequent alphabets may be interchanged and fixed, and the lowercase alphabets displayed on the keys of the keyboard KB are changed to uppercase characters. May be displayed.

  Next, the eye imaging cameras 37 and 38 monitor the detection of the eyelid state indicating the operation of searching the Internet after finishing the input to the input spaces IB1 and IB2 (step S400 in FIG. 4). If the eyelid state indicating the internet search operation is not detected (step S400: NO), the detection of the eyelid state indicating the determination operation is continuously monitored (step S380). When the state of the eyelid for determining the search button B1 indicating an operation for performing an Internet search is detected (step S400: YES), the control unit 10 performs an Internet search via the wireless communication unit 132, and the image display unit 20 The search result is displayed (step S410). When the search result is displayed, the position where the pointer P1 is displayed is changed based on the change in the line-of-sight direction determined by the image determination unit 142, as in the case where the control image CI is displayed. When the eyelid state indicating determination is detected, the line-of-sight input unit 175 performs various operations based on the position of the selectable icon or the like and the pointer P1 in the displayed search result.

  Next, the eye imaging cameras 37 and 38 monitor detection of an eyelid state indicating an operation for changing the search result displayed on the image display unit 20 to a different display image (step S420). As an operation for changing the displayed search result, when it is detected that the right eyelid is closed and the left eyelid is opened at a predetermined time (step S420: YES), the eye imaging cameras 37 and 38 detect the input. Is monitored (step S380). When the eyelid state for changing the displayed search result is not detected (step S420: NO), the eye imaging cameras 37 and 38 monitor the detection of the eyelid state indicating the operation for ending the line-of-sight operation input process. (Step S430). When the eyelid state indicating the operation for ending the line-of-sight operation input process is not detected (step S430: NO), the eye imaging cameras 37 and 38 monitor the detection of input (step S380). When the state where both eyelids are closed for a predetermined time and only the right eyelid is opened for a predetermined time is detected as an operation to end the line-of-sight operation input (step S430: YES), the control unit 10 performs a line-of-sight operation input process. finish.

  As described above, in the head-mounted display device 100 according to this embodiment, the image display unit 20 includes the pointer P1, the keyboard KB, the input spaces IB1 and IB2, the search button B1, the switching key B2, A control image CI including a menu switching key B3 is displayed. The line-of-sight input unit 175 controls the head-mounted display device 100 based on the positional relationship between the keyboard KB and the pointer P1. Therefore, in the head-mounted display device 100 according to the present embodiment, various controls of the head-mounted display device 100 can be performed based on the user's line-of-sight direction without the user using his / her hand. The user's operability is improved. Further, even if there is no user interface such as a keyboard having a shape as an object, the user can control the head-mounted display device 100, and the portability of the head-mounted display device 100 is improved.

  In the head-mounted display device 100 according to this embodiment, the eye imaging cameras 37 and 38 capture the right eye and the left eye of the user, and the image determination unit 142 determines the open / closed state of both eyelids. The line-of-sight input unit 175 selects an alphabet key on the keyboard KB based on the positional relationship between the keyboard KB and the pointer P1, and determines the selected key based on the open / closed state of both eyelids. Therefore, in the head-mounted display device 100 according to the present embodiment, the head-mounted display device 100 is controlled based on the user's line-of-sight direction and the open / closed state of both eyelids. In addition, the head-mounted display device 100 can be controlled, and the operability for the user is improved. In addition, since the determination is made based on the combination of the open / closed states of both eyelids, many determination combinations corresponding to the type of operation can be provided to the user, and the operability for the user is improved.

  In the head-mounted display device 100 according to the present embodiment, the image determination unit 142 performs pattern matching on the object detected by the distance sensor 63 as to whether the object is the same as the preset registered object. . When the specific object is a preset target object by pattern matching of the image determination unit 142, the image display unit 20 sets the control image set according to the size of the captured specific object. Display CI. Therefore, in the head-mounted display device 100 of the present embodiment, the size of the control image CI is changed according to the size of the object recognized by the image determination unit 142. The size of the work image CI can be adjusted, and the convenience for the user is improved. Further, since the control image CI is changed from the non-display state to the display state, the user only has to hold the palm HD in front of the user's eyes, so that the convenience for the user is improved.

B. Variations:
In addition, this invention is not limited to the said embodiment, It can implement in a various aspect in the range which does not deviate from the summary, For example, the following deformation | transformation is also possible.

B1. Modification 1:
FIG. 8 is an explanatory diagram illustrating an example of the control image CIa displayed on the image display unit 20. FIG. 8 shows the visual field VR including the outside scene SC visually recognized by the user and the displayed control image CIa. The control image CIa is an image representing an operation unit (remote control) of a television connected to the head-mounted display device 100 via the wireless communication unit 132. The control image CIa includes a power button PW for turning on / off the TV, volume buttons Vup, Vdw for adjusting the TV volume, and channel change buttons Cup, Cdw for changing the TV channel. It is. When the volume button Vup is selected and determined, the volume of the television is increased by a predetermined value, and when the volume button Vdw is selected and determined, the volume of the television is decreased by a predetermined value. When the channel change button Cup is selected and determined, the television set receives and sets the number of channels to a larger number by a predetermined number. When the channel change button Cdw is selected and determined, the television set receives and sets the number of channels to be a smaller number by a predetermined number. The control image CIa also includes a DVD operation button group PB having a plurality of buttons for operating a DVD player connected to the television by wire. As shown in FIG. 8, the pointer P1 is displayed at a position overlapping the stop button SB for stopping the reproduction of the DVD player included in the DVD operation button group PB. Similar to the above embodiment, when the eye imaging cameras 37 and 38 detect the state of the eyelid indicating the determination operation, the line-of-sight input unit 175 stops the content being played back on the DVD player. In this modification, the adjustment of the device connected to the head-mounted display device 100 is operated according to the user's line-of-sight direction and the state of the eyelid.

  FIG. 9 is an explanatory diagram illustrating an example of the control image CIb displayed on the image display unit 20. FIG. 9 shows the visual field VR including the outside scene SC visually recognized by the user and the displayed control image CIb. The control image CIb shown in FIG. 8 is an image that displays a plurality of icon images represented by characters associated with content such as moving images and music stored in the storage unit 120 as a list. The control image CIb includes a cursor image C1, a list image C2, and a pointer P1. In this modification, the cursor image C1 has a cross shape, and is an image in which the vertical and horizontal directions are selected. As shown in FIG. 8, when an eyelid state indicating a determination operation is detected while the pointer P1 is overlapped with an arrow pointing to the right of the cross, a map icon image is selected in the list image C2. The selected display image SI displaying that it is moving moves to the right. In this modification, since there is no other icon image on the right of the map icon image in the list image C2, the selection display image SI loops and moves to the left mail icon image, and becomes the mail icon image. They are displayed overlapping. In this modification, the pointer P1 is superimposed on any icon image of the list image C2 even if the determination operation is not performed with the pointer P1 superimposed on any of the cross directions of the cursor image C1. It is also possible to select and decide a specific icon image. When the determination operation is performed in a state where the specific icon image is displayed overlapping the selection display image SI and the pointer P1 is overlapped with the determination key RB, the icon displayed overlapping the selection display image SI is displayed. An image is determined and an operation associated with the icon image is performed.

B2. Modification 2:
In the above embodiment, the determination operation is detected based on the open / closed state of the eyelid. However, the method for detecting the determination operation is not limited to the open / closed state of the eyelid, and various modifications are possible. Even when the eyelids are open, various operations may be performed based on the color and illuminance of the outside scene visually recognized by the user. In this modification, the illuminance sensors 45 and 47 detect the illuminance of an outside scene that is assumed to be visually recognized by the right and left eyes of the user. Therefore, even if the user does not close the eyelid, for example, if the surface of the left optical image display unit 28 is covered with the palm of the user, the left eye will not be able to visually recognize the outside scene. Illuminance having a value smaller than the illuminance detected by the eye illuminance sensor 45 is detected. When the illuminance processing unit 145 determines that the illuminance detected by the left eye illuminance sensor 47 is smaller than the illuminance detected by the right eye illuminance sensor 45 by a predetermined threshold or more, the line-of-sight input unit 175 selects the left eyelid in the above embodiment. The same operation as in the closed state is performed. Therefore, in the head-mounted display device 100 according to this modified example, when the user is tired from the operation due to the state of the eyelid, the user can perform a sensuous operation using a part of the body such as the palm, User convenience is improved.

  In another modification, instead of the illuminance sensors 45 and 47, RGB color sensors that detect RGB data in the line-of-sight directions of the right eye and the left eye may be arranged. Various operations may be performed according to the values detected by the RGB color sensors corresponding to the right eye and the left eye. For example, when the surface of the left optical image display unit 28 is covered, the RGB color sensor corresponding to the left eye becomes RGB data representing a color close to black. Conversely, when the RGB data detected by the RGB color sensor corresponding to the right eye does not represent a color close to black, it may be determined that the user's left eyelid is closed.

B3. Modification 3:
In the above embodiment, the control image CI is displayed at a predetermined position. However, the position at which the control image CI is displayed does not need to be fixed at a predetermined position or the like. The display position of the image CI and the method for changing the display position can be variously modified. In the head-mounted display device 100 of this modification, the 9-axis sensor 66 and the direction determination unit 161 estimate the orientation of the user's head and calculate the angle formed by the estimated orientation of the head and the direction of gravity. To do. For example, when the user faces upward and the angle calculated by the direction determination unit 161 is equal to or larger than a predetermined angle, the image display unit 20 displays the control image CI other than the upper part in the maximum image display region PN. To display. Therefore, in the head-mounted display device 100 according to this modification, it is estimated that the user has changed the orientation of the head and the object that the user wants to view is changed, and the position that does not hinder the user's view Since the control image CI is automatically displayed, the convenience for the user is improved.

  Further, the position where the image display unit 20 displays the control image CI may be changed based on the user's line-of-sight direction, the open / closed state of the eyelid, and the like. For example, when the eye imaging cameras 37 and 38 detect a state in which only the right eyelid of the user is closed for a predetermined time and then detect a state in which only the left eyelid is closed for a predetermined time within a certain period of time, the user The position of the control image CI in the maximum image display area PN may be changed based on the viewing direction. In the head-mounted display device 100 according to this modified example, the position for displaying the control image CI can be set without using a hand or the like based on the user's intention, so that the user's operability and convenience are improved. To do.

  Further, in the above embodiment, when the palm HD as the setting object is detected, the control image CI is displayed according to the size of the captured palm HD, but the setting object and the control image CI are displayed. The relationship is not limited to this and can be variously modified. For example, the type of the control image CI to be displayed may be changed according to the size and shape of the captured palm HD. Further, based on the positional relationship between the captured palm HD and the image display unit 20, the display position of the control image CI may be set at a position other than the center of the maximum image display area PN. In this modification, based on the position and size of the captured setting object, the user can visually recognize the transparent outside scene SC and the control image CI so that they do not overlap in many areas. Improved convenience.

B4. Modification 4:
In the above embodiment, the distance sensor 63 detects an object within a predetermined distance from the distance sensor 63. However, the head-mounted display device 100 does not necessarily include the distance sensor 63, and the head-mounted The configuration of the mold display device 100 can be variously modified. For example, the outside scene imaging camera 61 and the image determination unit 142 may perform pattern matching on the captured object and determine whether the object is a setting object regardless of the distance to the object.

  In the above-described embodiment, the open / closed state of the eyelid is determined by the eye imaging cameras 37 and 38 and the image determination unit 142. However, the method of detecting the shielding object that blocks the user's line-of-sight direction is not limited to this, and various modifications Is possible. For example, an infrared sensor may be used instead of the eye imaging cameras 37 and 38. The infrared sensor reflects the emitted infrared light to the user's eye or eyelid, and determines the open / closed state of the user's eyelid based on the reflected light. The image determination unit 142 binarizes the RGB data of the image acquired by the eye imaging cameras 37 and 38 into black and white instead of pattern matching, and opens and closes the user's eyelid based on the binarized data. The state may be determined.

  The eye imaging cameras 37 and 38 may be provided with a stabilizer function. With the stabilizer function, for example, when the user is walking, the eye imaging cameras 37 and 38 can detect and determine the state of the eyelid more accurately in order to reduce shaking caused by walking.

  Further, the power consumption of the power source 130 may be suppressed based on the operation unit 135 and the detected eyelid state. For example, when the trackpad 14 in the operation unit 135 accepts an operation from the user, power supply to the eye imaging cameras 37 and 38 and the illuminance sensors 45 and 47 that detect the state of the user's eyelid is stopped. May be. Further, when the head-mounted display device 100 and the device connected to the head-mounted display device 100 are operated depending on the user's line-of-sight direction and eyelid state, power supply to the operation unit 135 is stopped. May be. In this modification, when an operation from the user is received by one of the operation unit 135 and the control image CI, by stopping power to the operation unit that has not received the operation, The power consumed by the head-mounted display device 100 can be suppressed. Further, instead of the control unit 10, for example, it may be connected to a mobile terminal such as a smartphone to suppress power consumption of the connected mobile terminal.

  In the above embodiment, the line-of-sight input unit 175 operates the head-mounted display device 100 based on the user's line-of-sight direction and the state of the eyelid. However, the method of operating the head-mounted display device 100 is not necessarily required. Is not limited to this, and various modifications are possible. For example, the eye imaging cameras 37 and 38 and the image determination unit 142 detect only the user's line-of-sight direction, and any key included in the control image CI is selected based on the line-of-sight direction. It may be determined by any button included. Further, instead of the buttons included in the operation unit 135, a separately provided user interface may be used, or a determination button or the like formed on the image display unit 20 may be used. In this modification, the selection is performed only in the direction of the line of sight, and the determination is not performed on the eyelids, but is performed with the buttons provided on the operation unit 135, so that the user can easily perform the determination operation. User convenience and operability are improved.

  In the above embodiment, as specific examples corresponding to the objects in the claims, keys corresponding to the alphabets included in the keyboard KB, search buttons B1, switching keys B2, and menu switching keys B3, and pointers in the claims As a specific example corresponding to, the pointer P1 has been described as an example, but the object and the pointer can be variously modified. For example, as an object or a pointer in the claims, a shape, a color, a figure, or the like may be modified such as a mark, a sign, a marker, a symbol, a selection button, a soft key, and a soft keyboard. .

  In the above-described embodiment, the eye imaging cameras 37 and 38 are at a predetermined distance from the eye imaging cameras 37 and 38, and the opening / closing of the eyelids in the line-of-sight direction is detected. It is not necessary to detect the shield. For example, an operation similar to detecting the opening / closing of the eyelid may be performed by detecting a shield that exists within a certain range up to a predetermined distance.

B5. Modification 5:
The outside scene imaging camera 61 may be disposed in the control unit 10 or may be disposed separately from the control unit 10 and the image display unit 20, for example, on an upper part of a helmet worn by the user. .

  In the above embodiment, as shown in FIG. 2, the 9-axis sensor 66, the distance sensor 63, the outside scene imaging camera 61, the eye imaging cameras 37 and 38, and the illuminance sensors 45 and 47 are arranged in the image display unit 20. However, the image display unit 20 is not necessarily arranged, and the arrangement can be variously modified. For example, the right-eye imaging cameras 37 and 38 are separate from the image display unit 20 and may be mounted on the head separately from the image display unit 20. You may form so that attachment or detachment is possible.

  In the above embodiment, the operation unit 135 is formed in the control unit 10, but the mode of the operation unit 135 can be variously modified. For example, a mode in which a user interface that is the operation unit 135 is provided separately from the control unit 10 may be used. In this case, the operation unit 135 is separate from the control unit 10 in which the power supply 130 and the like are formed. Therefore, the operation unit 135 can be reduced in size and user operability is improved. Further, a nine-axis sensor that detects the movement of the operation unit 135 is formed in the operation unit 135, and various operations are performed based on the detected movement, so that the user can sensuously feel the head-mounted display device 100. Can be operated.

  For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image generation unit may use LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. In the case of the laser retinal projection type, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes.

  Further, for example, the head-mounted display may be a head-mounted display in which the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit does not completely cover the user's eye. Good. The head mounted display may be a so-called monocular type head mounted display.

  The earphone may be an ear-hook type or a headband type, or may be omitted. Further, for example, it may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet.

B6. Modification 6:
The configuration of the head-mounted display device 100 in the above embodiment is merely an example and can be variously modified. For example, one of the direction key 16 and the track pad 14 provided in the control unit 10 may be omitted, in addition to the direction key 16 and the track pad 14, or in place of the direction key 16 and the track pad 14, etc. An operation interface may be provided. Moreover, the control part 10 is a structure which can connect input devices, such as a keyboard and a mouse | mouth, and is good also as what receives an input from a keyboard or a mouse | mouth.

  As the image display unit, instead of the image display unit 20 worn like glasses, another type of image display unit such as an image display unit worn like a hat may be adopted. Further, the earphones 32 and 34 and the outside scene imaging camera 61 can be omitted as appropriate. Moreover, in the said embodiment, although LCD and a light source are utilized as a structure which produces | generates image light, it replaces with these and you may employ | adopt other display elements, such as an organic EL display. In the above embodiment, the 9-axis sensor 66 is used as a sensor for detecting the movement of the user's head. Instead, one or two of an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor are used. You may use the sensor comprised from. In the above embodiment, the head-mounted display device 100 is a binocular optical transmission type. However, the present invention can be applied to other types of head-mounted display such as a video transmission type and a monocular type. The same applies to the apparatus.

  In the above-described embodiment, the head-mounted display device 100 may guide image light representing the same image to the left and right eyes of the user so that the user can visually recognize the two-dimensional image. It is also possible to guide the user to view the three-dimensional image by guiding image light representing different images to the eyes.

  In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. For example, in the above-described embodiment, the image processing unit 160 and the sound processing unit 170 are realized by the CPU 140 reading and executing a computer program, but these functional units may be realized by a hardware circuit. Good.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

  Moreover, in the said embodiment, as shown in FIG. 1 and FIG. 2, the control part 10 and the image display part 20 are formed as a separate structure, However, about the structure of the control part 10 and the image display part 20, about. However, the present invention is not limited to this, and various modifications are possible. For example, all of the components formed in the control unit 10 may be formed inside the image display unit 20 or a part thereof may be formed. In addition, the power source 130 in the above embodiment may be formed independently and replaceable, or the configuration formed in the control unit 10 may be overlapped and formed in the image display unit 20. For example, the CPU 140 shown in FIG. 2 may be formed in both the control unit 10 and the image display unit 20, or a function performed by the CPU 140 formed in the control unit 10 and the CPU formed in the image display unit 20. May be configured separately.

  Further, the control unit 10 may be built in a personal computer (PC), and the image display unit 20 may be used instead of the PC monitor, or the control unit 10 and the image display unit 20 may be integrated. Further, it may be an embodiment of a wearable computer attached to a user's clothes.

  The present invention is not limited to the above-described embodiments and modifications, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments and the modifications corresponding to the technical features in each form described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Control part 11 ... Decision key 12 ... Illumination part 13 ... Display switching key 14 ... Trackpad 15 ... Luminance switching key 16 ... Direction key 17 ... Menu key 18 ... Power switch 20 ... Image display part 21 ... Right holding part 22 ... Right display drive unit 23 ... Left holding unit 24 ... Left display drive unit 26 ... Right optical image display unit 28 ... Left optical image display unit 30 ... Earphone plug 32 ... Right earphone 34 ... Left earphone 37 ... Right eye imaging camera (detection unit) , Gaze direction detector)
38 ... Left-eye imaging camera (detection unit, gaze direction detection unit)
40 ... Connection part 42 ... Right cord 44 ... Left cord 45 ... Right eye illuminance sensor (detection unit)
46 ... Connecting member 47 ... Left eye illuminance sensor (detection unit)
48 ... Body code 51, 52 ... Transmission unit 53, 54 ... Reception unit 61 ... Outside scene imaging camera 63 ... Distance sensor (image recognition unit)
66 ... 9-axis sensor (direction detection unit)
DESCRIPTION OF SYMBOLS 100 ... Head-mounted display apparatus 110 ... Input information acquisition part 120 ... Memory | storage part 130 ... Power supply 132 ... Wireless communication part 135 ... Operation part 140 ... CPU
142 ... Image determination unit (gaze direction detection unit, image recognition unit)
145 ... Illuminance processing unit (detection unit)
DESCRIPTION OF SYMBOLS 150 ... Operating system 160 ... Image processing part 161 ... Direction determination part (orientation detection part)
170 ... voice processing unit 175 ... gaze input unit (gaze input control unit)
180 ... interface 190 ... display control unit 201 ... right backlight control unit 202 ... left backlight control unit 211 ... right LCD control unit 212 ... left LCD control unit 221 ... right backlight 222 ... left backlight 241 ... right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate VSync ... Vertical synchronization signal HSync ... Horizontal synchronization signal PCLK ... Clock signal Data ... Image data USB interface ... Micro P1 ... Pointer B1 ... Search Button (object)
B2 ... Switching key (object)
B3 ... Menu switching key (object)
C1 ... Cursor image (control image)
C2 ... List image (control image)
CI, CIa, CIb ... control image (control image)
KB ... keyboard RB ... decision key (object)
PB ... DVD operation button group SB ... Stop button (object)
SC ... Outside view HD ... Hand RE ... Right eye LE ... Left eye SI ... Selected display image EL ... End IL ... Illumination light PL ... Image light PN ... Maximum image display area VR ... Field of view PW ... Power button (object)
IB1, IB2 ... input space SP1 ... tip of thumb SP2 ... tip of index finger SP3 ... tip of little finger Vup, Vdw ... volume button (object)
Cup, Cdw ... Channel change button (object)

Claims (9)

A head-mounted display device,
A gaze direction detection unit that detects the gaze direction of the user;
A control image including a plurality of selectable objects and a pointer indicating a position corresponding to the detected line-of-sight direction while transmitting the outside scene while being mounted on the user's head is displayed as an image. An image display unit that is a part of the displayable area and that is displayed in an area that overlaps the viewing direction of the user;
A control unit that controls content according to an object that is in a selectable state by a positional relationship with the pointer among the plurality of objects;
A head-mounted display device comprising: The head-mounted display device according to claim 1, further comprising:
A detection unit that is present at a predetermined distance from the user's eyes and that detects a shield in the line-of-sight direction;
The control unit is a head-mounted display device that determines the object in the selectable state based on a detection result of the shielding object. The head-mounted display device according to claim 2,
The detection unit detects the state of the user's eyelid as the shield,
The control unit is a head-mounted display device that controls contents according to the detected state of the eyelid and the positional relationship. The head-mounted display device according to claim 3,
The detection unit detects the open / closed state of the user's right eyelid and the open / closed state of the left eyelid,
The head-mounted display device that controls contents according to the positional relationship and a combination of the detected opening / closing state of the right eyelid and the opening / closing state of the left eyelid. The head-mounted display device according to any one of claims 2 to 4,
The detection unit detects a right-eye pre-index value and a left-eye pre-index value indicating at least one of color or illuminance in a range from each of a user's right eye and left eye to the predetermined distance,
The said control part is a head mounted display apparatus which controls the content according to the said positional relationship, the detected said right front index value, and the said left front index value. A head-mounted display device according to any one of claims 2 to 5,
The image display unit is a head-mounted display device that sets the region for displaying the control image based on the detected line-of-sight direction and the detection result of the shielding object. The head-mounted display device according to claim 1,
The plurality of objects are objects representing keys constituting the keyboard,
The control unit inputs a character corresponding to a key assigned to the object on which the pointer overlaps as a control of contents according to an object that can be selected by a positional relationship with the pointer. Type display device. The head-mounted display device according to any one of claims 1 to 6, further comprising:
It has an orientation detection unit that detects the orientation of the user's head,
The image display unit is a head-mounted display device that specifies the viewing direction of the user based on the detected orientation of the head. A method for controlling a head-mounted display device,
The direction of the user's line of sight is detected by a sensor provided in the head-mounted display device,
In a state in which the head-mounted display device is mounted on the user's head, a control image including a plurality of selectable objects and a pointer indicating a position corresponding to the detected line-of-sight direction, It is a part of the displayable area in the image display unit that can transmit the outside scene, and is displayed in an area that overlaps the viewing direction of the user,
A control method for performing control of contents according to an object that can be selected by a positional relationship with the pointer among the plurality of objects.

Images