JP6304415B2 - Head-mounted display device and method for controlling head-mounted display device - Google Patents

Description

  The present invention relates to a head-mounted display device.

  A head-mounted display device (Head Mounted Display, HMD) that is a display device mounted on the head is known. The head-mounted display device, for example, generates image light representing an image using a liquid crystal display and a light source, and guides the generated image light to a user's eye using a projection optical system or a light guide plate This causes the user to visually recognize the virtual image. There are two types of head-mounted display devices: a transmission type in which the user can visually recognize the outside scene in addition to a virtual image, and a non-transmission type in which the user cannot visually recognize the outside scene. The transmissive head-mounted display device includes an optical transmissive type and a video transmissive type. On the other hand, there is known a camera that detects the state of the user's eyes, images based on the detected eye state, and images the outside scene.

  Japanese Patent Application Laid-Open No. 2004-133830 discloses a camera that detects a wink of a user's eyes and captures an outside scene based on the detected wink, and captures a shooting mode after a predetermined period of time has elapsed after the wink is detected. There is disclosed a technique capable of selecting a mode and a non-delay mode in which imaging is performed without delay. Patent Document 2 discloses a camera that suppresses camera shake during imaging by imaging an outside scene based on the detected wink of the user's eyes. Further, in Patent Document 3, in a head-mounted display device in which an image of content or the like is visually recognized by one eye of a user, an eye movement of one eye is imaged and an eyelid operation of the other eye is performed. A technique for capturing an image and setting an image to be visually recognized by a user based on the captured eye movement and eyelid movement is disclosed.

JP 2010-41643 A JP-A-6-208158 JP 2010-152443 A

  However, with the techniques described in Patent Document 1 and Patent Document 2, when the user captures an outside scene, the user must check the image captured by looking at the viewfinder, which improves operability during imaging. There was room for improvement. In addition, although the imaging function of the camera is controlled based on the state of the eyelid of the user, there is a demand for controlling functions other than the imaging function based on the state of the eyelid. Further, in the head-mounted display device described in Patent Document 3, an image that is visually recognized based on the eye movement of the user's eye is set. An operation method was desired.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to one aspect of the present invention, a transmissive head-mounted display device is provided. The head-mounted display device includes: an imaging unit that images an outside scene; a detection unit that detects a shielding object that blocks a user's line of sight from a user's eyes; and a detection result of the shielding object An imaging operation unit for operating the imaging unit; generating an area image light that is an image light indicating an imaging area in a state of being mounted on a user's head, and causing the user to visually recognize the area image light. And an image display unit that transmits the outside scene. According to the head-mounted display device of this embodiment, the user visually recognizes the area image light representing the whole or a part of the area of the outside scene to be imaged while the user visually recognizes the outside scene. Therefore, the user can easily measure the timing of image capturing by visually recognizing the outside scene without looking through the viewfinder and the like, and the convenience for the user is improved. In addition, since the user can take an image of the outside scene without touching the imaging unit, the user can take an image while suppressing camera shake.

(2) In the head-mounted display device according to the above aspect, the detection unit detects a state of the user's eyelid as the shield; the imaging operation unit is based on the detected state of the eyelid. The operation may be performed. According to the head-mounted display device of this form, the user can pick up the outside scene using the image pickup unit without using the hand, and the convenience for the user is improved.

(3) In the head-mounted display device according to the above aspect, the detection unit detects an open / closed state of a user's right eyelid and an open / closed state of the left eyelid; the imaging operation unit detects the detected right eyelid The operation may be performed based on a combination of the open / closed state of the left eyelid and the open / closed state of the left eyelid. According to the head-mounted display device of this form, the user can pick up the outside scene using the image pickup unit without using the hand, and the convenience for the user is improved.

(4) The head-mounted display device according to the above aspect further includes: a gaze direction estimation unit that estimates a user's gaze direction; and the image display unit has a position corresponding to the gaze direction with respect to the captured outside scene. Generating a line-of-sight position image light that is an image light to be displayed, and causing the user to visually recognize the line-of-sight position image light; the imaging operation unit is configured to capture an image of an outside scene to be imaged based on the estimated line-of-sight direction and the combination A partial area is set; the area image light may be image light indicating the set partial area. According to the head-mounted display device of this aspect, the region image light that represents an image of the set partial region by setting the partial region captured by the imaging unit without the user using the hand. Can be visually recognized by the user, and user convenience, convenience, and operability are improved.

(5) In the head-mounted display device of the above aspect, the operation includes adjustment of a focal length of a lens of the imaging unit; and the region image light includes a case where the focal length is included in a predetermined distance The area image light is generated in a peripheral part of the partial area, and the external image representing the external scene image of the partial area is generated when the focal distance is not included in the predetermined distance. Image light may be generated. According to this type of head-mounted display device, the user can set the focal length of the imaging unit without using his / her hand, which improves the convenience for the user. In addition, when the set focal length is a focal length that becomes an external scene similar to the visually recognized external scene, the region image light is image light indicating a partial region. In addition, when the set focal length is a focal length that becomes an outside scene different from the outside scene that is visually recognized, the area image light is outside scene image light that represents an outside scene image of a part of the area. Therefore, the image display unit generates image light according to the comparison between the outside scene visually recognized by the user and the region image light, and thus the visibility and operability of the user are improved.

(6) In the head-mounted display device according to the above aspect, the detection unit is a right-eye front index indicating at least one of color and illuminance in a range from each of a user's right eye and left eye to the predetermined distance. The imaging operation unit may perform the operation based on the detected right-eye front index value and the left-eye front index value. According to the head-mounted display device of this form, when the user is tired from an operation caused by a change in the state of the eyelid such as a wink, the user can use the part of the body such as the hand to An operation similar to a change can be performed sensuously, and convenience for the user is improved.

(7) The head-mounted display device according to the above aspect further includes: an image recognition unit that recognizes a specific target included in an outside scene to be imaged based on the operation; and the imaging operation unit is imaged When the specific object is recognized in the outside scene, a part of the outside scene to be imaged is set based on the specific object; the outside scene image light includes the set part of the area. The image light to represent may be sufficient. According to this form of the head-mounted display device, even when the specific target is not included in the captured outside scene, the position of the specific target is automatically estimated and the specific target is imaged. Therefore, convenience for the user is improved.

(8) In the head-mounted display device according to the aspect described above, when the specific target is not recognized in the captured outside scene, the imaging operation unit recognizes the specific target in the captured external scene. A direction from the imaging unit to the specific target is estimated based on a change in the position of the specific target at the time; the imaging unit may image the estimated direction. According to the head-mounted display device of this aspect, even if a transmitter or the like is not attached to a specific target, the specific target is automatically tracked and imaged by image recognition of the specific target. Therefore, user convenience and operability are improved. In addition, since the user visually recognizes image light representing an external scene including a specific target different from the external scene visually recognized by the user, the user needs to visually recognize the specific target even when imaging the specific target. Therefore, the convenience for the user is improved.

(9) The head-mounted display device according to the above aspect further includes: a direction detection unit that detects a direction of the user's head with respect to the direction of gravity; and the detected direction of the head and the direction of gravity form When the angle is greater than or equal to a predetermined value, based on the angle, the region image light indicates all or a part of the outside scene that is imaged with an axis parallel to the horizontal plane orthogonal to the gravity direction as the optical axis. It may be generated as image light. According to the head-mounted display device of this embodiment, even when the orientation of the user's head is temporarily changed, the imaging unit captures an outside scene with the axis parallel to the horizontal plane as the optical axis. Therefore, user convenience improves.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with another new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including one or more or all of the four elements of the imaging unit, the detection unit, the imaging operation unit, and the image display unit. . That is, this apparatus may or may not have an imaging unit. Moreover, the apparatus may or may not have the detection unit. Further, the apparatus may or may not have an imaging operation unit. Further, the apparatus may or may not have an image display unit. The imaging unit may image an outside scene, for example. For example, the detection unit may detect a shielding object that blocks a user's line of sight at a predetermined distance from the user's eyes. For example, the imaging operation unit may operate the imaging unit based on the detection result of the shielding object. The image display unit generates, for example, region image light that is image light indicating an imaging region in a state where the image display unit is worn on the user's head, and allows the user to visually recognize the region image light and transmits the outside scene. May be. Such a device can be realized as, for example, a head-mounted display device, but can also be realized as a device other than the head-mounted display device. According to such an embodiment, at least one of various problems such as improvement and simplification of the apparatus, integration of the apparatus, and improvement of convenience can be solved. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device.

  The present invention can also be realized in various forms other than the head-mounted display device. For example, a method for controlling a head-mounted display device, a head-mounted display system, a computer program for realizing the functions of the head-mounted display system, a recording medium on which the computer program is recorded, a carrier wave including the computer program It can be realized in the form of a data signal or the like embodied therein.

2 is an explanatory diagram showing an external configuration of a head mounted display device 100. FIG. It is explanatory drawing which shows the outline of the relationship between the various sensors in the image display part 20, and a user's eyelid. 3 is a block diagram functionally showing the configuration of the head-mounted display device 100. FIG. It is explanatory drawing which shows a mode that image light is inject | emitted by the image light production | generation part. It is explanatory drawing which shows the flow of a captured image display process. It is explanatory drawing which shows the flow of a captured image display process. It is explanatory drawing which shows an example of the process allocated to the open / closed state of the eyelid in a captured image display process. It is explanatory drawing which shows an example of the visual field VR visually recognized by the user. It is explanatory drawing which shows an example of the visual field VR visually recognized by the user. It is explanatory drawing which shows an example of the visual field VR visually recognized by the user. It is explanatory drawing which shows an example of the visual field VR visually recognized by the user. It is explanatory drawing which shows an example of the visual field VR visually recognized by the user. It is explanatory drawing which shows an example of the visual field VR visually recognized by the user. It is explanatory drawing which shows an example of the visual field VR visually recognized by the user. It is explanatory drawing which shows an example of the visual field VR which a user visually recognizes in imaging adjustment mode.

Next, embodiments of the present invention will be described in the following order based on the embodiments.
A. Embodiment:
A-1. Configuration of head mounted display device:
A-2. Captured image display processing:
B. Variations:

A. Embodiment:
A-1. Configuration of head mounted display device:
FIG. 1 is an explanatory diagram showing an external configuration of the head-mounted display device 100. The head-mounted display device 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The head-mounted display device 100 of the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. In this specification, a virtual image visually recognized by the user with the head-mounted display device 100 is also referred to as a “display image” for convenience. Moreover, emitting image light generated based on image data is also referred to as “displaying an image”.

  The head-mounted display device 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when mounted on the user's head, and a control unit 10 (controller 10) that controls the image display unit 20. I have.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and an outside scene imaging. A camera 61, a track pad 36, a right eye imaging camera 37, a left eye imaging camera 38, a right eye illuminance sensor 45, and a left eye illuminance sensor 47 are included. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

  The display drive units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 and the like (see FIG. 4). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 3) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display driving units 22 and 24 to the eyes of the user. The light control plate is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. In addition, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The outside scene imaging camera 61 is disposed at a position corresponding to the eyebrow of the user when the user wears the image display unit 20. The outside scene imaging camera 61 is formed so as to be able to change the imaging direction with respect to the image display unit 20, captures an outside scene that is an external scenery, and acquires an outside scene image. The outside scene imaging camera 61 in the present embodiment is a monocular camera, but may be a stereo camera. The angle of view of the outside scene imaging camera 61 used in the present embodiment is 40 degrees in the vertical direction and 60 degrees in the left and right directions starting from the optical axis. The outside scene imaging camera 61 corresponds to the imaging unit in the claims.

  The track pad 36 detects the operation of the user's finger on the operation surface of the track pad 36 and outputs a signal corresponding to the detected content. When the user wears the image display unit 20 on the head, the track pad 36 is disposed on the surface opposite to the user's temporal region facing the right display driving unit 22. As the track pad 36, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. In the present embodiment, the track pad 36 is disposed in the right display drive unit 22. However, in other embodiments, the track pad 36 may be disposed in the left display drive unit 24, or may be disposed in the image display unit 20. The structure which can be attached or detached may be sufficient.

  The right-eye imaging camera 37 and the left-eye imaging camera 38 (hereinafter also referred to as “eye imaging cameras 37 and 38”) are small CCD cameras that respectively capture the right eye and the left eye of the user. The right eye illuminance sensor 45 and the left eye illuminance sensor 47 (hereinafter also referred to as “illuminance sensors 45, 47”) are sensors that detect the illuminance of the outside scene visually recognized by the right eye and the left eye of the user. The eye imaging cameras 37 and 38 correspond to a detection unit and a gaze direction estimation unit in claims. The illuminance sensors 45 and 47 correspond to detection units in the claims, and the illuminance detected by the right eye illuminance sensor 45 and the illuminance detected by the left eye illuminance sensor 47 are respectively the right-eye front index value and the left Corresponds to the anterior index value.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 that are connected to the control unit 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24. The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. A connector (not shown) that fits each other is provided at each of the end of the main body cord 48 opposite to the connecting member 46 and the control unit 10. By fitting / releasing the connector of the main body cord 48 and the connector of the control unit 10, the control unit 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for controlling the head-mounted display device 100. The control unit 10 includes a determination key 11, a lighting unit 12, a display switching key 13, a track pad 14, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. Yes. The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control unit 10. The lighting unit 12 notifies the operation state of the head-mounted display device 100 by its light emission state. Examples of the operating state of the head-mounted display device 100 include power ON / OFF. For example, an LED (Light Emitting Diode) is used as the lighting unit 12. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content video between 3D and 2D, for example. The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the head-mounted display device 100 by detecting a slide operation of the switch.

  FIG. 2 is an explanatory diagram showing an outline of the relationship between various sensors in the image display unit 20 and the user's eyelid. FIG. 2 shows an outline when the image display unit 20 and the user wearing the image display unit 20 on the head are viewed from above the user's head. Since each of the right eye imaging camera 37 and the left eye imaging camera 38 is imaging the user's right eye RE and left eye LE, the user's right eyelid RL and left eyelid LL are also imaged simultaneously. Therefore, the eye imaging cameras 37 and 38 image the open / closed state of both eyelids of the user.

  Each of the right eye illuminance sensor 45 and the left eye illuminance sensor 47 includes a light emitting element and a light receiving element, and between the right eye RE and the left optical image display unit 28 with respect to light emission of the illuminance sensors 45 and 47. The illuminance is detected by detecting the reflected light from between the left eye LE and the right optical image display unit 26.

  FIG. 3 is a block diagram functionally showing the configuration of the head-mounted display device 100. As shown in FIG. 3, the control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, an operation reception unit 135, a CPU 140, an interface 180, a transmission unit 51 (Tx51), and a transmission unit. 52 (Tx52). The operation accepting unit 135 accepts an operation by the user, and includes an enter key 11, a display switching key 13, a track pad 14, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18.

  The input information acquisition unit 110 acquires a signal corresponding to an operation input by the user. Examples of the signal corresponding to the operation input include an operation input to the track pad 14, the direction key 16, and the power switch 18 arranged in the operation receiving unit 135, and an operation to the track pad 36 arranged in the image display unit 20. There is an input. The power supply 130 supplies power to each part of the head-mounted display device 100. As the power supply 130, for example, a secondary battery can be used. The storage unit 120 stores various computer programs. The storage unit 120 is configured by a ROM, a RAM, or the like. The CPU 140 reads out and executes the computer program stored in the storage unit 120, thereby operating the operating system 150 (OS150), the display control unit 190, the sound processing unit 170, the direction determination unit 161, the illuminance processing unit 145, and the image determination. Unit 142, imaging setting unit 175, and image processing unit 160.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 controls driving of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD control unit according to control signals. The left LCD 242 driving ON / OFF by 212, the left backlight 222 driving ON / OFF by the left backlight control unit 202, and the like are individually controlled. Thus, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light.

  The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The direction determination unit 161 is an angle formed by the inclination of the user's head estimated based on the orientation of the image display unit 20 detected by the 9-axis sensor 66 described later, and the horizontal direction perpendicular to the gravity direction and the gravity direction. Is calculated. The direction determination unit 161 and the 9-axis sensor 66 correspond to a direction detection unit in the claims.

  The illuminance processing unit 145 performs various calculations on the illuminance of the outside scene visually recognized by the right and left eyes of the user detected by the illuminance sensors 45 and 47. The illuminance processing unit 145 according to the present embodiment calculates the difference between the illuminance of the right-eye outside scene and the illuminance of the left-eye outside scene, and the change in illuminance detected over a certain period. The illuminance processing unit 145 determines whether the calculated illuminance difference and the illuminance difference before and after the illuminance change are equal to or greater than a preset threshold value. For example, when the user intentionally covers only one of the right optical image display unit 26 and the left optical image display unit 28 with his / her hand or the like under a fine daylight, the illuminance processing unit 145 It is determined that there is a difference greater than or equal to the threshold value between the illuminance of the outside scene of the eye and the illuminance of the outside scene of the left eye.

  The image determination unit 142 determines the opening / closing state of the eyelids in each of the right eye and the left eye by analyzing the images of the right eye and the left eye of the user captured by each of the eye imaging cameras 37 and 38. Estimate the gaze direction. The image determination unit 142 determines a thin state or the like by performing pattern matching of the open / closed state of the eyelid for each of the right eye and the left eye of the captured user. Further, the image determination unit 142 determines whether or not the specific object is included in the captured outside scene image, for example, by performing pattern matching to be compared with an image of the specific object stored in advance. In addition, the image determination unit 142 can identify an object that is moving during a certain period of time by comparing the outside scene images captured during a certain period. The image determination unit 142 corresponds to a detection unit, a gaze direction estimation unit, and an image recognition unit in claims.

  The imaging setting unit 175 determines various types of the outside scene imaging camera 61 based on the angle calculated by the direction determination unit 161, the determination of the illuminance processing unit 145, the eyelid opening / closing state determined by the image determination unit 142, the estimated gaze direction, and the like. Setting is performed, and an image signal indicating all or a part of the captured outside scene is generated for the image display unit 20.

  The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). Thereafter, the image processing unit 160 stores the converted digital image signal as image data Data (RGB data) of the target image in the DRAM in the storage unit 120 for each frame. Note that the image processing unit 160 may execute image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data as necessary. .

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control unit 10 and the image display unit 20.

  In addition, the image processing unit 160 transmits image data generated based on the image signal generated by the imaging setting unit 175 to the image display unit 20, and displays an image representing an image based on the image data to the image display unit 20. Produce light.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker (not shown) connected to the connecting member 46 ( (Not shown). For example, when the Dolby (registered trademark) system is adopted, processing on the audio signal is performed, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, for example.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and an outside scene imaging. A camera 61, a nine-axis sensor 66, a track pad 36, a right-eye imaging camera 37, a left-eye imaging camera 38, a right-eye illuminance sensor 45, and a left-eye illuminance sensor 47 are provided.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of the user's head is detected. Since the orientation of the image display unit 20 is known from the detected movement of the user's head, the direction determination unit 161 can estimate the user's line-of-sight direction.

  The right display driving unit 22 includes a receiving unit 53 (Rx53), a right backlight control unit 201 (right BL control unit 201) and a right backlight 221 (right BL221) that function as a light source, and a right LCD that functions as a display element. A control unit 211, a right LCD 241 and a right projection optical system 251 are included. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit 54 (Rx54), a left backlight control unit 202 (left BL control unit 202) and a left backlight 222 (left BL202) that function as a light source, and a left LCD that functions as a display element. A control unit 212 and a left LCD 242 and a left projection optical system 252 are included. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as “image light generation unit”. The left projection optical system 252 is configured by a collimating lens that converts the image light emitted from the left LCD 242 into a light beam in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

  FIG. 4 is an explanatory diagram illustrating a state in which image light is emitted by the image light generation unit. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby changing the illumination light IL emitted from the right backlight 221 into an image. Is modulated into an effective image light PL representing The same applies to the left side. As shown in FIG. 4, the backlight system is used in the present embodiment, but it may be configured to emit image light using a front light system or a reflection system.

A-2. Captured image display processing:
5 and 6 are explanatory diagrams showing the flow of the captured image display process. In the captured image display processing, when a predetermined state of the user's eyelid is detected, the display mode of the image display unit 20 is shifted to an imaging adjustment mode in which the user can capture an outside scene based on the state of the eyelid. To do.

  FIG. 7 is an explanatory diagram showing an example of processing assigned to the open / closed state of the eyelid in the captured image display processing. FIG. 7 shows a list of the display mode transition of the image display unit 20 and the state of the eyelids for performing processing in each display mode. In the present embodiment, for example, the state of the eyelid that shifts the display mode of the image display unit 20 to the imaging adjustment mode is a state in which the right eye that is the user's dominant eye is open and the left eye that is not the dominant eye This is a combination (wink) of the state where the eyelids are closed. In general, the dominant eye can easily see the outside scene compared to the other non-dominant eye, and the other non-dominant eye can easily close the eyelid compared to the dominant eye. For this reason, in the present embodiment, a predetermined combination of eyelid states is convenient for the user. The other eye, which is dominant and non-dominant, may be preset by the user, and in other embodiments, the combination of eyelid states is not limited to the dominant eye, and the right eyelid and the left eyelid It may be a combination of states.

  In the captured image display process, first, the eye imaging cameras 37 and 38 monitor whether a predetermined state of the user's eyelid is detected for a predetermined period (step S310 in FIG. 5). If the predetermined eyelid state is not detected for a predetermined period (step S310: NO), the eye imaging cameras 37 and 38 continue to monitor the eyelid state (step S310). When a predetermined eyelid state is detected for a predetermined period (step S310: YES), the imaging setting unit 175 shifts the display mode of the image display unit 20 to the imaging adjustment mode (step S320). In the imaging adjustment mode, the imaging setting unit 175 generates and stores an image of the outside scene captured by the outside scene imaging camera 61 based on the open / closed state of the user's both eyelids and the gaze direction of the user's right eye. Data and an image displayed on the image display unit 20 are set. In addition, in the display modes of the image display unit 20 described below including the imaging adjustment mode, the eyelids that change the display mode even after the user's both eyelids return to the normal state after shifting to each display mode. As long as the state is not detected, the display mode is not shifted to another display mode.

  Each of FIGS. 8 to 14 is an explanatory diagram showing an example of the visual field VR visually recognized by the user. FIG. 8 shows the visual field VR before shifting to the imaging adjustment mode, and FIG. 9 shows the visual field VR in the imaging adjustment mode. FIG. 8 shows an outside scene SC that is visible to the user through the image display unit 20, and the outside scene SC includes a yacht OB that is a subject. Further, FIG. 8 shows a maximum image display area PN that is the maximum area in which an image generated by the image display unit 20 can be displayed. The solid line indicating the outer frame of the maximum image display area PN is shown for explaining the present embodiment, and is not actually displayed, so that it is not visually recognized by the user. The maximum image display area PN in the present embodiment has 960 horizontal pixels and 540 vertical pixels, but may have a different number of pixels in other embodiments.

  9 is different from FIG. 8 in that an image range outer frame IA representing the range of image data generated by the image display unit 20 is displayed. The image range outer frame IA is a blue line, and is the same as the outer frame of the maximum image display area PN in the initial state after shifting to the imaging adjustment mode. The When the focal length of the optical zoom lens of the outside scene imaging camera 61 is changed in the imaging adjustment mode, the imaging setting unit 175 causes the outer frame of the image range when the focal length is 35 mm or more and 50 mm or less in the imaging adjustment mode. Display IA. Details of the case where the focal length is less than 35 millimeters and more than 50 millimeters will be described later. Note that the image range outer frame IA is not limited to the blue line, but may be another color, the line type may be different, or the range in which the user can visually recognize the region surrounded by the image range outer frame IA. Various modifications are possible.

  When the image range outside frame IA is displayed, the imaging setting unit 175 next designates a partial area of the outside scene SC that the outside scene imaging camera 61 captures based on the accepted operation (step S330 in FIG. 5). . FIG. 10 shows a visual field VR visually recognized by the user when a partial area of the outside scene SC to be imaged is designated in the imaging adjustment mode. FIG. 10 shows a pointer PO1 representing the estimated line-of-sight direction of the user and a captured image VI1 representing a predetermined area centered on the pointer PO1. When the thin state of the right eye is detected in the imaging adjustment mode, the image display unit 20 displays the pointer PO1. If the left eye gaze direction is changed while the pointer PO1 is displayed, the left eye imaging camera 38 and the image determination unit 142 estimate the gaze direction. The image display unit 20 changes the position where the pointer PO1 is displayed based on the estimated line-of-sight direction, and displays an image representing a predetermined area around the pointer PO1 in the captured image VI1.

  When the imaging setting unit 175 changes the focal length while the pointer PO1 is displayed, the image displayed in the captured image VI1 is enlarged or reduced. In this embodiment, when the right eye is thin, the focal length automatically increases gradually, and when reaching a certain focal length, the focal length gradually decreases. When the image determination unit 142 detects a change from the thin state of the right eye to the normal open state, the imaging setting unit 175 sets the focal length at the time when the right eye detects the normal open state. . When the focal length is less than 35 millimeters, the image display unit 20 displays an outside scene image obtained by enlarging a part of the area centered on the pointer PO1 on the captured image VI1. When the focal length exceeds 50 millimeters, the image display unit 20 displays an outside scene image obtained by reducing a part of the area centered on the pointer PO1 on the captured image VI1. When the focal length of the optical zoom lens is 35 mm or more and 50 mm or less, an outside scene image having a size close to the outside scene SC visually recognized by the user is captured. The outside scene SC can be visually recognized in a wider range. Further, when the focal length of the optical zoom lens is less than 35 millimeters or more than 50 millimeters, the captured image VI1 is displayed, so that the user can view the exterior scene SC while confirming the captured exterior scene image. Visible.

  FIG. 11 shows a visual field VR visually recognized by the user when the focal length of the outside scene imaging camera 61 is set to 35 mm or more and 50 mm or less in the imaging adjustment mode. When the focal length is set to 35 mm or more and 50 mm or less, the imaging setting unit 175 hides the captured image VI1 and displays the image range outer frame IA centered on the pointer PO1. In another embodiment, the captured image VI1 may continue to be displayed instead of the image range outer frame IA regardless of the focal length.

  Next, in a state where the pointer PO1 is displayed, the right eye imaging camera 37 detects the state of the eyelid indicating the transition to the imaging area fixing mode in which a part of the area centering on the pointer PO1 is fixed and imaging is continued. Is monitored (step S340 in FIG. 5). When the right eye imaging camera 37 detects that the right eyelid is closed for a predetermined period (step S340: YES), the imaging setting unit 175 changes the display mode of the image display unit 20 to a partial area centered on the pointer PO1. Shifts to the imaging region fixing mode in which imaging is continued (step S350).

  FIG. 12 shows a visual field VR visually recognized when the user faces further upward in the imaging region fixed mode. Based on the orientation of the image display unit 20 detected by the 9-axis sensor 66 and the direction determination unit 161, the imaging setting unit 175 images the direction of the pointer PO1 set by the outside scene imaging camera 61. The orientation of the outside scene imaging camera 61 with respect to is adjusted. When the orientation of the outside scene imaging camera 61 is adjusted, the right eye imaging camera 37 monitors the detection of the eyelid state indicating the operation of imaging the outside scene SC (step S360 in FIG. 5). If the right eyelid indicating the state of the eyelid indicating the imaging operation does not detect the open / closed state twice in a predetermined period (step S360: NO), the right-eye imaging camera 37 continues the eyelid state. Is monitored (step S360). When the predetermined eyelid state is detected in the process of step S360 (step S360: YES), the outside scene imaging camera 61 images the outside scene SC, and the imaging setting unit 175 displays the outside scene displayed in the captured image VI1. Image data representing the SC is generated and stored (step S370). The image display unit 20 displays an image representing the generated image data in a predetermined range in the maximum image display area PN for a certain period, and then returns to the imaging area fixed mode.

  If the eyelid state that shifts to the imaging region fixing mode is not detected in the process of step S340 (step S340: NO), the right-eye imaging camera 37 is the same as the process of step S360 in the imaging adjustment mode. The detection of the state of the eyelid indicating the operation for imaging is monitored (step S365). When the eyelid state indicating the imaging operation is not detected (step S365: NO), the right-eye imaging camera 37 monitors the eyelid state indicating the shift to the imaging region fixing mode (step S340). In the process of step S365, when an eyelid state indicating an imaging operation is detected (step S365: YES), the outside scene imaging camera 61 images the outside scene SC, and the imaging setting unit 175 displays the captured image VI1. Image data representing the displayed outside scene SC is generated and stored (step S370). The image display unit 20 displays an image representing the generated image data in a predetermined range in the maximum image display area PN for a certain period, and then returns to the imaging adjustment mode.

  Next, the eye imaging cameras 37 and 38 monitor the detection of the eyelid state indicating the operation for ending the imaging process (step S380 in FIG. 6). The eye imaging cameras 37 and 38 indicating the operation for ending the imaging process, when a state where both the user's eyelids indicating the operation for ending the imaging process are not detected is detected (step S380: NO), again, step S320. The subsequent processing is performed. When the eye imaging cameras 37 and 38 detect the state of the eyelid indicating an operation for ending the imaging process (step S380: YES), the eyelid state detection indicating the shift to the thumbnail display mode for displaying a list of stored outside scene images is performed. Is monitored (step S390). When the eye imaging cameras 37 and 38 detect three blinks on both eyelids within a predetermined period indicating the state of the eyelid indicating the transition to the thumbnail display mode (step S390: YES), the imaging setting unit 175 Then, the display mode of the image display unit 20 is shifted to the thumbnail display mode (step S400).

  FIG. 13 shows a visual field VR visually recognized by the user in the thumbnail display mode. As illustrated in FIG. 13, the user visually recognizes the captured images IMG1 and IMG2 and the pointer PO2 displayed on the image display unit 20. In the thumbnail display mode, when the eye imaging cameras 37 and 38 detect the eyelid state (wink) in which the user's right eye is closed and the left eye is opened for a predetermined period, the image display unit 20 displays the pointer PO2. When the line of sight of the left eye of the user is changed while the pointer PO2 is displayed, the left eye imaging camera 38 and the image determination unit 142 estimate the line of sight. Based on the estimated change in the viewing direction, the image display unit 20 changes the position at which the pointer PO2 indicating the viewing direction of the user is displayed.

  Next, the left-eye imaging camera 38 monitors detection of an eyelid state indicating an operation for selecting one image from the captured images IMG1 and IMG2 displayed in the maximum image display area PN (step S410 in FIG. 6). . As shown in FIG. 13, when the left eye imaging camera 38 indicating the operation of selecting the captured image IMG1 detects the blinking of the left eye twice in a predetermined period in a state where the pointer PO2 overlaps the captured image IMG1 (step (S410: YES), the imaging setting unit 175 shifts the display mode of the image display unit 20 to the selected image enlargement mode (step S420).

  FIG. 14 shows a visual field VR visually recognized by the user in the selected image enlargement mode. As shown in FIG. 14, in the selected image enlargement mode in which the captured image IMG1 is selected, the captured image IMG1 is enlarged and displayed in the entire area of the maximum image display area PN. When the captured image IMG1 is displayed in an enlarged manner, the eye imaging cameras 37 and 38 monitor the detection of the eyelid state indicating the operation for ending the selected image enlargement mode (step S425 in FIG. 6). When the state where both eyelids are closed for a certain period indicating the state of the eyelids that end the selected image enlargement mode is not detected (step S425: NO), the captured image IMG1 is continuously enlarged and displayed in the maximum image display area PN. The When the state of the eyelid that ends the selected image enlargement mode is detected (step S425: YES), the imaging setting unit 175 shifts the display mode of the image display unit 20 to the thumbnail display mode, and after step S400. Processing is performed.

  When the predetermined eyelid state is not detected in the process of step S390 or the process of step S410 (step S390: NO, step S410: NO), the eye imaging cameras 37 and 38 perform the captured image display process. The detection of the state of the eyelid indicating the operation for ending is performed (step S430). If the state of changing the eyelid from the wink with the left eye closed to the wink with the right eye indicating the end of the captured image display process is not detected (step S430: NO), the process after step S320 in FIG. 5 is performed again. Processing is performed. When the eyelid state indicating the end of the captured image display process is detected in the process of step S430 in FIG. 6 (step S430: YES), the captured image display process ends.

  As described above, in the head-mounted display device 100 according to the present embodiment, the eye imaging cameras 37 and 38 and the image determination unit 142 detect the state of the eyelid as a shielding object that blocks the user's line of sight, and perform imaging settings. The unit 175 operates the image display unit 20 based on the detection result of the eyelid which is a shielding object that blocks the line of sight analyzed by the image determination unit 142. Therefore, in the head-mounted display device 100 according to the present embodiment, image data representing a partial region of the outside scene SC is generated when the user captures an image while viewing the outside scene SC. Therefore, the user can take an image while visually recognizing the outside scene SC without looking through the viewfinder and the like, and can easily measure the timing of taking an image, thereby improving the convenience for the user. In addition, since the user can image the outside scene SC without touching the control unit 10 and the image display unit 20, the user can perform imaging while suppressing camera shake.

  In the head-mounted display device 100 according to the present embodiment, the right eye imaging camera 37 detects the open / closed state of the user's right eyelid, and the left eye imaging camera 38 detects the open / closed state of the user's left eyelid. . The imaging setting unit 175 performs various operations of the head-mounted display device 100 based on the combination of the detected open / closed state of the right eyelid and the left eyelid. Therefore, in the head-mounted display device 100 according to the present embodiment, the user can use the outside scene imaging camera 61 to pick up the outside scene SC without using the hand, and the convenience for the user is improved.

  In the head-mounted display device 100 according to the present embodiment, the eye imaging cameras 37 and 38 and the image determination unit 142 estimate the user's gaze direction, and the image display unit 20 estimates the user's gaze direction. A pointer PO1 representing is displayed. The imaging setting unit 175 designates a partial area of the outside scene SC based on the estimated line-of-sight direction and eyelid detection, and displays an image range outer frame IA or a captured image VI1 indicating the designated partial area. Displayed on the unit 20. Therefore, in the head-mounted display device 100 of the present embodiment, the user can select a part of the image captured by the outside scene imaging camera 61 and generate image data of the selected region without using a hand. , User convenience, convenience and operability are improved.

  In the head-mounted display device 100 according to the present embodiment, the focal length of the outside scene imaging camera 61 is set based on the eyelid state detected by the eye imaging cameras 37 and 38 and the image determination unit 142. The image display unit 20 displays the image range outer frame IA when the focal length is in the range of 35 mm or more and 50 mm or less, and when the focal length is in the other range, the image display unit 20 displays one frame centered on the pointer PO1. An image representing the region of the part is displayed. Therefore, in the head-mounted display device 100 according to the present embodiment, since the user can set the focal length of the image data to be generated without using a hand, the convenience for the user is improved. In addition, when the set focal length is a focal length for generating image data that becomes an outside scene SC similar to the visually recognized outside scene SC, the image display unit 20 displays a part of the area for generating the image data. The outside scene SC is transmitted to the user as indicated by the image range outer frame IA. When the set focal length is a focal length that generates image data that is an outside scene SC different from the outside scene SC that is visually recognized, the image display unit 20 displays an image representing the image data as the image display unit 20. To display. Therefore, since the image displayed by the image display unit 20 is changed in accordance with the comparison between the outside scene SC visually recognized by the user and the outside scene SC of the generated image data, the visibility and operability of the user are improved.

B. Variations:
In addition, this invention is not limited to the said embodiment, It can implement in a various aspect in the range which does not deviate from the summary, For example, the following deformation | transformation is also possible.

B1. Modification 1:
FIG. 15 is an explanatory diagram illustrating an example of the visual field VR visually recognized by the user in the imaging adjustment mode. In FIG. 15, the user visually recognizes when the user tilts his / her head to the right, that is, when the angle of the user's head is tilted with respect to the direction of gravity and the horizontal plane is less than 90 degrees. A field of view VR is shown. The captured image VI2 illustrated in FIG. 15 illustrates an external scene image that is captured when the optical axis of the external scene imaging camera 61 is an axis parallel to a horizontal plane orthogonal to the direction of gravity. In this modification, the direction determination unit 161 and the 9-axis sensor 66 estimate the orientation of the user's head, and the direction determination unit 161 calculates an angle formed by the estimated head direction and the gravity direction. When the angle calculated by the direction determination unit 161 is equal to or larger than the predetermined angle, the imaging setting unit 175 corrects the calculated angle by the calculated angle and displays the captured image VI2 displayed by the image display unit 20. The angle formed between the direction of gravity or the horizontal plane and the direction of the user's head is not limited to the example shown in FIG. 15 and can be variously modified. For example, the image displayed in the captured image VI2 may be corrected when a state in which the user tilts his / her neck up and down more than a predetermined angle is detected. Note that the captured image VI2 is an outside scene image that has been corrected by the calculated angle. However, the outside scene imaging camera 61 does not correct the angle, and the outside scene imaging camera 61 applies to the image display unit 20 based on the calculated angle. By changing the direction, the outside scene imaging camera 61 may image the outside scene SC with the axis parallel to the horizontal plane as the optical axis.

  In the head-mounted display device 100 in this modification, the 9-axis sensor 66 and the direction determination unit 161 detect the orientation of the user's head relative to the gravity direction, and the detected gravity direction and the orientation of the head are made. When the angle is equal to or greater than a predetermined angle, the outside scene SC captured by the outside scene imaging camera 61 is displayed with the axis parallel to the horizontal plane orthogonal to the direction of gravity as the optical axis. Therefore, in the head-mounted display device 100 in this modified example, it is assumed that the user wants to take an image even when the user turns his head and temporarily changes the orientation of the head. Since the outside scene SC parallel to the horizontal plane can be imaged, the convenience of the user is improved.

B2. Modification 2:
In the above embodiment, setting of the display mode of the image display unit 20 and setting of a part of the area in the captured outside scene SC based on the open / closed state of the eyelids detected by the eye imaging cameras 37 and 38, etc. Although the operations have been performed, these operations may be performed based on the detection result of the shielding object that blocks the user's line-of-sight direction other than the open / closed state of the eyelids. For example, even when the user's both eyelids are open, the image display unit 20 and the outside scene imaging camera 61 may be operated based on the color and illuminance of the outside scene SC visually recognized by the user. In this modification, the illuminance sensors 45 and 47 detect the illuminance of the outside scene SC assumed to be visually recognized by the user's right eye and left eye, and an operation is performed based on the detected illuminance value. Even if the user does not close the eyelid, for example, when the surface of the left optical image display unit 28 is covered by the user's hand, the illuminance value detected by the left eye illuminance sensor 47 is detected by the right eye illuminance sensor 45. It becomes smaller than the numerical value of illuminance. In this case, the illuminance processing unit 145 performs the same determination as the state in which the image determination unit 142 in the above embodiment closes the left eyelid, and the imaging setting unit 175 determines the image display unit 20 and the outside scene imaging camera based on the determination. 61 is performed. In addition, the illuminance sensors 45 and 47 detect the illuminance in a certain period, and the illuminance processing unit 145 compares the difference in illuminance at two time points, so that the image display unit is replaced with the open / closed state of the eyelid in the above embodiment. 20 and the outside scene imaging camera 61 may be operated.

  In the head-mounted display device 100 according to this modified example, predetermined ranges from the illuminance sensors 45 and 47 to the places where the surfaces of the optical image display units 26 and 28 are covered with the hands from the right and left eyes of the user, respectively. The imaging setting unit 175 operates the image display unit 20 and the like based on the detection result. Therefore, in the head-mounted display device 100 according to this modification, when the user is tired from an operation caused by a change in the state of the eyelid such as a wink, the user can use the part of the body such as the palm of the eyelid. An operation similar to the change of state can be performed sensuously, and convenience for the user is improved.

  Further, instead of the illuminance sensors 45 and 47, RGB color sensors that detect RGB data in the line-of-sight directions of the right eye and the left eye may be arranged. In this modification, the RGB color sensors arranged in the right eye and the left eye respectively detect RGB data in the range from the user's right eye and left eye to the surface of the optical image display units 26 and 28. To do. The imaging setting unit 175 may perform various operations based on values detected by the RGB color sensors corresponding to the right eye and the left eye. For example, when the surface of the left optical image display unit 28 is covered by the user's hand, the RGB color sensor corresponding to the left eye becomes RGB data representing a color close to black. On the other hand, when the RGB data detected by the RGB color sensor corresponding to the right eye is RGB data representing white under a clear daytime, the wink in which the user's left eyelid in the above embodiment is closed It may be determined in the same way.

  In place of the illuminance sensors 45 and 47, an infrared sensor may be arranged. The light receiving unit of the infrared sensor detects the reflected light reflected by the shield that blocks the line of sight of the infrared ray emitted from the infrared sensor, so that the image display unit 20 and the outside scene can be used instead of the open / closed state of the eyelid in the above embodiment. Operation of the imaging camera 61 may be performed. In this modification, the shielding object can be detected even in a relatively dark place at night, and the convenience for the user is improved.

  In the above embodiment, the focal length of the optical zoom lens is automatically changed when the right eye of the user is thin. However, the focal length adjustment method is not limited to this and can be variously modified. It is. For example, when the surface of the left optical image display unit 28 is covered by a user's hand, the focal length may be set to gradually increase due to the change in illuminance detected by the left eye illuminance sensor 47. Further, when the surface of the right optical image display unit 26 is covered by the user's hand, the focal length may be set to be gradually reduced by the change in illuminance detected by the right eye illuminance sensor 45.

  In the above embodiment, the outside scene imaging camera 61 is formed so as to be able to change the imaging direction with respect to the image display unit 20, but may be fixed to the image display unit 20, or the image display unit 20 may be formed separately. The outside scene imaging camera 61 can change the focal length of the optical zoom lens. However, the outside scene imaging camera 61 may be configured to generate an image in which a part of the imaging range is enlarged by digital zoom.

B3. Modification 3:
In the above embodiment, when the display mode of the image display unit 20 is the imaging region fixed mode, a part of the region included in the captured outside scene SC is specified. However, the image determination unit 142 specifies It is also possible to adopt an aspect in which the target is recognized and the outside scene imaging camera 61 tracks and captures the recognized specific target. In this modified example, when a specific target included in the captured outside scene SC is selected based on the state of the eyelid or the line-of-sight direction, the image determination unit 142 recognizes the specific target. The image determination unit 142 specifies a specific object based on the calculated change by calculating a change in the specific object in the outside scene image at a plurality of time points from the outside scene image captured for a certain period of time. To do. Thereafter, when the specific object is included in the captured outside scene SC, the imaging setting unit 175 sets a partial area of the outside scene SC including the specific object. The outside scene imaging camera 61 changes the direction with respect to the image display unit 20 and images a set part of the area, and the image display unit 20 displays the outside scene SC including a specific object. Further, for example, when the recognized moving speed of the specific object suddenly increases and the specific object is not included in the captured outside scene SC, the imaging setting unit 175 causes the outside scene to be captured to be captured. The position of the specific object is estimated based on the movement change of the specific object when the specific object is included in the SC. The outside scene imaging camera 61 changes the imaging direction to the estimated position of the specific target object, and performs imaging.

  In the head-mounted display device 100 in this modification, the specific object is automatically tracked by image recognition of the specific object without attaching a transmitter or the like to the specific object. Since the image is taken, the convenience and operability of the user are improved. In addition, since the outside scene image including the specific object to be imaged is displayed on the image display unit 20 even if it is different from the outside scene SC to be visually recognized, the user wants to image the outside scene SC including the specific object. In addition, it is not necessary to visually recognize a specific object, and convenience for the user is improved. Further, even when the specific object is out of the outside scene SC to be imaged, the position of the specific object is automatically estimated and the outside scene SC including the specific object is imaged, so that the user Improved convenience.

B4. Modification 4:
In the above embodiment, the outside scene imaging camera 61 stores an outside scene image obtained by imaging the outside scene SC. However, the imaging is not limited to a still image at a specific time point, and may be a moving image. A mode in which the mode of the image display unit 20 is changed into a mode for capturing an image and a mode for recording a moving image based on the detected open / closed state of the eyelid and the estimated user's line-of-sight direction. Also good.

  In the above embodiment, the focal length of the outside scene imaging camera 61 is set according to the open / closed state of the right eye of the user detected by the right-eye imaging camera 37 and the image determination unit 142. However, the present invention is not limited to this and can be variously modified. For example, in the imaging adjustment mode, when the user's finger touches the track pad 36 arranged on the image display unit 20, the pointer PO1 is displayed, and the focal length is set based on the movement of the user's finger. Good. For example, when the user's finger touching the track pad 36 is moved from the right ear toward the front of the user, the image display unit 20 expands a partial area centered on the pointer PO1. An image may be displayed on the captured image VI1. Conversely, when the user's finger touching the track pad 36 is moved from the front of the user toward the right ear, the image display unit 20 reduces a part of the area centered on the pointer PO1. An outside scene image may be displayed on the captured image VI1.

B5. Modification 5:
The outside scene imaging camera 61 may be disposed in the control unit 10 or may be disposed separately from the control unit 10 and the image display unit 20, for example, on an upper part of a helmet worn by the user. .

  In the above embodiment, the operation accepting unit 135 is formed in the control unit 10, but the mode of the operation accepting unit 135 can be variously modified. For example, a mode in which the user interface that is the operation reception unit 135 is provided separately from the control unit 10 may be used. In this case, the operation receiving unit 135 is separate from the control unit 10 in which the power supply 130 and the like are formed. Therefore, the operation receiving unit 135 can be reduced in size, and user operability and portability are improved.

B6. Modification 6:
For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image generation unit may use LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. In the case of the laser retinal projection type, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes.

  Further, for example, the head-mounted display may be a head-mounted display in which the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit does not completely cover the user's eye. Good. The head mounted display may be a so-called monocular type head mounted display.

  The earphone may be an ear-hook type or a headband type, or may be omitted. Further, for example, it may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet.

B7. Modification 7:
The configuration of the head-mounted display device 100 in the above embodiment is merely an example and can be variously modified. For example, one of the direction key 16 and the track pad 14 provided in the control unit 10 may be omitted, in addition to the direction key 16 and the track pad 14, or in place of the direction key 16 and the track pad 14, etc. An operation interface may be provided. Moreover, the control part 10 is a structure which can connect input devices, such as a keyboard and a mouse | mouth, and is good also as what receives an input from a keyboard or a mouse | mouth.

  As the image display unit, instead of the image display unit 20 worn like glasses, another type of image display unit such as an image display unit worn like a hat may be adopted. The earphones 32 and 34 can be omitted as appropriate. Moreover, in the said embodiment, although LCD and a light source are utilized as a structure which produces | generates image light, it replaces with these and you may employ | adopt other display elements, such as an organic EL display. In the above embodiment, the 9-axis sensor 66 is used as a sensor for detecting the movement of the user's head. Instead, one or two of an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor are used. You may use the sensor comprised from. In the above embodiment, the head-mounted display device 100 is a binocular optical transmission type. However, the present invention can be applied to other types of head-mounted display such as a video transmission type and a monocular type. The same applies to the apparatus.

  In the above-described embodiment, the head-mounted display device 100 may guide image light representing the same image to the left and right eyes of the user so that the user can visually recognize the two-dimensional image. It is also possible to guide the user to view the three-dimensional image by guiding image light representing different images to the eyes.

  In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. For example, in the above-described embodiment, the image processing unit 160 and the sound processing unit 170 are realized by the CPU 140 reading and executing a computer program, but these functional units may be realized by a hardware circuit. Good.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

  Moreover, in the said embodiment, as shown to FIG. 1 and FIG. 3, the control part 10 and the image display part 20 are formed as a separate structure, However, About the structure of the control part 10 and the image display part 20, about. However, the present invention is not limited to this, and various modifications are possible. For example, all of the components formed in the control unit 10 may be formed inside the image display unit 20 or a part thereof may be formed. In addition, the power source 130 in the above embodiment may be formed independently and replaceable, or the configuration formed in the control unit 10 may be overlapped and formed in the image display unit 20. For example, the CPU 140 shown in FIG. 3 may be formed in both the control unit 10 and the image display unit 20, or a function performed by the CPU 140 formed in the control unit 10 and the CPU formed in the image display unit 20. May be configured separately.

  Further, the control unit 10 may be built in a personal computer (PC), and the image display unit 20 may be used instead of the PC monitor, or the control unit 10 and the image display unit 20 may be integrated. Further, it may be an embodiment of a wearable computer attached to a user's clothes.

  The present invention is not limited to the above-described embodiments and modifications, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments and the modifications corresponding to the technical features in each form described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Control part 11 ... Decision key 12 ... Illumination part 13 ... Display switching key 14 ... Trackpad 15 ... Luminance switching key 16 ... Direction key 17 ... Menu key 18 ... Power switch 20 ... Image display part 21 ... Right holding part 22 ... Right display drive unit 23 ... Left holding unit 24 ... Left display drive unit 26 ... Right optical image display unit 28 ... Left optical image display unit 30 ... Earphone plug 32 ... Right earphone 34 ... Left earphone 36 ... Track pad 37 ... Right eye imaging Camera (detection unit, gaze direction estimation unit)
38 ... Left-eye imaging camera (detection unit, gaze direction estimation unit)
40 ... Connection part 42 ... Right cord 44 ... Left cord 45 ... Right eye illuminance sensor (detection unit)
46 ... Connecting member 47 ... Left eye illuminance sensor (detection unit)
48 ... Body code 51, 52 ... Transmission unit 53, 54 ... Reception unit 61 ... Outside scene imaging camera (imaging unit)
66 ... 9-axis sensor (direction detection unit)
DESCRIPTION OF SYMBOLS 100 ... Head-mounted display apparatus 110 ... Input information acquisition part 120 ... Memory | storage part 130 ... Power supply 135 ... Operation reception part 140 ... CPU
142 ... Image determination unit (detection unit, gaze direction estimation unit, image recognition unit)
145 ... Illuminance processing unit 150 ... Operating system 160 ... Image processing unit 161 ... Direction determination unit (orientation detection unit)
170: Audio processing unit 175 ... Imaging setting unit 180 ... Interface 190 ... Display control unit 201 ... Right backlight control unit 202 ... Left backlight control unit 211 ... Right LCD control unit 212 ... Left LCD control unit 221 ... Right backlight 222 ... Left backlight 241 ... Right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate VSync ... Vertical synchronization signal HSync ... Horizontal synchronization signal PCLK ... Clock signal Data ... Image data IMG1, IMG2 ... Captured image IA ... Image range Outer frame OB ... Yacht SC ... Outside view RE ... Right eye LE ... Left eye RL ... Right eyelid LL ... Left eyelid EL, ER ... Edge IL ... Illumination light PL ... Image light PN ... Maximum image display area AP ... Tip portion VR ... Field of view VI1, VI2 ... Captured image PO1, PO2 ... Pointer

Claims (14)

A transmissive head-mounted display device,
An imaging unit for imaging an outside scene;
A illumination sensor that detects a shield that blocks the line of sight used for person, detects a right front of eyes index value indicating irradiation of the outside scene and the left front of eyes index value that each viewing the right and left eyes of a user An illuminance sensor ;
An imaging operation unit that operates the imaging unit based on the detected right-eye front index value and the left-eye front index value, and an illuminance value as the right-eye front index value detected by the illuminance sensor; An imaging operation unit for operating the imaging unit based on a difference from an illuminance value as the left-eye front index value ;
A head-mounted display device comprising: an image display unit that guides region image light, which is image light indicating an imaging region, to the user's eyes and transmits an outside scene while being mounted on the user's head . A transmissive head-mounted display device,
An imaging unit for imaging an outside scene;
An RGB color sensor that detects an obstruction that blocks a user's line of sight, and detects an anterior right eye index value and an anterior left eye index value that indicate colors in the respective eye gaze directions of the user's right eye and left eye A sensor ,
An imaging operation unit for operating the imaging unit based on the detected right-eye front index value and the left-eye front index value, and an RGB value as the right-eye front index value detected by the RGB color sensor And an imaging operation unit that operates the imaging unit based on a difference between the RGB value as the left-eye front index value,
A head-mounted display device comprising: an image display unit that guides region image light, which is image light indicating an imaging region, to the user's eyes and transmits an outside scene while being mounted on the user's head . The head-mounted display device according to claim 1 or 2 , further comprising:
A gaze direction estimation unit for estimating the gaze direction of the user,
The image display unit guides the eye position image light, which is image light representing a position corresponding to the line-of-sight direction with respect to the captured outside scene, to the user's eyes,
The imaging operation unit sets a partial region of the outside scene to be imaged based on the estimated line-of-sight direction and the combination of the right-eye front index value and the left-eye front index value,
The head-mounted display device, wherein the region image light is image light indicating the set partial region. The head-mounted display device according to claim 3 ,
The operation includes adjustment of the focal length of the lens of the imaging unit,
When the focal distance is included in a predetermined distance, the regional image light is generated in a peripheral portion of the partial area, and the focal distance is not included in the predetermined distance. A head-mounted display device that generates outside scene image light representing an outside scene image of the partial area from the area image light. The head-mounted display device according to claim 4 , further comprising:
An image recognizing unit for recognizing a specific target included in the photographed outside scene based on the operation;
When the specific target is recognized in the captured outside scene, the imaging operation unit sets a partial area of the captured external scene based on the specific target,
The head-mounted display device, wherein the outside scene image light is image light representing the set partial area. The head-mounted display device according to claim 5 ,
When the specific target is not recognized in the outside scene to be imaged, the imaging operation unit is based on a change in the position of the specific target when the specific target is recognized in the outside scene to be captured. Estimating the direction from the imaging unit to the specific object;
The imaging unit is a head-mounted display device that images an estimated direction. The head-mounted display device according to any one of claims 1 to 6 , further comprising:
It has an orientation detection unit that detects the orientation of the user's head relative to the direction of gravity.
When the detected angle between the head direction and the gravitational direction is equal to or greater than a predetermined value, the region image light is transmitted on an axis parallel to the horizontal plane perpendicular to the gravitational direction based on the angle. A head-mounted display device that generates image light indicating all or a part of an outside scene imaged as an axis. A control method for a transmissive head-mounted display device,
The illumination sensor, and detects the right front of the eyes index value and the left front of eyes index value indicating the illuminance of the outside scene which is, respectively it visually for the right eye and the left eye of the user,
The operation of the imaging unit included in the head-mounted display device is performed based on the difference between the illuminance value as the right-eye front index value detected by the illuminance sensor and the illuminance value as the left-eye front index value. A method for controlling a head-mounted display device. A control method for a transmissive head-mounted display device,
  The RGB color sensor detects a right-eye pre-index value and a left-eye pre-index value indicating colors in the respective eye gaze directions of the user's right eye and left eye,
  An operation of an imaging unit included in the head-mounted display device is performed based on a difference between an RGB value as the right-eye front index value detected by the RGB color sensor and an RGB value as the left-eye front index value. A method for controlling a head-mounted display device comprising: A transmissive head-mounted display device,
An imaging unit for imaging an outside scene;
A illumination sensor that detects a shield that blocks the line of sight used for business, the illuminance detecting the right front of eyes index value indicating the illuminance of the outside scene and the left front of eyes index values each of the right and left eyes of the user to visually recognize A sensor ,
An imaging operation unit that operates the imaging unit based on the detection result of the shielding object, and the illuminance value as the right-eye front index value and the illuminance value as the left-eye front index value detected by the illuminance sensor; An imaging operation unit for operating the imaging unit based on the difference between
An image display unit that guides area image light indicating an imaging area to the user's eyes and transmits an outside scene in a state of being mounted on the user's head;
A motion sensor disposed in the image display unit;
A gaze direction estimation unit that estimates the gaze direction of the user using a detection signal from the motion sensor,
The image display unit is a head-mounted type that guides image light representing a line-of-sight display indicating the line-of-sight direction and image light representing a region display indicating a region centered on the line-of-sight direction display to a user's eyes. Display device. A transmissive head-mounted display device,
  An imaging unit for imaging an outside scene;
  An RGB color sensor that detects an obstruction that blocks a user's line of sight, and detects an anterior right eye index value and an anterior left eye index value that indicate colors in the respective eye gaze directions of the user's right eye and left eye A sensor,
  An imaging operation unit for operating the imaging unit based on the detection result of the shielding object, wherein the RGB value as the right eye front index value and the RGB value as the left eye front index value detected by the RGB color sensor An imaging operation unit for operating the imaging unit based on the difference between
  An image display unit that guides area image light indicating an imaging area to the user's eyes and transmits an outside scene in a state of being mounted on the user's head;
  A motion sensor disposed in the image display unit;
  A gaze direction estimation unit that estimates the gaze direction of the user using a detection signal from the motion sensor,
  The image display unit is a head-mounted type that guides image light representing a line-of-sight display indicating the line-of-sight direction and image light representing a region display indicating a region centered on the line-of-sight direction display to a user's eyes. Display device. A transmissive head-mounted display device according to claim 10 or 11 ,
The imaging operation unit sets a partial region of the outside scene to be imaged based on the estimated line-of-sight direction and the detection result of the shielding object,
The head-mounted display device, wherein the region image light is image light indicating the set partial region. A transmissive head-mounted display device,
An imaging unit for imaging an outside scene;
A illumination sensor that detects a shield that blocks the line of sight used for business, the illuminance detecting the right front of eyes index value indicating the illuminance of the outside scene and the left front of eyes index values each of the right and left eyes of the user to visually recognize A sensor ,
An imaging operation unit that operates the imaging unit based on the detection result of the shielding object, and the illuminance value as the right-eye front index value and the illuminance value as the left-eye front index value detected by the illuminance sensor; An imaging operation unit for operating the imaging unit based on the difference between
An image display unit that guides area image light indicating an imaging area to the user's eyes and transmits an outside scene in a state of being mounted on the user's head;
A motion sensor disposed in the image display unit,
When the angle formed by the direction of the head detected by the motion sensor and the direction of gravity is equal to or greater than a predetermined value, the image display unit causes the region image light to move in the direction of gravity based on the angle. A head-mounted display device that generates, as image light, an entire or partial region of an outside scene that is imaged using an axis parallel to an orthogonal horizontal plane as an optical axis. A transmissive head-mounted display device,
  An imaging unit for imaging an outside scene;
  An RGB color sensor that detects an obstruction that blocks a user's line of sight, and detects an anterior right eye index value and an anterior left eye index value that indicate colors in the respective eye gaze directions of the user's right eye and left eye A sensor,
  An imaging operation unit for operating the imaging unit based on the detection result of the shielding object, wherein the RGB value as the right eye front index value and the RGB value as the left eye front index value detected by the RGB color sensor An imaging operation unit for operating the imaging unit based on the difference between
  An image display unit that guides area image light indicating an imaging area to the user's eyes and transmits an outside scene in a state of being mounted on the user's head;
  A motion sensor disposed in the image display unit,
  When the angle formed by the direction of the head detected by the motion sensor and the direction of gravity is equal to or greater than a predetermined value, the image display unit causes the region image light to move in the direction of gravity based on the angle. A head-mounted display device that generates, as image light, an entire or partial region of an outside scene that is imaged using an axis parallel to an orthogonal horizontal plane as an optical axis.

Images