JP6369583B2 - Head-mounted display device and method for controlling head-mounted display device - Google Patents

Description

  The present invention relates to a head-mounted display device.

  A technique called augmented reality (AR) that adds information to a real environment using a computer is known. As methods for realizing augmented reality, a method based on image recognition and a method based on a pass-through method are known. In the case of a method based on image recognition, for example, information for additional presentation is generated by recognizing an image of an outside scene captured by a WEB camera or the like. In the case of a method based on the pass-through method, additional presentation information is generated using current position information acquired by, for example, GPS and azimuth information acquired by, for example, an electronic compass. In the non-transmissive head-mounted display device in which the user's field of view is blocked when the head-mounted display device is mounted, the image of the outside scene and the additional presentation information generated as described above Is displayed on the liquid crystal display. Thereby, the user can experience augmented reality. On the other hand, in a transmissive head-mounted display device in which the user's view is not blocked with the head-mounted display device mounted, only the additional presentation information generated as described above is displayed on the liquid crystal display. Is done. The user can experience augmented reality by visually recognizing both the additional presentation information displayed as a virtual image via the liquid crystal display and the actual outside scene that is seen through the lens in front of the user. .

  A technique is known that uses the augmented reality as described above to guide the user about work contents and work procedures to be performed by the user. For example, in Patent Document 1, when an animation is displayed on a translucent liquid crystal display panel included in a translucent head-mounted display device, a work object itself that the user visually recognizes via the liquid crystal display panel is disclosed. On the other hand, a technique for superimposing animation is described. Patent Document 2 describes a technique for selecting information related to work performed by a user in a retinal scanning head-mounted display device and enabling an image based on the information to be visually recognized by the user. Has been.

Japanese Patent No. 3735086 JP 2011-114781 A

  With the techniques described in Patent Documents 1 and 2, it is possible to support the work performed by the user. However, when a user tries to start work with an animation or image displayed as a virtual image, the virtual image displayed in front of the user's eyes obstructs the user's view and hinders the work. There was a problem of fear of becoming.

  Such a problem is not limited to work support using augmented reality. Such a problem is a problem common to all virtual image displays in a head-mounted display device in which a user such as a transmission type or a retinal scanning type can visually recognize a virtual image and an outside scene.

  For this reason, a head-mounted display device capable of improving the visibility of the outside scene according to the user's intention has been desired.

SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.
A first aspect of the present invention is a head-mounted display device in which a user can visually recognize a virtual image and an outside scene, and an image display unit that allows the user to visually recognize the virtual image, and infrared rays are emitted from the light emitting unit to the eyes. Then, the reflected light is received by the light receiving unit, and the output signal indicating the state where the infrared ray hits one of the white eye, the eyelid, and the black eye is received from the light receiving unit, and the output signal indicating the same state is received a predetermined number of times. When receiving continuously, the detection unit detects the user's line-of-sight direction from the state, and the display of the virtual image by the image display unit is given priority to the outside scene according to the detected line-of-sight direction. The display switching unit that switches between an outside scene priority display and a virtual image priority display that prioritizes the virtual image, and the detection by the detection unit depending on whether the usage state of the head-mounted display device is dynamic or static. A change section for changing the sensitivity. That. The change unit repeatedly detects the movement of the head on which the head-mounted display device is mounted or the current position of the head-mounted display device, and changes the amount of movement of the head or the current position. When it is detected that the amount of change is greater than or equal to a predetermined first threshold, the usage state of the head-mounted display device is estimated to be dynamic, and the sensitivity of the detection is set low, and the head When the movement amount or the change amount of the current position is detected to be less than or equal to a second threshold value that is smaller than the first threshold value , the usage state of the head-mounted display device is estimated to be static, and Set the detection sensitivity high. When the detection sensitivity is high, the detection unit sets the number of times less than when the detection sensitivity is low, or when the detection unit receives the output signal from the light receiving unit. Set the interval short.
According to this head-mounted display device, the user's line-of-sight direction detected by the detection unit reflects what the user intends to see, that is, the user's intention. Then, the display switching unit, according to the line-of-sight direction, displays the virtual image by the image display unit between an outside scene priority display that prioritizes the outside scene and improves the visibility of the outside scene, and a virtual image priority display that prioritizes the virtual image. Switch between. As a result, in the head-mounted display device, the visibility of the outside scene can be improved according to the user's intention. The changing unit can change the sensitivity of detection by the detecting unit in accordance with the usage state of the head-mounted display device. For this reason, the display switching unit changes whether to finely switch between the external scene priority display and the virtual image priority display or roughly switches between the external scene priority display and the virtual image priority display according to the usage state of the head-mounted display device. be able to. As a result, the convenience of the user of the head-mounted display device can be improved.

(1) According to an aspect of the present invention, a head-mounted display device that allows a user to visually recognize a virtual image and an outside scene is provided. The head-mounted display device includes: an image display unit that allows the user to visually recognize the virtual image; a detection unit that detects the user's line-of-sight direction; and the image display unit according to the detected line-of-sight direction. A display switching unit that switches the display of the virtual image between an external scene priority display that prioritizes the outside scene and a virtual image priority display that prioritizes the virtual image. According to this type of head-mounted display device, the user's line-of-sight direction detected by the detection unit reflects what the user wants to see, that is, the user's intention. Then, the display switching unit, according to the line-of-sight direction, displays the virtual image by the image display unit between an outside scene priority display that prioritizes the outside scene and improves the visibility of the outside scene, and a virtual image priority display that prioritizes the virtual image. Switch between. As a result, in the head-mounted display device, the visibility of the outside scene can be improved according to the user's intention.

(2) The head-mounted display device of the above aspect may further include a change unit that changes the sensitivity of the detection by the detection unit in accordance with a usage state of the head-mounted display device. According to the head-mounted display device of this aspect, the changing unit can change the sensitivity of detection by the detection unit according to the usage state of the head-mounted display device. For this reason, the display switching unit changes whether to finely switch between the external scene priority display and the virtual image priority display or roughly switches between the external scene priority display and the virtual image priority display according to the usage state of the head-mounted display device. be able to. As a result, the convenience of the user of the head-mounted display device can be improved.

(3) In the head mounted display device according to the above aspect; the changing unit changes the sensitivity of the detection to be low when the usage state is dynamic, and the detection when the usage state is static. You may change the sensitivity of. According to this form of the head-mounted display device, the changing unit changes the detection sensitivity of the detecting unit to a low level in a dynamic usage state in which the user's body, head, and line of sight often change rapidly. . For this reason, the excessive change of the gaze direction detected by the detection part can be suppressed. The change unit changes the detection sensitivity of the detection unit to a high level in a static usage state where the user's body, head, and line of sight do not change much. For this reason, the detection unit can detect even a small change in the line-of-sight direction.

(4) The head-mounted display device of the above aspect further includes: a motion detection unit that detects the movement of the user's head; and the change unit uses the detected head movement to perform the use It may be estimated whether the state is dynamic or static. According to the head-mounted display device of this aspect, the changing unit estimates whether the usage state is dynamic or static using the movement of the user's head detected by the motion detection unit. The sensitivity of detection by the detection unit can be automatically changed.

(5) In the head-mounted display device according to the above aspect; the line-of-sight direction includes at least a first direction in which the user's line of sight faces a substantially forward direction and the user's line of sight in a substantially vertical direction. And a second direction facing. According to the head-mounted display device of this embodiment, at least the first direction in which the user's line of sight faces the front direction and the user's line of sight in the substantially vertical direction in the line-of-sight direction. It can be made clear that the second direction is included.

(6) In the head-mounted display device of the above aspect; the display switching unit; has at least two operation modes of a first operation mode and a second operation mode; in the first operation mode When the line-of-sight direction is the first direction, the virtual image priority display is used, and when the line-of-sight direction is the second direction, the outside scene priority display is performed; in the second operation mode, the line-of-sight direction is displayed. May be the outside scene priority display when the first direction is set, and the virtual image priority display may be set when the line-of-sight direction is the second direction. According to the head-mounted display device of this aspect, the display switching unit performs virtual image priority display when the line-of-sight direction is the first direction in the first operation mode, and outside scenes when the line-of-sight direction is the second direction. Since priority display is used, the visibility of the outside scene can be improved when the user turns his gaze downward. In addition, in the second operation mode, the display switching unit performs the outside scene priority display when the line-of-sight direction is the first direction, and the virtual image priority display when the viewing direction is the second direction. The visibility of the outside scene can be improved when the line of sight is directed. Thus, since the display switching unit has a plurality of operation modes with different gaze directions that improve the visibility of the outside scene, the user can operate the display switching unit in an operation mode according to preference. As a result, the convenience of the user of the head-mounted display device can be improved.

(7) In the head-mounted display device according to the above aspect; the display switching unit may be configured to hide the virtual image, reduce the display size of the virtual image, reduce the display of the virtual image, or reduce the virtual image. The outside scene priority display may be performed by using any means of whether the display location is near the end of the user's visual field. According to the head-mounted display device of this aspect, the display switching unit determines whether to hide the virtual image, reduce the display size of the front virtual image, or reduce the display of the virtual image in the external scene priority display. Either means can be used. If the virtual image is not displayed, the visibility of the outside scene can be improved. If the display size of the virtual image is reduced, the visibility of the outside scene can be improved in other portions where the virtual image is not displayed. If the display of the virtual image is thinned, it is possible to improve the visibility of the outside scene that the user sees through the virtual image. If the display location of the virtual image is in the vicinity of the end of the visual field of the user, the visibility of the outside scene can be improved in other portions where the virtual image is not displayed.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with a new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including a part or all of the three elements of the image display unit, the detection unit, and the display switching unit. That is, this apparatus may or may not have an image display unit. Moreover, this apparatus may have a detection part and does not need to have it. Further, this device may or may not have a display switching unit. Such a device can be realized, for example, as a head-mounted display device, but can also be realized as a device other than the head-mounted display device. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device.

  The present invention can be realized in various modes. For example, a head-mounted display device and a head-mounted display device control method, a head-mounted display system, these methods, devices, or The present invention can be realized in the form of a computer program for realizing the function of the system, a recording medium on which the computer program is recorded, or the like.

It is explanatory drawing which shows schematic structure of the head mounted display apparatus in one Embodiment of this invention. It is a block diagram which shows the composition of a head mount display functionally. It is explanatory drawing which shows an example of the virtual image visually recognized by the user. It is a flowchart which shows the procedure of a display switching process. It is explanatory drawing explaining a user's gaze direction. It is explanatory drawing explaining the detection method of a gaze direction. It is explanatory drawing for demonstrating the virtual image priority display in the case of hand work mode, and an outside scene priority display. It is explanatory drawing for demonstrating the virtual image priority display in the case of a live condition mode, and an outside scene priority display. It is explanatory drawing which shows the variation of a virtual image priority display. It is a flowchart which shows the procedure of the additional process 1. It is a block diagram which shows functionally the structure of the head mounted display in 2nd Embodiment. It is a flowchart which shows the procedure of a state estimation process. It is explanatory drawing for demonstrating the change method of the detection sensitivity of a gaze direction. It is explanatory drawing which shows the structure of the external appearance of the head mounted display in a modification.

A. First embodiment:
A-1. Configuration of head mounted display device:
FIG. 1 is an explanatory diagram showing a schematic configuration of a head-mounted display device according to an embodiment of the present invention. The head-mounted display device 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The head mounted display 100 according to the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene.

  The head-mounted display 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when attached to the user's head, and a control unit (controller) 10 that controls the image display unit 20.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a light emitting unit. 61 and the light receiving part 62 are included. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 is disposed inside the right holding unit 21, in other words, on the side facing the user's head when the user wears the image display unit 20. Further, the left display driving unit 24 is disposed inside the left holding unit 23. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

  The display driving unit includes a liquid crystal display (hereinafter referred to as “LCD”) 241, 242, projection optical systems 251, 252, and the like (see FIG. 2). Details of the configuration of the display driving unit will be described later. The optical image display unit as an optical member includes light guide plates 261 and 262 (see FIG. 2) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display driving unit to the user's eyes. The light control plate is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 (the side opposite to the user's eye side). The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. Further, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The light emitting unit 61 is disposed inside the image display unit 20 and at a position facing the user's eyebrow when the user wears the image display unit 20. The light emitting unit 61 is a light emitter that emits infrared light, which is invisible light. For example, an infrared light emitting diode can be used. The light receiving unit 62 is disposed inside the image display unit 20 at a position facing the user's eyebrow when the user wears the image display unit 20. The light receiving unit 62 is a light receiving body that receives infrared light which is invisible light, and for example, an infrared photodiode is used.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48 connected to the control unit 10, a right cord 42 and a left cord 44 in which the main body cord 48 branches into two, and a connecting member 46 provided at the branch point. . The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24. The connecting member 46 is provided with a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. Each end of the main body cord 48 opposite to the connecting member 46 and the control unit 10 are provided with connectors (not shown) that are fitted to each other. The connector of the main body cord 48 and the control unit 10 The control unit 10 and the image display unit 20 are connected to or disconnected from each other by the fitting / releasing of the connector. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for controlling the head mounted display 100. The control unit 10 includes a lighting unit 12, a touch pad 14, a cross key 16, and a power switch 18. The lighting unit 12 notifies the operation state of the head mounted display 100 (for example, ON / OFF of the power supply) by the light emission mode. For example, an LED (Light Emitting Diode) can be used as the lighting unit 12. The touch pad 14 detects a contact operation on the operation surface of the touch pad 14 and outputs a signal corresponding to the detected content. As the touch pad 14, various touch pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The cross key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power state of the head mounted display 100 by detecting a slide operation of the switch.

  FIG. 2 is a block diagram functionally showing the configuration of the head mounted display 100. The control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a wireless communication unit 132, a GPS module 134, a CPU 140, an interface 180, and transmission units (Tx) 51 and 52. The parts are connected to each other by a bus (not shown).

The input information acquisition unit 110 acquires a signal corresponding to an operation input to the touch pad 14, the cross key 16, the power switch 18, and the like, for example. The storage unit 120 includes a ROM, a RAM, a DRAM, a hard disk, and the like. The storage unit 120 stores an operation mode 122. The operation mode 122 stores a predetermined “operation mode” of the display switching unit 144. The display switching unit 144 of the present embodiment has the following two operation modes.
(1) A hand work mode in which virtual image priority display (details will be described later) is set when the user's line-of-sight direction is a substantially forward direction, and outside scene priority display (details will be described later) when the user is in a substantially vertical direction. The hand work mode corresponds to the “first operation mode” in the claims.
(2) A live mode in which the outside scene priority display is set when the user's line-of-sight direction is a substantially forward direction, and the virtual image priority display is set when the user's line-of-sight direction is a substantially vertical direction. The actual mode corresponds to the “second operation mode” in the claims.

  The power supply 130 supplies power to each part of the head mounted display 100. As the power supply 130, for example, a secondary battery can be used. The wireless communication unit 132 performs wireless communication with other devices in accordance with a predetermined wireless communication standard such as a wireless LAN or Bluetooth. The GPS module 134 detects its current position by receiving a signal from a GPS satellite.

  The CPU 140 reads out and executes a computer program stored in the storage unit 120, whereby an operating system (OS) 150, an image processing unit 160, an audio processing unit 170, a display control unit 190, a detection unit 142, and a display switching unit. It functions as 144. The detection unit 142 detects the direction of the user's line of sight (hereinafter also referred to as “line of sight”) using the output signal from the light receiving unit 62. Details will be described later. The display switching unit 144 executes display switching processing. The display switching process is a display of a virtual image by the image display unit 20 according to the line-of-sight direction detected by the detection unit 142, between an outside scene priority display giving priority to the outside scene and a virtual image wired display in which the virtual image is wired. It is a process to switch by. Details will be described later. The detection unit 142, the light emitting unit 61, and the light receiving unit 62 correspond to a “detection unit” in the claims.

  The image processing unit 160 generates a signal based on content (video) input via the interface 180 or the wireless communication unit 132. Then, the image processing unit 160 supplies the generated signal to the image display unit 20 via the connection unit 40. The signal supplied to the image display unit 20 differs between the analog format and the digital format. In the case of the analog format, the image processing unit 160 generates and transmits a clock signal PCLK, a vertical synchronization signal VSync, a horizontal synchronization signal HSync, and image data Data. Specifically, the image processing unit 160 acquires an image signal included in the content. For example, in the case of a moving image, the acquired image signal is an analog signal generally composed of 30 frame images per second. The image processing unit 160 separates a synchronization signal such as a vertical synchronization signal VSync and a horizontal synchronization signal HSync from the acquired image signal, and generates a clock signal PCLK by a PLL circuit or the like according to the period. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like. The image processing unit 160 stores the converted digital image signal in the DRAM in the storage unit 120 for each frame as image data Data of RGB data. On the other hand, in the digital format, the image processing unit 160 generates and transmits a clock signal PCLK and image data Data. Specifically, when the content is in digital format, the clock signal PCLK is output in synchronization with the image signal, so that the generation of the vertical synchronization signal VSync and the horizontal synchronization signal HSync and the A / D conversion of the analog image signal are performed. It becomes unnecessary. The image processing unit 160 performs image processing such as resolution conversion processing, various tone correction processing such as adjustment of luminance and saturation, and keystone correction processing on the image data Data stored in the storage unit 120. May be executed.

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. Note that the image data Data transmitted through the transmission unit 51 is also referred to as “right-eye image data Data1”, and the image data Data transmitted through the transmission unit 52 is also referred to as “left-eye image data Data2”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control unit 10 and the image display unit 20.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 uses the control signal to turn on / off the right LCD 241 by the right LCD control unit 211, turn on / off the right backlight 221 by the right backlight control unit 201, and control the left LCD. Each of the right display drive unit 22 and the left display drive unit 24 is controlled by individually controlling ON / OFF driving of the left LCD 242 by the unit 212 and ON / OFF driving of the left backlight 222 by the left backlight control unit 202. Controls the generation and emission of image light. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light. Further, the display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals for the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and transmits the acquired audio signal to a speaker (not shown) in the right earphone 32 and a speaker (not shown) in the left earphone 34 connected to the connecting member 46. To supply. For example, when a Dolby (registered trademark) system is adopted, processing is performed on an audio signal, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, respectively.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer PC, a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as the right optical image display unit 26, a left light guide plate 262 as the left optical image display unit 28, and a light emitting unit. 61, a light receiving unit 62, and a 9-axis sensor 66.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, it functions as a motion detection unit that detects the movement of the user's head. . Here, the movement of the head includes changes in the speed, acceleration, angular velocity, direction, and direction of the head. The nine-axis sensor 66 corresponds to a “motion detector” in the claims.

  The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, a right LCD control unit 211 that functions as a display element, and a right An LCD 241 and a right projection optical system 251 are included. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are also collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data1 input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby converting the illumination light emitted from the right backlight 221 into an image. Modulate into effective image light to represent. In this embodiment, the backlight method is adopted, but image light may be emitted using a front light method or a reflection method.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The optical image display unit can use any method as long as a virtual image is formed in front of the user's eyes using image light. For example, a diffraction grating or a transflective film may be used. good.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. That is, the left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, and a left LCD control unit 212 that functions as a display element. And a left LCD 242 and a left projection optical system 252.

  FIG. 3 is an explanatory diagram illustrating an example of a virtual image visually recognized by the user. FIG. 3 illustrates the visual field VR of the user. As described above, the image light guided to both eyes of the user of the head mounted display 100 forms an image on the retina of the user, so that the user visually recognizes the virtual image VI. In the example of FIG. 3, the virtual image VI is an OS standby screen of the head mounted display 100. Further, the user views the outside scene SC through the right optical image display unit 26 and the left optical image display unit 28. As described above, the user of the head mounted display 100 of the present embodiment can see the virtual image VI and the outside scene SC behind the virtual image VI for the portion of the visual field VR where the virtual image VI is displayed. In addition, the portion of the visual field VR where the virtual image VI is not displayed can be seen through the optical scene display portion and directly through the outside scene SC.

A-2. Display switching process:
FIG. 4 is a flowchart showing the procedure of the display switching process. The display switching process is a process of switching the virtual image display by the image display unit 20 between an external scene priority display that prioritizes the outside scene and a virtual image wired display in which the virtual image is wired according to the user's line-of-sight direction. . The start trigger of the display switching process can be arbitrarily determined. For example, the start trigger may be that the head-mounted display 100 is activated, in other words, that the power is turned on. Further, for example, a processing start request from the OS 150 or a specific application may be used as a start trigger. When the display switching process is called and executed from a specific application, the display switching process operates as a subroutine of the specific application.

  In the display switching process, the display switching unit 144 checks the operation mode stored in the operation mode 122 of the storage unit 120 (step S102). The display switching unit 144 shifts the process to step S110 when the operation mode is “hand work mode”, and shifts the process to step S120 when the operation mode is “actual mode” (step S104).

  When the operation mode is the hand work mode (step S104: hand work), the display switching unit 144 causes the detection unit 142 to detect the user's line-of-sight direction (step S110).

A-2-1. Gaze direction detection:
FIG. 5 is an explanatory diagram for explaining the viewing direction of the user. FIG. 5A shows the state of the user's right eye RE and left eye LE when the line-of-sight direction is the forward direction. As shown in the figure, when the user's line of sight is directed substantially in the forward direction, in other words, in the forward direction, the iris RI of the right eye RE is positioned substantially at the center of the right eye RE. The same applies to the iris LI of the left eye LE. The forward direction as shown in FIG. 5A corresponds to the “first direction” in the claims.

  FIG. 5B shows the state of the user's right eye RE and left eye LE when the line-of-sight direction is the vertical direction. As shown in the figure, when the user's line of sight is in a substantially vertical direction, in other words, facing down, the eyelid RLi of the right eye RE is in a lowered state and is in the center of the right eye RE. The same applies to the eyelid LLi of the left eye LE. The vertical direction as shown in FIG. 5B corresponds to the “second direction” in the claims.

  FIG. 5C shows a state of the right eye RE and the left eye LE of the user when the line-of-sight direction is the lateral direction. As shown in the figure, when the user's line of sight is in a substantially lateral direction, in other words, facing the side, the iris RI of the right eye RE is located at the end of the right eye RE, and the right eye RE A white eye RW is located in the center. The same applies to the iris LI of the left eye LE.

  Note that the user's line-of-sight direction (forward direction, vertical direction, lateral direction) described above means a general direction in which the user's line-of-sight is facing, and allows some deviation.

  FIG. 6 is an explanatory diagram for explaining a method of detecting the line-of-sight direction. FIG. 6A is an explanatory diagram for describing the light emitting unit 61 and the light receiving unit 62. FIG. 6A shows an overview when the right eye RE of the user wearing the head mounted display 100 is viewed from the side. The light emitting unit 61 and the light receiving unit 62 are disposed inside the image display unit 20 and at positions facing the user's eyebrows when the user wears the image display unit 20. The light emitting unit 61 emits infrared light, which is invisible light, toward the center of the user's right eye RE and the center of the left eye LE. The infrared light emitted from the light emitting unit 61 is reflected to the right eye RE and the left eye LE of the user. The light receiving unit 62 receives reflected light reflected by the user's right eye RE and left eye LE, and transmits an output signal corresponding to the magnitude of the received infrared light to the detection unit 142.

  FIG. 6B is an explanatory diagram for explaining how the detection unit 142 detects the user's line-of-sight direction based on the output signal from the light receiving unit 62. The infrared reflectance of the user's right eye RE and left eye LE differs depending on whether the infrared ray hits the iris (black eye), the infrared ray hits the eyelid, or the infrared ray hits the white eye. . Specifically, the reflectance of infrared rays is lowest when the infrared rays hit the iris, and then increases in the order of eyelids and white eyes. For this reason, the detection value indicated by the output signal from the light receiving unit 62 increases in the order of iris, eyelid, and white.

  Therefore, when the detection value indicated by the output signal from the light receiving unit 62 is equal to or greater than the first threshold Th1, the detection unit 142 determines that the infrared ray has hit the white eye. Similarly, when the detection value indicated by the output signal from the light receiving unit 62 is smaller than the first threshold Th1 and larger than the second threshold Th2, the detection unit 142 determines that the infrared ray has hit the eyelid. Similarly, when the detection value indicated by the output signal from the light receiving unit 62 is equal to or less than the second threshold Th2, the detection unit 142 determines that the infrared ray has hit the iris (black eye).

  The detection unit 142 performs the above determination at each time interval t at which the output signal from the light receiving unit 62 is received, and detects that the state is the user's line-of-sight direction when the same state continues three or more times. . That is, when it is determined that the infrared ray hits the white eye continuously for three or more times, the user's line-of-sight direction detected by the detection unit 142 is the “lateral direction”. Similarly, when it is determined that the infrared ray hits the eyelid continuously three or more times, the user's line-of-sight direction detected by the detection unit 142 is “vertical direction”. Similarly, when it is determined that the infrared ray has hit the iris continuously three or more times, the user's line-of-sight direction detected by the detection unit 142 is the “forward direction”.

  In the example of FIG. 6B, the detection unit 142 detects that the user's line-of-sight direction has changed in the order of eyelid, iris (black eye), and eyelid. Note that the portion (white eye determination) illustrated by the broken line in FIG. 6B is ignored.

  In step S112 of FIG. 4, the display switching unit 144 determines whether or not the user's line-of-sight direction is the vertical direction. When the user's line-of-sight direction is not the vertical direction (step S112: NO), in other words, when the line-of-sight direction is the lateral direction or the forward direction, the display switching unit 144 displays the virtual image displayed by the image display unit 20 as a virtual image. Prioritized display (step S114), and then the process proceeds to step S110. On the other hand, when the user's line-of-sight direction is the vertical direction (step S112: YES), the display switching unit 144 sets the virtual image display by the image display unit 20 to the outside scene priority display (step S116), and thereafter the process is performed in step S110. Transition to.

A-2-2. Virtual image priority display and external scene priority display:
In the present embodiment, “virtual image priority display” means performing display with priority to the outside scene by improving the visibility of the outside scene SC compared to the virtual image VI during display of the virtual image VI (FIG. 3). means. “Outside scene priority display” means that during the display of the virtual image VI, display of the virtual image is prioritized by improving the visibility of the virtual image VI compared to the external scene SC.

  FIG. 7 is an explanatory diagram for describing virtual image priority display and outside scene priority display in the hand work mode. FIG. 7A shows a virtual image priority display in the hand work mode. In the virtual image priority display, similarly to FIG. 3, the virtual image VI is displayed in a wide area from the center of the user's visual field VR to the vicinity of the end. For this reason, the user can clearly visually recognize the virtual image VI in the visual field VR. In the example of FIG. 7A, a circuit diagram which is a drawing for assisting the user's work (soldering) is displayed as the virtual image VI. In the case of virtual image priority display, the display switching unit 144 does nothing to the image processing unit 160 and the display control unit 190.

  FIG. 7B shows a state of the outside scene priority display in the hand work mode. In the external scene priority display, the virtual image VI is not displayed. Specifically, the display switching unit 144 transmits a request to hide the virtual image to the display control unit 190. The display control unit 190 that has received the request stops the generation of the image light by turning off the control signals for the right display drive unit 22 and the left display drive unit 24. As a result, the user can clearly see the outside scene SC through the optical image display unit (the right optical image display unit 26 and the left optical image display unit 28) in the visual field VR.

  Thus, in the hand work mode (first operation mode), the display switching unit 144 performs virtual image priority display when the user's line-of-sight direction is the forward direction (first direction), and the vertical direction (second Direction), the outside scene is given priority display. For this reason, when the user turns the line of sight forward, the virtual image VI can be visually recognized, and when the line of sight is directed downward, the visibility of the outside scene SC can be improved. Therefore, the hand work mode is suitable for “scenes in which a user works while directing his gaze downward” such as assembly work, manufacturing work, writing work, and surgery.

  In step S104 of FIG. 4, when the operation mode is the live mode (step S104: live state), the display switching unit 144 causes the detection unit 142 to detect the user's line-of-sight direction (step S120). Details are the same as in step S110. Thereafter, in step S122, the display switching unit 144 determines whether or not the user's line-of-sight direction is the forward direction. When the user's line-of-sight direction is not the forward direction (step S122: NO), in other words, when the line-of-sight direction is the lateral direction or the vertical direction, the display switching unit 144 displays the virtual image displayed by the image display unit 20 as a virtual image. Prioritized display (step S124), and then the process proceeds to step S120. On the other hand, when the user's line-of-sight direction is the forward direction (step S122: YES), the display switching unit 144 sets the display of the virtual image by the image display unit 20 to the outside scene priority display (step S126), and thereafter the process is performed in step S120. Transition to.

  FIG. 8 is an explanatory diagram for explaining virtual image priority display and outside scene priority display in the live mode. FIG. 8A shows the appearance of priority display of the outside scene in the live mode. Similar to the hand work mode, the virtual image VI is not displayed in the outside scene priority display. The details are as described with reference to FIG. FIG. 8B shows a virtual image priority display in the live mode. Similar to the hand work mode, in the virtual image priority display, the virtual image VI is displayed in a wide area from the center of the user's visual field VR to the vicinity of the end. The details are as described with reference to FIG.

  As described above, in the live mode (second operation mode), the display switching unit 144 sets the outside scene priority display when the line-of-sight direction is the forward direction (first direction), and the vertical direction (second direction). In some cases, virtual image priority display is used. Therefore, it is possible to improve the visibility of the outside scene SC when the user turns the line of sight forward, and to make the virtual image VI visible when the user turns the line of sight downward. Accordingly, the live mode is suitable for “scenes in which a user works while pointing his / her eyes forward”, such as live broadcasting of sports, moderator and news reading, dubbing work, driving, and the like.

A-2-3. Variation of virtual image priority display:
FIG. 9 is an explanatory diagram showing variations of virtual image priority display. FIG. 9A shows an example of a variation. In the virtual image priority display described with reference to FIGS. 7A and 8B, the virtual image VI is displayed in a wide area from the center of the user's visual field VR to the vicinity of the end. However, in the virtual image priority display, the display switching unit 144 may display a virtual image VI that occupies at least a part of the visual field VR of the user. For example, the display switching unit 144 can virtually divide the user's visual field VR vertically and horizontally, and display the virtual image VI on either of them. In this case, for example, the display switching unit 144 may specify an area for displaying the virtual image VI to the image processing unit 160. The image processing unit 160 that has received the designation of the region generates image data that combines the image to be displayed as the virtual image VI and the dummy black data, and uses the generated image data as the right-eye image data Data1 and the left-eye image. It transmits to the image display part 20 as data Data2. The display switching unit 144 may generate image data that combines an image to be displayed as the virtual image VI and dummy black data, and may transmit the generated image data to the image processing unit 160. The image processing unit 160 that has received the image data transmits the received image data to the image display unit 20 as right-eye image data Data1 and left-eye image data Data2.

  In the example of FIG. 9A, the user's visual field VR is vertically divided and a virtual image VI is displayed below (see the hatched area). As described above, the mode in which the user's visual field VR is divided vertically and the virtual image VI is displayed below is suitable for the live mode in the sense that the virtual image VI is displayed away from the line-of-sight direction in which the user's visual field is to be secured. ing. For the same reason, in the hand work mode, a mode in which the user's visual field VR is divided vertically and the virtual image VI is displayed upward is suitable.

  FIG. 9B shows another example of the variation. As indicated by the hatched area in FIG. 9B, the display switching unit 144 may display the virtual image VI with the lower width increased and the upper width decreased to allow the user to feel a sense of depth. Good.

In addition, as the virtual image priority display, for example, the following various modes can be adopted.
In order to emphasize the display of the virtual image VI, color conversion processing is performed on the image data. Specifically, for example, the image display unit 20 is configured to include a camera that captures an outside scene (external scenery) in the user's field of view and acquires an outside scene image. In addition, the display switching unit 144 performs a color conversion process for enhancing the complementary color of the outside scene image acquired by the camera on the image data.
・ Process the image data to emphasize the display of the virtual image VI. Specifically, for example, the display switching unit 144 thickens the lines and borders of the image included in the image data.
In order to emphasize the display of the virtual image VI, the backlight blinking operation is stopped. The backlight blinking operation is an operation for reducing afterimages by turning off the backlight when switching from one image to the next. In this case, the display switching unit 144 transmits a blinking operation stop instruction to the right backlight control unit 201 and the left backlight control unit 202.
In order to emphasize the display of the virtual image VI, the color rendering property or color temperature of the backlight is increased. In this case, the right backlight 221 and the left backlight 222 are assumed to have a structure in which color rendering properties and color temperature are variable.
In order to emphasize the display of the virtual image VI, the transmittance of the external light on the light control plate arranged on the front side of the image display unit 20 is lowered.

A-2-4. Variations on the outside scene priority display:
In the outside scene priority display described with reference to FIGS. 7B and 8A, the virtual image VI is not displayed. However, in the external scene priority display, the display switching unit 144 may reduce the display size of the virtual image VI, may reduce the display of the virtual image VI, and may change the display location of the virtual image VI to the end of the user's visual field VR. It may be near the part. For example, the display switching unit 144 can reduce the size of the virtual image VI in the user's visual field VR to the size of an icon in the entire personal computer screen. In this case, the display switching unit 144 may specify an area for displaying the virtual image VI to the image processing unit 160. Details are the same as “A-2-3. Variation of virtual image priority display”. Further, the display switching unit 144 may generate image data in which an icon-sized image to be displayed as the virtual image VI and dummy black data are combined, and may transmit the generated image data to the image processing unit 160. Details are the same as “A-2-3. Variation of virtual image priority display”. By reducing the display size of the virtual image VI in this way, the display switching unit 144 can improve the visibility of the outside scene SC in other parts where the virtual image VI is not displayed. Further, since the user can visually recognize the small virtual image VI, the user can recognize the presence of the virtual image even during the outside scene priority display.

  For example, the display switching unit 144 can reduce the display of the virtual image VI. In this case, the display control unit 190 of the control unit 10 specifies the luminance of the right backlight 221 together with a control signal that specifies driving ON / OFF of the right backlight 221 to the right backlight control unit 201. Send a control signal. The same applies to the left backlight control unit 202. As a control signal for designating the luminance of the backlight, for example, a PWM (Pulse Width Modulation) signal can be used. The display switching unit 144 transmits a request for thinning the display of the virtual image VI to the display control unit 190. The display control unit 190 that has received the request sends control signals (PWM signals) for reducing the luminance of the right backlight 221 and the left backlight 222 to the right backlight control unit 201 and the left backlight control unit 202. Send. If the illumination light from the right backlight 221 and the left backlight 222 becomes dark, the image light emitted by the image light generation unit becomes weak. Therefore, the virtual image VI displayed in the user's visual field VR is thin and blurred. Display. If the display of the virtual image VI is thinned in this way, the visibility of the outside scene SC that the user sees through the virtual image VI can be improved. Further, since the user can visually recognize the virtual image VI displayed lightly, the user can recognize the presence of the virtual image even while the outside scene priority display is being performed.

  In the above, the display control unit 190 may gradually decrease the duty ratio of the PWM signal transmitted to the right backlight control unit 201 and the left backlight control unit 202 with time. The reduction range of the duty ratio, time, etc. can be arbitrarily determined. As the duty ratio of the PWM signal is decreased stepwise, the illumination light from the right backlight 221 and the left backlight 222 becomes darker (dimmed) stepwise. In this way, the virtual image VI displayed in the user's visual field VR gradually changes to a display that is faintly blurred.

In addition, for example, the following various aspects can be adopted as the outside scene priority display.
In order to make the virtual image VI inconspicuous, color conversion processing is performed on the image data. Specifically, for example, the display switching unit 144 performs a color conversion process on the image data so as to be a protective color of the outside scene image acquired by the camera.
・ Process the image data to make the virtual image VI inconspicuous. Specifically, for example, the display switching unit 144 thins lines and borders of an image included in the image data.
In order to make the virtual image VI inconspicuous, the color rendering property or color temperature of the backlight is lowered. In this case, the right backlight 221 and the left backlight 222 are assumed to have a structure in which color rendering properties and color temperature are variable.
In order to make the virtual image VI inconspicuous, the transmittance of external light in the light control plate arranged on the front side of the image display unit 20 is increased.

  Note that the visibility of the virtual image VI is related to the intensity of image light and the intensity of external light. Therefore, the virtual image priority display and the outside scene priority display may be switched by adjusting the intensity of the image light based on the intensity of the external light.

  As described above, according to the first embodiment, the user's line-of-sight direction detected by the detection unit 142 reflects what the user intends to see, that is, the user's intention. Then, the display switching unit 144 displays the virtual image VI displayed by the image display unit 20 in accordance with the line-of-sight direction detected by the detection unit 142, with the outside scene priority display that prioritizes the outside scene SC and improves the visibility of the outside scene SC. Switching between virtual image priority display and virtual image VI priority. As a result, in the head mounted display 100 (head-mounted display device), the visibility of the outside scene SC can be improved according to the user's intention.

  Furthermore, the display switching unit 144 has a plurality of operation modes (hand work mode as the first operation mode, live mode as the second operation mode) with different gaze directions that improve the visibility of the outside scene SC. The user can operate the display switching unit 144 in an operation mode according to preference. As a result, the convenience of the user of the head mounted display 100 (head-mounted display device) can be improved.

A-3. Additional processing in the display switching process:
In the procedure management process (FIG. 4), the following additional processes 1 and 2 may be further performed. The additional processes 1 and 2 may be added alone or in combination.

A-3-1. Additional processing 1:
FIG. 10 is a flowchart showing the procedure of the additional process 1. In the additional processing 1, the detection unit 142 calibrates the line-of-sight direction of the user according to the characteristics of the user's eyes, the characteristics of the user's eye movement, the usage environment of the head mounted display 100, and the like. .

  The detection unit 142 acquires a detection value when the user's line-of-sight direction is the forward direction (step S12). Specifically, the detection unit 142 displays a virtual image VI including a message such as “please look straight ahead” on the image display unit 20. Then, infrared light is emitted from the light emitting unit 61 and an output signal from the light receiving unit 62 is received. The output signal received in step S12 is a detection value of the light receiving unit 62 when the infrared ray hits the black eye. Next, the detection unit 142 obtains a detection value when the user's line-of-sight direction is the vertical direction (step S14). Specifically, the detection unit 142 causes the image display unit 20 to display a virtual image VI including a message such as “Please move your line of sight while keeping your face straight”. Then, infrared light is emitted from the light emitting unit 61 and an output signal from the light receiving unit 62 is received. The output signal received in step S14 is a detection value of the light receiving unit 62 when infrared rays hit the eyelid. Next, the detection unit 142 acquires a detection value when the user's line-of-sight direction is the vertical direction (step S16). Specifically, the detection unit 142 displays a virtual image VI including a message such as “Please move your line of sight while keeping your face straight” on the image display unit 20. Then, infrared light is emitted from the light emitting unit 61 and an output signal from the light receiving unit 62 is received. The output signal received in step S16 is a detection value of the light receiving unit 62 when the infrared ray hits the white eye.

Based on the detection values acquired in steps S12, S14, and S16, the detection unit 142 determines thresholds Th1 and Th2 used in the process of detecting the user's line-of-sight direction, and stores the thresholds Th1 and Th2 in the storage unit 120 (step S18). The threshold values Th1 and Th2 can be determined as follows, for example.
Th1 = (detected value acquired in step S14 + detected value acquired in step S16) / 2
Th2 = (detected value acquired in step S12 + detected value acquired in step S14) / 2

  As described above, according to the additional processing 1, the detection unit 142 detects the detection value of the light receiving unit 62 when the user's line-of-sight direction is the forward direction, and the detection value of the light receiving unit 62 when the user's line-of-sight direction is the vertical direction. The detection value of the light receiving unit 62 in the case of the lateral direction is acquired, and threshold values Th1 and Th2 used in the detection process of the user's gaze direction are determined based on the acquired detection value. In this way, the detection unit 142 can detect the characteristics of the eyes of the user who actually uses the head mounted display 100 (for example, the size of the eyes, the color and size of the iris, the color and size of the pupil), the eye The threshold values Th1 and Th2 are determined in consideration of various conditions such as characteristics of movement (for example, wrinkles of movement, amount of movement), usage environment of the head mounted display 100 (for example, ambient brightness), and the detection unit 142 The user's line-of-sight direction detection process can be calibrated.

A-3-2. Additional processing 2:
In the additional process 2, the display switching unit 144 switches between the change of the current position coordinates of the user and the movement of the head in addition to the switching between the external scene priority display and the virtual image priority display according to the user's line-of-sight direction. Based on at least one of them, the external scene priority display and the virtual image priority display are switched. Specifically, for example, the display switching unit 144 switches from virtual image priority display to outside scene priority display when the current position coordinates change by a predetermined amount or more. The current position coordinates of the user are acquired from the position information of the control unit 10 detected by the GPS module 134. In this way, the display switching unit 144 can estimate that the user is moving and improve the visibility of the outside scene SC. For example, the display switching unit 144 switches from virtual image priority display to outside scene priority display when the movement of the user's head is equal to or greater than a predetermined amount. The movement of the user's head is acquired from the movement information detected by the 9-axis sensor 66. In this way, the display switching unit 144 can improve the visibility of the outside scene SC by estimating that the user has turned other attention by turning around or looking up.

B. Second embodiment:
In the second embodiment of the present invention, a configuration will be described in which the detection sensitivity of the user's line-of-sight direction by the detection unit can be changed according to the usage state of the head-mounted display device. Below, only the part which has a different structure and operation | movement from 1st Embodiment is demonstrated. In the figure, the same components as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment described above, and detailed description thereof is omitted.

B-1. Configuration of head mounted display device:
FIG. 11 is a block diagram functionally showing the configuration of the head mounted display 100a in the second embodiment. The difference from the first embodiment shown in FIG. 2 is that a control unit 10 a is provided instead of the control unit 10. The control unit 10a includes a CPU 140a instead of the CPU 140. The CPU 140a includes a changing unit 146 in addition to the processing units described in the first embodiment. The changing unit 146 performs state estimation processing.

B-2. Display switching process:
The display switching process in the second embodiment is the same as that in the first embodiment.

B-3. Additional processing in the display switching process:
The additional processing in the second embodiment is the same as that in the first embodiment.

B-4. State estimation process:
FIG. 12 is a flowchart illustrating the procedure of the state estimation process. The state estimation process is a process of estimating the usage state of the head mounted display 100a and changing the detection sensitivity of the gaze direction by the detection unit 142 according to the estimation result. The state estimation process may be executed as an initial setting before the display switching process (FIG. 4) is executed, or may be executed in parallel with the display switching process. When the state estimation process is executed before the display switching process, the detection sensitivity in the line-of-sight direction in steps S110 and S120 of the display switching process can be adjusted by this process. When the state estimation process is executed in parallel with the display switching process, the detection sensitivity in the line-of-sight direction in steps S110 and S120 of the display switching process can be dynamically changed by this process.

  First, the changing unit 146 acquires the current position coordinates of the user (step S202). Specifically, the changing unit 146 acquires position information of the control unit 10 detected by the GPS module 134 and causes the storage unit 120 to store the acquired position coordinates. The changing unit 146 acquires the movement of the user's head (step S204). Specifically, the changing unit 146 acquires the motion information detected by the 9-axis sensor 66 and causes the storage unit 120 to store the acquired motion information. The changing unit 146 determines whether or not the current position coordinates of the user and the movement of the head have been stored in the storage unit 120 a predetermined number of times (step S205). The predetermined number of times can be arbitrarily determined. When the number is less than the predetermined number (step S205: NO), the changing unit 146 causes the process to transition to step S202, and continues to acquire the current position coordinates and the head movement.

  When the number is the predetermined number or more (step S205: YES), the changing unit 146 estimates the usage state of the head mounted display 100a (step S206). Specifically, for example, the changing unit 146 can estimate the usage state by the following procedures a to e.

(A) Of the head movements stored in the storage unit 120, the changing unit 146 compares the head movement stored in the nth time with the head movement stored in the (n + 1) th time, and the amount of change between the two Is detected to be greater than or equal to the third threshold Th3 for a predetermined number of times (for example, three times) or more, it is estimated that the usage state of the head mounted display 100a is “dynamic”.
(B) Similarly, the changing unit 146 compares the head movement stored in the nth time and the head movement stored in the (n + 1) th time among the head movements stored in the storage unit 120. When it is detected that the amount of change is not more than the fourth threshold Th4 a predetermined number of times (for example, three times) or more, it is estimated that the usage state of the head mounted display 100a is “static”.

(C) The changing unit 146 compares the current position stored in the nth time with the current position stored in the (n + 1) th time among the current positions stored in the storage unit 120, and the change amount between them is the fifth. When it is detected that the threshold value Th5 is equal to or greater than a predetermined number of times (for example, three times), the usage state of the head mounted display 100a is estimated to be “dynamic”.
(D) Similarly, the changing unit 146 compares the current position stored in the nth time with the current position stored in the (n + 1) th time among the current positions stored in the storage unit 120, and the amount of change between the two. Is detected to be less than or equal to the sixth threshold Th6 for a predetermined number of times (for example, three times) or more, it is estimated that the usage state of the head mounted display 100a is “static”.

(E) The changing unit 146 estimates that the usage state of the head mounted display 100a is “neither” when none of the above a to d applies. The predetermined threshold values Th3 to Th6 in the above can be arbitrarily determined. The same applies to the predetermined number of times.

  In FIG.12 S208, the change part 146 branches a process according to the estimation result of step S206. When it is estimated that the usage state is “dynamic” (step S208: dynamic), the changing unit 146 changes the detection sensitivity of the gaze direction by the detecting unit 142 to a low level (step S210). Specifically, the changing unit 146 transmits a request for reducing the detection sensitivity to the detecting unit 142. After step S210 ends, the change unit 146 ends the process. In this way, the changing unit 146 changes the sensitivity of detection by the detecting unit 142 to be low in a dynamic usage state in which the user's body, head, and line of sight often change rapidly. For this reason, the excessive change of the gaze direction detected by the detection part 142 can be suppressed.

  When it is estimated that the usage state is “static” (step S208: static), the changing unit 146 changes the detection sensitivity of the line-of-sight direction by the detecting unit 142 to a higher level (step S212). Specifically, the changing unit 146 transmits a request for increasing the detection sensitivity to the detecting unit 142. After step S212 ends, the changing unit 146 ends the process. In this way, the changing unit 146 changes the sensitivity of detection by the detecting unit 142 to be high in a static usage state where the user's body, head, and line of sight do not change much. For this reason, the detection unit 142 can detect even a small change in the line-of-sight direction.

  Further, when it is estimated that the usage state is “neither” (step S208: neither), the changing unit 146 ends the process without changing the detection sensitivity of the detecting unit 142.

  FIG. 13 is an explanatory diagram for explaining a method of changing the detection sensitivity in the line-of-sight direction. FIG. 13A is an explanatory diagram illustrating a state in which the detection unit 142 in the initial state detects the user's line-of-sight direction based on the output signal from the light receiving unit 62. In the initial state, the detection unit 142 determines whether the infrared ray hits the white eye, the eyelid, or the black eye at every time interval t at which the output signal from the light receiving unit 62 is received, and the same state continues “three times or more”. In this case, the state is detected as the user's line-of-sight direction. The details are as described with reference to FIG.

  FIG. 13B is an explanatory diagram illustrating a state in which the detection unit 142 that has received a request to lower the detection sensitivity detects the user's line-of-sight direction based on an output signal from the light receiving unit 62. The detection unit 142 that has received the request for lowering the detection sensitivity increases the number of “three times or more” in the initial state. In the example of FIG. 13B, “5 times or more” is set. In this way, small fluctuations in the user's line-of-sight direction within (t × 5), for example, portions indicated by broken lines in the figure are ignored, so the detection sensitivity in the line-of-sight direction in the detection unit 142 is reduced. Can do.

  FIG. 13C is an explanatory diagram illustrating a state in which the detection unit 142 that has received a request to increase the detection sensitivity detects the user's line-of-sight direction based on an output signal from the light receiving unit 62. The detection unit 142 that has received the request to increase the detection sensitivity reduces the number of “three times or more” in the initial state. In the example of FIG. 13C, “twice or more” is set. In this way, the detection unit 142 can detect the change in the line-of-sight direction simply by continuing the state in which the user's line-of-sight has changed (t × 2). Sensitivity can be increased.

  As described above, according to the second embodiment, the changing unit 146 can change the detection sensitivity of the detecting unit 142 according to the usage state of the head mounted display 100a (head-mounted display device). . For this reason, the display switching unit 144 changes whether to finely switch between the external scene priority display and the virtual image priority display or roughly switches between the external scene priority display and the virtual image priority display according to the usage state of the head mounted display 100a. Can do. As a result, the convenience of the user of the head mounted display 100a can be improved.

  Further, according to the second embodiment, the changing unit 146 uses the movement of the user's head detected by the 9-axis sensor 66 (motion detecting unit), and the usage scene is dynamic or static. And the sensitivity of detection by the detection unit 142 can be automatically changed. As a result, the convenience of the user of the head mounted display 100a can be further improved.

C. Variations:
In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. In addition, the following modifications are possible.

・ Modification 1:
In the said embodiment, it illustrated about the structure of the head mounted display. However, the configuration of the head-mounted display can be arbitrarily determined without departing from the gist of the present invention. For example, each component can be added, deleted, converted, and the like.

  The allocation of components to the control unit and the image display unit in the above embodiment is merely an example, and various aspects can be employed. For example, the following aspects may be adopted. (I) A mode in which processing functions such as a CPU and a memory are mounted on the control unit, and only a display function is mounted on the image display unit. (Iii) a mode in which the control unit and the image display unit are integrated (for example, a mode in which the control unit is included in the image display unit and functions as a glasses-type wearable computer), (iv) instead of the control unit A mode in which a smartphone or a portable game machine is used, and (v) a mode in which the connection unit (code) is eliminated by adopting a configuration in which the control unit and the image display unit can perform wireless communication and wireless power feeding.

  In the above embodiment, for convenience of explanation, the control unit includes a transmission unit, and the image display unit includes a reception unit. However, each of the transmission unit and the reception unit of the above-described embodiment has a function capable of bidirectional communication, and can function as a transmission / reception unit. For example, the control unit shown in FIG. 2 is connected to the image display unit via a wired signal transmission path. However, the control unit and the image display unit may be connected by a connection via a wireless signal transmission path such as a wireless LAN, infrared communication, or Bluetooth (registered trademark).

  For example, the configurations of the control unit and the image display unit illustrated in FIG. 2 can be arbitrarily changed. Specifically, for example, the touch pad may be omitted from the control unit, and the operation may be performed using only the cross key. Further, the control unit may be provided with another operation interface such as an operation stick. Moreover, it is good also as what receives input from a keyboard or a mouse | mouth as a structure which can connect devices, such as a keyboard and a mouse | mouth, to a control part. Further, for example, in addition to an operation input using a touch pad or a cross key, an operation input using a foot switch (a switch operated by a user's foot) may be acquired. For example, after providing a line-of-sight detection unit such as an infrared sensor in the image display unit, the user's line of sight may be detected, and an operation input by a command associated with the movement of the line of sight may be acquired. For example, a user's gesture may be detected using a camera, and an operation input by a command associated with the gesture may be acquired. When detecting a gesture, a user's fingertip, a ring attached to the user's hand, a medical device to be used by the user's hand, or the like can be used as a mark for motion detection. If it is possible to acquire an operation input using a foot switch or a line of sight, the input information acquisition unit can acquire an operation input from the user even in a task in which it is difficult for the user to release his / her hand.

  For example, the head mounted display is a binocular transmissive head mounted display, but may be a monocular head mounted display. Further, it may be configured as a non-transmissive head mounted display in which the transmission of the outside scene is blocked when the user wears the head mounted display.

  FIG. 14 is an explanatory diagram showing an external configuration of a head mounted display in a modified example. In the example of FIG. 14A, the difference from the head-mounted display 100 shown in FIG. 1 is that the image display unit 20b includes a right optical image display unit 26b instead of the right optical image display unit 26. Instead of the left optical image display unit 28, a left optical image display unit 28b is provided. The right optical image display unit 26b is formed smaller than the optical member of the first embodiment, and is disposed obliquely above the right eye of the user when the head mounted display is mounted. Similarly, the left optical image display unit 28b is formed smaller than the optical member of the first embodiment, and is disposed obliquely above the left eye of the user when the head mounted display is mounted. In the case of the example of FIG. 14B, the difference from the head mounted display 100 shown in FIG. 1 is that the image display unit 20 c includes a right optical image display unit 26 c instead of the right optical image display unit 26. Instead of the left optical image display unit 28, a left optical image display unit 28c is provided. The right optical image display unit 26c is formed smaller than the optical member of the first embodiment, and is disposed obliquely below the right eye of the user when the head mounted display is mounted. The left optical image display unit 28c is formed smaller than the optical member of the first embodiment, and is disposed obliquely below the left eye of the user when the head mounted display is mounted. Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is also arbitrary, and the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit completely covers the user's eye. It can also be realized as an unmounted head mounted display.

  For example, functional units such as an image processing unit, a display control unit, a detection unit, a display switching unit, a change unit, and an audio processing unit are executed by the CPU by developing a computer program stored in the ROM or hard disk on the RAM. It was described as being realized. However, these functional units may be configured using an ASIC (Application Specific Integrated Circuit) designed to realize the function.

  For example, in the above-described embodiment, it is assumed that the image display unit is a head mounted display that is mounted like glasses, but the image display unit is a normal flat display device (liquid crystal display device, plasma display device, organic EL display device, etc. ). Also in this case, the connection between the control unit and the image display unit may be a connection via a wired signal transmission path or a connection via a wireless signal transmission path. If it does in this way, a control part can also be utilized as a remote control of a usual flat type display device.

  Further, as the image display unit, instead of the image display unit worn like glasses, an image display unit of another shape such as an image display unit worn like a hat may be adopted. Further, the earphone may be an ear-hook type or a headband type, or may be omitted. Further, for example, it may be configured as a head-up display (HUD, Head-Up Display) mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet.

  For example, in the above-described embodiment, the secondary battery is used as the power source. However, the power source is not limited to the secondary battery, and various batteries can be used. For example, a primary battery, a fuel cell, a solar cell, a thermal cell, or the like may be used.

  For example, in the above embodiment, the image light generation unit is configured using a backlight, a backlight control unit, an LCD, and an LCD control unit. However, the above aspect is merely an example. The image light generation unit may include a configuration unit for realizing another method together with or in place of these configuration units. For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image generation unit can use a digital micromirror device or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type head-mounted display device.

Modification 2
In the said embodiment, an example of the display switching process was shown. However, the procedure of the display switching process is merely an example, and various modifications can be made. For example, some steps may be omitted, and other steps may be added. Further, the order of the steps to be executed may be changed.

  For example, the acquisition of the mode in step S102 and the branching of processing according to the acquired mode may be omitted. In this case, the head mounted display switches between virtual image priority display and outside scene priority display according to the user's line-of-sight direction. The correspondence relationship between the line-of-sight direction, the virtual image priority display, and the outside scene priority display may be determined in advance and stored in the storage unit, or may be designated by the user.

  For example, in step S102, the display switching unit may acquire the operation mode using another method instead of checking the operation mode stored in the storage unit. For example, the display switching unit may acquire the operation mode from the OS, or may acquire the operation mode designated by the user via the designation screen displayed on the image display unit.

  For example, in steps S110 and S120, the detection unit detects the user's line-of-sight direction using an output signal from the light receiving unit. However, a detection part may detect a user's gaze direction using other methods. For example, instead of the light emitting unit and the light receiving unit, the image display unit is arranged in an inner direction of the image display unit (a position facing the user's eye wearing the head mounted display), and the image of the user's eye is displayed. The camera is configured to have a camera for photographing. And a detection part may detect a user's eyes | visual_axis direction (front direction, a side direction, a vertical direction) by image-analyzing the image of the user's eye image | photographed with the camera. In addition, for example, the image display unit includes a muscle activity sensor that can detect the tension and relaxation of muscles (for example, the eyebrows muscles and the ocular muscles) around the user's eyes instead of the light emitting unit and the light receiving unit. And And a detection part may detect a user's gaze direction based on the output value from a muscle activity sensor.

  For example, the display switching unit may switch between virtual image priority display and outside scene priority display when the user's line-of-sight direction is the lateral direction.

  For example, the line-of-sight direction detected by the detection unit can be arbitrarily increased or decreased as long as it includes the forward direction and the vertical direction. For example, detection in the lateral direction may be omitted. Further, for example, an obliquely upward direction and an obliquely downward direction may be detected.

・ Modification 3:
In the said embodiment, an example of the state estimation process was shown. However, the procedure of the state estimation process is merely an example, and various modifications can be made. For example, some steps may be omitted, and other steps may be added. Further, the order of the steps to be executed may be changed.

  For example, the process of estimating the usage state of the head mounted display (that is, steps S202 to S206) may be omitted. In this case, for example, the changing unit may acquire the usage state from the OS and change the detection sensitivity of the line-of-sight direction by the detecting unit. Further, the changing unit may acquire the usage state designated by the user via the designation screen displayed on the image display unit, and may change the detection sensitivity of the gaze direction by the detection unit.

  For example, in steps S210 and S212, the initial state “3 times”, “5 times” when the detection sensitivity of the detection unit is lowered, and “2 times” when the detection sensitivity of the detection unit is increased are merely examples. These values can be arbitrarily changed as long as the relationship “number of times when detection sensitivity is increased <number of times in initial state <number of times when detection sensitivity is lowered” is satisfied.

  For example, as a method of changing the detection sensitivity of the line-of-sight direction in the detection unit, the detection sensitivity can be changed by adjusting the time interval t at which the detection unit receives the output signal from the light receiving unit, instead of the method described in FIG. It may be changed.

-Modification 4:
In the embodiment described above, the user's line-of-sight direction detected by the detection unit is any one of the forward direction, the vertical direction, and the lateral direction. However, when the detection unit can detect up to the angle in the user's line-of-sight direction, the display switching unit is listed below together with the conditions exemplified in the above embodiment or instead of the conditions exemplified in the above embodiment. The virtual image priority display and the outside scene priority display may be switched using each of the conditions. When the angle of the user's line-of-sight direction is within a viewing angle range of about 200 ° horizontal and about 125 ° vertical (for example, 75 ° downward and 50 ° upward), virtual image priority display is performed.
-When the angle of the user's line of sight is within the range of about 30 ° horizontal and about 20 ° vertical, which is an effective visual field with excellent information receiving ability, virtual image priority display is set.
・ When the angle of the user's line-of-sight direction is within the range of 60 ° to 90 ° horizontal and 45 ° to 70 ° vertical, which is a stable gaze point where the gazing point appears quickly and stably, virtual image priority display is performed. .
-Virtual image priority when the angle of the user's line of sight is within a range of about 20 ° horizontal where the induction of self-motion sensation (vection) induced by the image begins to about 110 ° where the self-motion sensation is saturated Display.

-Modification 5:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

10. Control unit (controller)
DESCRIPTION OF SYMBOLS 12 ... Lighting part 14 ... Touch pad 16 ... Cross key 18 ... Power switch 20 ... Image display part 21 ... Right holding part 22 ... Right display drive part 23 ... Left holding part 24 ... Left display drive part 26 ... Right optical image display part 28 ... Left optical image display unit 30 ... Earphone plug 32 ... Right earphone 34 ... Left earphone 40 ... Connection unit 42 ... Right cord 44 ... Left cord 46 ... Connecting member 48 ... Body code 51 ... Transmission unit 52 ... Transmission unit 53 ... Reception 54: Receiving unit 61: Light emitting unit (detecting unit)
62. Light receiving part (detection part)
DESCRIPTION OF SYMBOLS 110 ... Input information acquisition part 100 ... Head mounted display (head mounted display apparatus)
120 ... Storage unit 130 ... Power supply 140 ... CPU
142... Detection unit (detection unit)
144: Display switching unit (display switching unit)
146 ... change part (change part)
DESCRIPTION OF SYMBOLS 160 ... Image processing part 170 ... Audio | voice processing part 180 ... Interface 190 ... Display control part 201 ... Right backlight control part 202 ... Left backlight control part 211 ... Right LCD control part 212 ... Left LCD control part 221 ... Right backlight 222 ... Left backlight 241 ... Right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate PCLK ... Clock signal VSync ... Vertical sync signal HSync ... Horizontal sync signal Data ... Image data Data1 ... Right eye image data Data2 ... Left Image data for eyes OA ... External device PC ... Personal computer SC ... Outside view RE ... Right eye LE ... Left eye VI ... Virtual image ER ... End part EL ... End part AP ... Tip part VR ... Field of view RI ... Iris LI ... Iris RLi ... Eyelid LLi ... eyelid RW ... white eye LW ... white eye Th1 ... first threshold Th2 ... second threshold

Claims (9)

A head-mounted display device that allows a user to visually recognize a virtual image and an outside scene,
An image display unit that allows the user to visually recognize the virtual image;
Irradiate the eyes with infrared rays from the light emitting unit, receive the reflected light at the light receiving unit, receive an output signal indicating that the infrared ray hits one of white eyes, eyelids, or black eyes from the light receiving unit, and the same state A detection unit that detects the user's line-of-sight direction from the state when the output signal indicating the above is continuously received a predetermined number of times or more ,
A display switching unit that switches the display of the virtual image by the image display unit between an outside scene priority display that prioritizes the outside scene and a virtual image priority display that prioritizes the virtual image, according to the detected line-of-sight direction,
A change unit that changes the sensitivity of the detection by the detection unit according to whether the usage state of the head-mounted display device is dynamic or static;
With
The changing unit is
Repeatedly detecting the movement of the head on which the head-mounted display device is mounted, or the current position of the head-mounted display device;
When it is detected that the amount of change in the movement of the head or the amount of change in the current position is greater than or equal to a predetermined first threshold, it is estimated that the usage state of the head-mounted display device is dynamic. To set the detection sensitivity low,
The use state of the head-mounted display device is static when it is detected that the amount of change in the movement of the head or the amount of change in the current position is less than or equal to a second threshold value that is smaller than the first threshold value . And set the sensitivity of the detection high ,
When the detection sensitivity is high, the detection unit sets the number of times less than when the detection sensitivity is low, or when the detection unit receives the output signal from the light receiving unit. Set the interval short,
Head-mounted display device. The head-mounted display device according to claim 1,
The changing unit is
Repeatedly detecting the movement of the head on which the head-mounted display device is mounted;
If the amount of change movement of the head is detected more than a predetermined number of times said at first threshold value or more, the sensitivity of the detection with estimated that use state of the head-mounted display device is a dynamic Set low,
If the amount of change movement of the head is detected more than a predetermined number of times said at second threshold value or less, the sensitivity of the detection with estimated that use state of the head-mounted display device is static the high set, head-mounted display device. The head-mounted display device according to claim 1,
The changing unit is
Repeatedly detecting the current position of the head-mounted display device,
If the amount of change in the current position is detected more than a predetermined number of times said at first threshold value or more, the sensitivity of the detection with estimated that use state of the head-mounted display device is a dynamic Set it low,
If the amount of change in the current position is detected a predetermined number of times or more is not more than the second threshold value, the sensitivity of the detection with estimated that use state of the head-mounted display device is static It is set high, head-mounted display device. The head-mounted display device according to any one of claims 1 to 3,
The head direction includes at least a first direction in which the user's line of sight faces a substantially forward direction and a second direction in which the user's line of sight faces a substantially vertical direction. Wearable display device. The head-mounted display device according to claim 4,
The display switching unit
When the line-of-sight direction is the first direction, the virtual image priority display is provided, and when the line-of-sight direction is the second direction, the outside scene priority display is provided.
Head-mounted display device. The head-mounted display device according to claim 4,
The display switching unit
When the line-of-sight direction is the first direction, the operation mode is the outside scene priority display, and when the line-of-sight direction is the second direction, the virtual image priority display.
Head-mounted display device. The head-mounted display device according to any one of claims 1 to 6,
The display switching unit is configured to hide the virtual image, to reduce the display size of the virtual image, to reduce the display of the virtual image, or to display the virtual image near the end of the visual field of the user. A head-mounted display device that performs the external scene priority display using any one of the means. The head-mounted display device according to any one of claims 1 to 7,
The display switching unit is a head-mounted display device that performs the switching according to a change in an angle in the line-of-sight direction. A method for controlling a head-mounted display device in which a user can visually recognize a virtual image and an outside scene,
(A) making the user visually recognize the virtual image;
(B) irradiating the eyes with infrared rays from the light emitting unit and receiving the reflected light at the light receiving unit, and receiving an output signal from the light receiving unit indicating that the infrared rays hit one of the white eyes, eyelids, and black eyes; A step of detecting the visual line direction of the user from the state when the output signal indicating the same state is continuously received a predetermined number of times or more ;
(C) switching the display of the virtual image according to the step (a) between an outside scene priority display prioritizing the outside scene and a virtual image priority display prioritizing the virtual image according to the detected line-of-sight direction;
(D) changing the sensitivity of the detection depending on whether the usage state of the head-mounted display device is dynamic or static;
With
The step (d)
A step of repeatedly detecting the movement of the head on which the head-mounted display device is mounted, or the current position of the head-mounted display device;
When it is detected that the amount of change in the movement of the head or the amount of change in the current position is greater than or equal to a predetermined first threshold, it is estimated that the usage state of the head-mounted display device is dynamic. And setting the sensitivity of the detection low,
When it is detected that the movement of the head or the amount of change in the current position is equal to or smaller than a second threshold value that is smaller than the first threshold value , it is estimated that the usage state of the head-mounted display device is static. And setting the sensitivity of the detection high,
Only including,
When the detection sensitivity is high, the number of times is set smaller than when the detection sensitivity is low, or the time interval for receiving the output signal from the light receiving unit is set short.
A method for controlling a head-mounted display device.

Images