JP6427942B2 - Head-mounted display device, control system, and head-mounted display device control method - Google Patents

Description

  The present invention relates to a head-mounted display device.

  A head-mounted display device (Head Mounted Display, HMD) that is a display device mounted on the head is known. The head-mounted display device, for example, generates image light representing an image using a liquid crystal display and a light source, and guides the generated image light to a user's eye using a projection optical system or a light guide plate This causes the user to visually recognize the virtual image. There are two types of head-mounted display devices: a transmission type in which the user can visually recognize the outside scene in addition to a virtual image, and a non-transmission type in which the user cannot visually recognize the outside scene. The transmissive head-mounted display device includes an optical transmissive type and a video transmissive type.

  In Patent Literature 1, bidirectional transmission / reception of data between the controller and the controlled device is performed by transmitting display control data related to the control command received by the controlled device that receives the control command transmitted from the controller to the controller. A device control system to be performed is disclosed. In Patent Document 2, when the user's fingertip as a predetermined mark is included in a range equal to or greater than the field of view of the user who has captured the image, the coordinate value of the fingertip is output to An input image processing method for performing a corresponding control process is disclosed. Patent Document 3 discloses an information input method for operating a controlled device based on an outside scene captured by a twin-lens camera and a captured user's hand position and orientation.

Japanese Patent Laid-Open No. 5-268678 JP 2000-148381 A JP 2013-205983 A JP 2006-48628 A JP 2005-69734 A JP 2000-163196 A

  However, in the technique described in Patent Document 1, when there are a plurality of controlled devices, the user cannot control any controlled device, and thus cannot perform control by an intuitive operation. There was a problem of improving usability. Moreover, in the technique described in Patent Document 2, the user needs to continue fixing the position of the fingertip until the target controlled device is selected, which causes fatigue of the user, further improving usability. There was a problem of wanting. Further, in Patent Document 3, the controlled device is operated based on the position and posture of the hand, but the controlled device is operated more intuitively by operating the controlled device in consideration of other information. There was a problem that I wanted to do.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to one aspect of the present invention, a transmissive head-mounted display device is provided. The head-mounted display device displays a virtual image based on image data, and transmits an image of an outside scene; an imaging unit that images the outside scene; a specific object from the captured outside scene; A detection unit for detecting a selectable target; and displaying the virtual image associated with the detected target using the image display unit, and detecting a change in the position of the specific object detected. A control unit that determines control of the target based on the relationship with the detected position of the target and executes control of the determined target. According to the head-mounted display device of this aspect, the user can simultaneously view the target object to be operated and the control associated with the change in the position of the detected specific object without changing the line-of-sight direction. It is possible to control the object intuitively, and the convenience for the user is improved.

(2) In the head-mounted display device according to the above aspect, the control unit displays the virtual image associated in advance with a combination of the detected specific object and the detected object. You may display using. According to the head-mounted display device of this form, the user can recognize the change in the position of a specific object necessary for executing the control of the target object as visual information. Usability of the display device is improved.

(3) In the head-mounted display device according to the above aspect, the control unit includes the object as the virtual image previously associated with a combination of the detected specific object and the detected object. The virtual image indicating a change in the position of the specific object necessary for executing the control may be displayed using the image display unit. According to the head-mounted display device of this aspect, the user can recognize the content of the control of the object to be executed next in response to the change in the position of the specific object of the user as visual information. Usability for the user is improved.

(4) In the head-mounted display device according to the above aspect, the control unit displays, as the virtual image previously associated with the detected object, the virtual image indicating the control content of the object to be executed. You may display using the said image display part. According to the head-mounted display device of this form, the user can recognize the contents of the control of the object to be executed next in response to the change in the position of the specific object as visual information. Usability is improved.

(5) In the head-mounted display device according to the above aspect, the control unit includes the specific object when the position after the change of the specific object overlaps the target object in the captured outside scene. You may determine control of the said object previously matched with the said target object which overlaps with the position after the change of an object. According to the head-mounted display device of this aspect, since the control of the target object corresponding to the target object on which the specific object overlaps is executed after the position of the specific object is changed, the user can It is easy to change a specific object corresponding to the control of an object.

(6) The head-mounted display device according to the above aspect further includes: a distance specifying unit that specifies a distance between the detected object and the image display unit; and the control unit is detected and specified. Alternatively, the object whose distance is equal to or less than a threshold value may be set as the selectable object. According to the head-mounted display device of this aspect, when a plurality of temporary objects are detected in the captured image, an object close to the user is set as a selectable object, so that it is selected by the user. Therefore, the convenience of the user is improved because the user can visually recognize an object that is likely to be displayed.

(7) The head-mounted display device according to the above aspect further includes: an audio acquisition unit that acquires external audio; and the control unit detects a change in the position of the specific object detected and the acquired audio The control of the object may be determined based on the combination. According to the head-mounted display device of this aspect, since the control of the target object is executed by the combination of the change in the position of the specific object and the sound, the target object that is executed only by the change in the position of the specific object The user can intuitively control more objects than the above control.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with another new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including some or all of the four elements of the image display unit, the imaging unit, the detection unit, and the control unit. That is, this apparatus may or may not have an image display unit. Further, the apparatus may or may not have an imaging unit. Moreover, the apparatus may or may not have the detection unit. The device may or may not have a control unit. For example, the image display unit may display a virtual image based on the image data and transmit the outside scene. The imaging unit may image the outside scene, for example. For example, the detection unit may detect a specific object and a selectable object from the captured outside scene. For example, the control unit displays the virtual image associated with the detected object using the image display unit, and detects the change in the position of the specific object detected. Control of the object may be determined based on the relationship with the position, and the control of the determined object may be executed. Such a device can be realized as, for example, a head-mounted display device, but can also be realized as a device other than the head-mounted display device. According to such a form, it is possible to solve at least one of various problems such as improvement and simplification of the operability of the device, integration of the device, and improvement of convenience of the user who uses the device. it can. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device.

  The present invention can also be realized in various forms other than the head-mounted display device. For example, a display device, a head-mounted display device, a control method for the display device, a control system, a head-mounted display system, a display device, a computer program for realizing the functions of the control system and the display device, and the computer program It can be realized in the form of a recorded recording medium, a data signal including the computer program and embodied in a carrier wave.

It is explanatory drawing which shows the external appearance structure of HMD. It is a block diagram which shows the structure of HMD functionally. It is explanatory drawing which shows a mode that image light is inject | emitted by the image light production | generation part. It is explanatory drawing which shows the flow of an apparatus confirmation process. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when an operation target is detected. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when judgment object is detected. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when the confirmed image is displayed. It is a block diagram which shows functionally the structure of the control system in 2nd Embodiment. It is explanatory drawing which shows the flow of control processing. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when an operation target is detected. It is explanatory drawing which shows an example of the visual field which a user visually recognizes in gesture detection mode. It is explanatory drawing which shows an example of the visual field which a user visually recognizes, when the image of the gesture detection mode corresponding to a setting gesture is displayed. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when the image after control is displayed. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when the icon of an image is selected. It is explanatory drawing which shows the external appearance structure of HMD in a modification.

Next, embodiments of the present invention will be described in the following order based on the embodiments.
A. First embodiment:
A-1. Configuration of head mounted display device:
A-2. Device confirmation processing:
B. Second embodiment:
C. Variation:

A. First embodiment:
A-1. Configuration of head mounted display device:
FIG. 1 is an explanatory diagram showing an external configuration of a head-mounted display device 100 (HMD 100). The HMD 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The HMD 100 of the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. In this specification, a virtual image visually recognized by the user with the HMD 100 is also referred to as a “display image” for convenience. Moreover, emitting image light generated based on image data is also referred to as “displaying an image”.

  The HMD 100 includes an image display unit 20 that allows a user to visually recognize a virtual image while being mounted on the user's head, and a control unit 10 (controller 10) that controls the image display unit 20.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. And an infrared LED 64, a TOF sensor 63, and a microphone 69. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

  The display driving units 22 and 24 include liquid crystal displays 241 and 242 (hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252 (see FIG. 2). Details of the configuration of the display driving units 22 and 24 will be described later. The optical image display units 26 and 28 as optical members include light guide plates 261 and 262 (see FIG. 2) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display driving units 22 and 24 to the eyes of the user. The light control plate is a thin plate-like optical element, and is arranged so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. In addition, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The camera 61 is disposed at a position corresponding to the user's eyebrow when the user wears the image display unit 20. Therefore, the camera 61 captures an outside scene, which is an external scenery in the direction of the user's line of sight, in a state where the user wears the image display unit 20 on the head, and acquires a captured image. The camera 61 is a monocular camera, but may be a stereo camera. The camera 61 corresponds to an imaging unit in the claims.

  The infrared LED 64 and the TOF sensor 63 are arranged at positions corresponding to the eyebrows of the user when the user wears the image display unit 20 at the same position as the camera 61. The infrared LED 64 is a light emitting diode (LED) that emits infrared light. The TOF sensor 63 detects the reflected light that the infrared light emitted from the infrared LED 64 is reflected by a specific target.

  The microphone 69 is a device that acquires external sound. The microphone 69 is formed on the opposite side (outside) of the right display driving unit 22 when facing the user in the right display driving unit 22 when the user wears the image display unit 20.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 that are connected to the control unit 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24. The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. A connector (not shown) that fits each other is provided at each of the end of the main body cord 48 opposite to the connecting member 46 and the control unit 10. By fitting / releasing the connector of the main body cord 48 and the connector of the control unit 10, the control unit 10 and the image display unit 20 are connected or disconnected. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for controlling the HMD 100. The control unit 10 includes a determination key 11, a lighting unit 12, a display switching key 13, a track pad 14, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. Yes. The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control unit 10. The lighting unit 12 notifies the operation state of the HMD 100 according to the light emission state. The operation state of the HMD 100 includes, for example, power ON / OFF. For example, an LED is used as the lighting unit 12. The display switching key 13 detects a pressing operation and outputs a signal for switching the display mode of the content video between 3D and 2D, for example. The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the HMD 100 by detecting a slide operation of the switch.

  FIG. 2 is a block diagram functionally showing the configuration of the HMD 100. As illustrated in FIG. 2, the control unit 10 includes a storage unit 120, a power supply 130, a wireless communication unit 132, an operation unit 135, a CPU 140, an interface 180, a transmission unit 51 (Tx51), and a transmission unit 52 ( Tx52). The operation unit 135 receives an operation by the user and includes an enter key 11, a display switch key 13, a track pad 14, a luminance switch key 15, a direction key 16, a menu key 17, and a power switch 18.

  The power supply 130 supplies power to each part of the HMD 100. As the power supply 130, for example, a secondary battery can be used. The storage unit 120 stores various computer programs. The storage unit 120 is configured by a ROM, a RAM, or the like. Although details will be described later, the storage unit 120 stores the image data of the operation target detected when included in the captured image of the camera 61 and the image data of the determination target that is the basis of the determination for performing the control. ing. The storage unit 120 also stores a display image to be displayed on the image display unit 20 when an operation target or the like is detected.

  The CPU 140 reads out and executes the computer program stored in the storage unit 120, thereby operating the operating system 150 (OS 150), the display control unit 190, the sound processing unit 170, the image processing unit 160, the image determination unit 168, and the distance measurement. Functions as a unit 166, an image setting unit 165, and an interface 180.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 controls driving of the right LCD 241 by the right LCD control unit 211, driving ON / OFF of the right backlight 221 by the right backlight control unit 201, and left LCD control unit according to control signals. The left LCD 242 driving ON / OFF by 212, the left backlight 222 driving ON / OFF by the left backlight control unit 202, and the like are individually controlled. Thus, the display control unit 190 controls the generation and emission of image light by the right display driving unit 22 and the left display driving unit 24, respectively. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light.

  The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). Thereafter, the image processing unit 160 stores the converted digital image signal as image data (RGB data) of the target image in the DRAM in the storage unit 120 for each frame. Note that the image processing unit 160 may execute image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data as necessary. .

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. . Note that the image data transmitted via the transmission unit 51 is also referred to as “right eye image data”, and the image data transmitted via the transmission unit 52 is also referred to as “left eye image data”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control unit 10 and the image display unit 20.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and a speaker (not shown) in the right earphone 32 and a speaker (not shown) connected to the connecting member 46 ( (Not shown). For example, when the Dolby (registered trademark) system is adopted, processing on the audio signal is performed, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, for example. The sound processing unit 170 performs various processes on the external sound acquired by the microphone 69. As will be described in detail later, the sound processing unit 170 determines whether or not the acquired external sound is a determination sound that is a basis for determining to perform the device confirmation process stored in the storage unit 120 as various processes. .

  The image determination unit 168 detects whether or not the captured image includes the same image as the operation target or the determination target image data stored in the storage unit 120 by pattern matching or statistical identification. The image determination unit 168 detects, as a determination target, a target that is the same image as the detected determination target image data. The image determination unit 168 corresponds to the detection unit in the claims, and the determination target corresponds to a specific object in the claims.

  The distance measuring unit 166 uses the TOF (Time of Flight) method for the reflected infrared light detected by the TOF sensor 63 to reflect a specific target after the infrared light is emitted until the TOF sensor 63 receives light. It is a distance image sensor that measures the distance between the image display unit 20 and a specific object by calculating time. The distance measuring unit 166 determines whether or not the distance between the specific target representing the same image as the operation target image data detected by the image determining unit 168 and the image display unit 20 is equal to or less than a predetermined distance. judge. The distance measuring unit 166, the TOF sensor 63, and the infrared LED 64 correspond to the distance specifying unit in the claims.

  The image setting unit 165 is an operation target that can select a specific target whose distance from the image display unit 20 is equal to or less than a predetermined distance and that represents the same image as the image data of the operation target stored in the storage unit 120. Set as. When the selectable operation target is set, the image setting unit 165 causes the image display unit 20 to display an image associated with the operation target stored in the storage unit 120. That is, the image determination unit 168 detects the same image as the operation target image data, but a specific target whose distance from the image display unit 20 exceeds a predetermined distance is not set as a selectable operation target. . The image setting unit 165 displays an image associated with the detected determination object in advance on the image display unit 20. The image setting unit 165 determines and executes a control instruction set based on the change in the position of the determination target and the position of the operation target. Details of various images displayed on the image display unit 20 will be described later. The image setting unit 165 corresponds to a control unit in claims. The selectable operation target in the present embodiment corresponds to the selectable target in the claims, and the control instruction set by the image setting unit 165 corresponds to the control of the target in the claims.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer (PC), a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. A TOF sensor 63, an infrared LED 64, and a microphone 69.

  The right display driving unit 22 includes a receiving unit 53 (Rx53), a right backlight control unit 201 (right BL control unit 201) and a right backlight 221 (right BL221) that function as a light source, and a right LCD that functions as a display element. A control unit 211, a right LCD 241 and a right projection optical system 251 are included. The right backlight control unit 201 and the right backlight 221 function as a light source. The right LCD control unit 211 and the right LCD 241 function as display elements. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes a receiving unit 54 (Rx54), a left backlight control unit 202 (left BL control unit 202) and a left backlight 222 (left BL222) that function as a light source, and a left LCD that functions as a display element. A control unit 212 and a left LCD 242 and a left projection optical system 252 are included. The left backlight control unit 202 and the left backlight 222 function as a light source. The left LCD control unit 212 and the left LCD 242 function as display elements. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are also collectively referred to as “image light generation unit”. The left projection optical system 252 is configured by a collimating lens that converts the image light emitted from the left LCD 242 into a light beam in a parallel state. The left light guide plate 262 as the left optical image display unit 28 guides the image light output from the left projection optical system 252 to the left eye LE of the user while reflecting the image light along a predetermined optical path. The left projection optical system 252 and the left light guide plate 262 are collectively referred to as “light guide unit”.

  FIG. 3 is an explanatory diagram illustrating a state in which image light is emitted by the image light generation unit. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby changing the illumination light IL emitted from the right backlight 221 into an image. Is modulated into an effective image light PL representing The same applies to the left side. As shown in FIG. 3, the backlight system is adopted in the present embodiment, but the image light may be emitted using a front light system or a reflection system.

A-2. Device confirmation processing:
FIG. 4 is an explanatory diagram showing the flow of the device confirmation process. In the device confirmation process, after the control unit 10 sets a selectable operation target (for example, a meter of the control device), a predetermined movement (hereinafter simply referred to as “gesture”) of a determination target (for example, a user's finger) is set. Device), which is a control instruction based on a gesture, is executed.

  First, the camera 61 images the outside scene SC (step S10). Note that the camera 61 continuously captures the outside scene SC. In other words, the camera 61 images the outside scene SC as a moving image. The camera 61 transmits image data of the captured image to the image determination unit 168 of the CPU 140. Next, the image determination unit 168 performs image recognition processing such as pattern matching on the captured image captured by the camera 61, thereby operating image data stored in the storage unit 120 in the captured image. It is detected whether or not the same image is included (step S12). When an image of the same image data as the operation target is not detected in the captured image (step S12: NO), the process of step S10 is performed by the camera 61 until an image of the same image data as the operation target is detected in the captured image. Is repeated. In the process of step S12, when an image having the same image data as the operation target is detected in the captured image (step S12: YES), the distance measuring unit 166 of the CPU 140 causes the image determination unit 168 to select the operation target in the captured image. At the same time as the target of the image to be represented (hereinafter also simply referred to as “temporary target”) is detected, the distance between the temporary target and the image display unit 20 is measured by the TOF method using the TOF sensor 63 and the infrared LED 64. The distance measuring unit 166 determines whether or not the measured distance between the temporary object detected by the image determining unit 168 and the image display unit 20 is equal to or less than a predetermined distance (step S14). When it is determined that the measured distance between the temporary target and the image display unit 20 exceeds the predetermined distance (step S14: NO), the image determination unit 168 sets the temporary target as an operation target that can be selected. Instead, the process after step S10 is repeated by the camera 61. When it is determined in step S14 that the distance between the temporary object and the image display unit 20 is equal to or less than the predetermined distance (step S14: YES), the image setting unit 165 is detected by the image determination unit 168. The temporary target is set as a selectable operation target, and the image stored in the storage unit 120 associated with the operation target is displayed on the image display unit 20 (step S16).

  FIG. 5 is an explanatory diagram illustrating an example of a visual field VR visually recognized by the user when an operation target is set. As shown in FIG. 5, the user visually recognizes three control devices MC, which are devices different from the HMD 100, as the set operation targets as the transparent outside scene SC. The central control device MC of the three units visually recognized by the user includes a meter MT1 indicating the voltage value of another device controlled by the control device MC, a meter MT2 indicating the temperature, and a plurality of buttons. It has a button group BT and a lamp group LM including a plurality of lamps. Each button included in the button group BT is pressed to switch on and off to control other devices. Each button of the lamp group LM is included in the button group BT, and is turned on when the corresponding button arranged on the upper side is turned on, and is turned off when the button is turned off. The meter MT1 and the meter MT2 correspond to selectable objects in the claims.

  In the present embodiment, since the storage unit 120 stores the image data of the meter MT1 and the meter MT2, the image determination unit 168 detects the meter MT1 and the meter MT2 as operation targets. When the image determination unit 168 sets the meter MT1 or the meter MT2 as an operation target, the image setting unit 165 associates the meter MT1 with the meter MT1 as an image associated with each of the meter MT1 and the meter MT2 and stored in the storage unit 120. An image IM1 and an image IM2 indicating the respective positions with respect to MT2 are displayed in the image display maximum area PN of the image display unit 20. The image display maximum area PN indicates the maximum area where the image light generation unit can display an image and is not visually recognized by the user. Further, the image setting unit 165 displays the text image TX1 of “confirmation?” As an image stored in the storage unit 120 and associated with at least one of the meter MT1 and the meter MT2 as the set operation target. It is displayed in the maximum area PN. The text image TX1 is an image that prompts the user to confirm the numerical value indicated by the meter MT1 or the meter MT2.

  When the image determination unit 168 sets the meter MT1 or the like as an operation target and the image setting unit 165 displays an image such as the text image TX1 in the maximum image display area PN (step S16 in FIG. 4), the image determination unit In 168, it is detected whether or not the same image as the determination target image data stored in the storage unit 120 for determining a gesture is included in the captured image (step S18). When the determination target image is not detected in the captured image (step S18: NO), the image determination unit 168 waits for detection of the determination target (step S18). When an image to be determined is detected in the captured image (step S18: YES), the image setting unit 165 of the CPU 140 is stored in the storage unit 120, and the user performs a confirmation action prompted by the text image TX1. A gesture instruction image for instructing a gesture to be performed is displayed in the image display maximum area PN (step S20).

  FIG. 6 is an explanatory diagram illustrating an example of a visual field VR visually recognized by the user when a determination target is detected. FIG. 6 shows a state in which the index finger FF1 of the detected user's right hand HD1 as the determination target is raised. As shown in FIG. 6, the image setting unit 165 determines that the numerical values of the meter MT1 and the meter MT2 stored in the storage unit 120 are less than a predetermined threshold when the right hand HD1 that is a determination target is detected. A gesture instruction image indicating a gesture to be executed for confirmation is displayed in the maximum image display area PN. The index finger FF1 of the right hand HD1 corresponds to a specific object in the claims. The gesture instruction image corresponds to a virtual image indicating a change in the position of a specific object in the claims.

  The gesture instruction image includes a parabolic arrow image CS1 from the tip of the index finger FF1 of the right hand HD1 to the meter MT1, and a text image TX2 indicating “Yoshi!” Which is a judgment sound to be generated by the user. In the present embodiment, the image setting unit 165 confirms the confirmation action when the microphone 69 and the sound processing unit 170 detect the determination sound “Yoshi!” After the index finger FF1 moves to the position where the meter MT1 overlaps the captured image. The control instruction is executed. Note that, in the device confirmation process stored in the storage unit 120, when both the meter MT1 and the meter MT2 are detected, the device confirmation of the meter MT1 is set first. The user can see through the outside scene SC overlapping the text images TX1, TX2 and the arrow image CS1. The confirmation action control instruction corresponds to the control of the object determined in the claims.

  When the image setting unit 165 displays the gesture instruction image in the maximum image display area PN (step S20 in FIG. 4), the image determination unit 168 waits for detection of the gesture of the index finger FF1 along the arrow image CS1 ( Step S22). The image determination unit 168 detects the gesture of the index finger FF1 by performing image recognition processing such as pattern matching on the image of the index finger FF1 in each frame of the captured images continuously captured by the camera 61. When the gesture of the index finger FF1 along the arrow image CS1 is not detected (step S22: NO), the image determination unit 168 continues to wait for detection of the gesture of the index finger FF1 along the arrow image CS1 (step S22). . When the gesture of the index finger FF1 along the arrow image CS1 is detected (step S22: YES), the image setting unit 165 is in a state where the position of the index finger FF1 after the gesture overlaps the meter MT1 in the captured image. The voice processing unit 170 waits for the detection of the determination voice “Yoshi!” Shown in the text image TX2 via the microphone 69 (step S24). When the determination sound is not detected or the position of the index finger FF1 does not overlap the meter MT1 (step S24: NO), the image setting unit 165 continues to overlap the position of the index finger FF1 and the meter MT1 in the captured image. It waits for detection of the judgment voice in the state where it is present (step S24). In the captured image, when the position of the index finger FF1 and the meter MT1 overlap with each other and the determination sound “Yoshi!” Is detected (step S24: YES), the image setting unit 165 performs a confirmation act. It is determined that a gesture has been performed, and a confirmed image to be displayed after the confirmation action is completed is displayed in the image display maximum area PN (step S26), and the device confirmation process is terminated.

  FIG. 7 is an explanatory diagram illustrating an example of a visual field VR visually recognized by the user when a confirmed image is displayed. FIG. 7 shows a visual field VR visually recognized by the user when a confirmation action by the user is detected and a text image TX3 of “MT1 confirmation OK!” As a confirmed image is displayed. In the captured image, when the position of the index finger FF1 and the meter MT1 overlap, the image setting unit 165 indicates that the numerical value of the meter MT1 has been confirmed in addition to the image IM1 indicating the position of the meter MT1. In addition, a solid circle image IM3 surrounding the meter MT1 is displayed in the image display maximum region PN. By visually recognizing the text image TX3 and the image IM3 displayed in the image display maximum area PN, the user can confirm that the gesture has been processed normally and the control instruction for device confirmation has been executed.

  As described above, in the HMD 100 according to the present embodiment, the image determination unit 168 detects the operation target and the determination target included in the captured image, and the image setting unit 165 is associated with the detected operation target. The images IM1 and IM2 and the text image TX1 are displayed in the maximum image display area PN, and a device confirmation control instruction corresponding to the detected gesture to be determined is determined and executed. Therefore, in the HMD 100 of this embodiment, the user can visually recognize the operation target to be operated and the control associated with the gesture performed by the user at the same time without changing the line-of-sight direction, and intuitively control the operation target. This can be performed and the convenience for the user is improved.

  Further, in the HMD 100 according to the present embodiment, when the right hand HD1 as a determination target is detected, the image setting unit 165 displays an index finger as an image associated with the combination of the meter MT1 and the meter MT2 and the index finger FF1 of the right hand HD1. An arrow image CS1 showing the change of FF1 is displayed in the image display maximum area PN. Therefore, in the HMD 100 of the present embodiment, the user can recognize gestures necessary for executing the control instruction as visual information, and the usability of the HMD 100 for the user is improved.

  Further, in the HMD 100 according to the present embodiment, the image setting unit 165 detects the determination sound by the sound processing unit 170 in a state where the position of the index finger FF1 that is the determination target overlaps the meter MT1 that is the operation target. Then, the text image TX3 that is the confirmed image is displayed in the image display maximum area PN. Therefore, in the HMD 100 of the present embodiment, the control instruction corresponding to the operation target with which the determination target overlaps is executed after the gesture is performed, and thus the user can easily perform the gesture corresponding to the intended control instruction. Further, since the control instruction is executed by a combination of the gesture and the voice, the user can intuitively give more control instructions than the control instruction executed only by the gesture.

  In the HMD 100 according to the present embodiment, when the distance between the image display unit 20 measured by the distance measurement unit 166 and the temporary object is equal to or less than a predetermined distance, the image setting unit 165 selects the detected temporary object. Set as possible operation target. Therefore, in the HMD 100 of the present embodiment, when a plurality of temporary objects are detected in the captured image, an object close to the user is set as an operation object that can be selected, so that the user is highly likely to be selected. The user's convenience is improved.

B. Second embodiment:
In the second embodiment, unlike the first embodiment, in a control system 500 including the HMD 100a and the control device 300, a control instruction signal is transmitted and received between the HMD 100a and the control device 300. Thereby, after the HMD 100a receives information specifying the operation target from the control device 300, the control device 300 is controlled based on the control instruction input by the HMD 100a.

  FIG. 8 is a block diagram functionally showing the configuration of the control system 500 in the second embodiment. The control system 500 includes an HMD 100a and a control device 300. In FIG. 8, only one HMD 100a and one control device 300 are shown, but the control system 500 may include a plurality of HMDs 100a and control devices 300, or the HMD 100a and the control device 300. Information may be transmitted and received via different devices (for example, servers). The HMD 100 a according to the second embodiment includes a wireless communication unit 132 in the control unit 10 in addition to the configuration of the HMD 100 according to the first embodiment. In FIG. 8, since the HMD 100a has the same configuration other than the wireless communication unit 132 as compared with the HMD 100 of the first embodiment (FIG. 2), a part of the configuration of the HMD 100a is omitted.

  The wireless communication unit 132 of the HMD 100a performs wireless communication with other devices in accordance with a predetermined wireless communication method such as a wireless LAN or Bluetooth (registered trademark). The wireless communication unit 132 transmits the control instruction information determined by the user's gesture to the control device 300. In addition, the wireless communication unit 132 receives information specifying the operation target, the determination target, the determination sound, and the gesture from the control device 300. The wireless communication unit 132 in the second embodiment corresponds to the second communication unit in the claims.

  The control device 300 includes a wireless communication unit 330, a storage unit 320, and a CPU 310. The wireless communication unit 330 performs wireless communication with the HMD 100 in accordance with a predetermined wireless communication method such as a wireless LAN or Bluetooth. The wireless communication unit 330 in the second embodiment corresponds to the first communication unit in the claims. The storage unit 320 is configured by, for example, a ROM, a RAM, a DRAM, a hard disk, and the like. The storage unit 320 stores control content associated with a control instruction for controlling the control device 300. In addition, the storage unit 320 stores operation target image data detected when included in the captured image of the camera 61 and determination target image data that is a basis for determination for control. In addition, the memory | storage part 320 in 2nd Embodiment is corresponded to the target object specific part in a claim.

  The CPU 310 functions as the information processing unit 312 and the control processing unit 314 by reading out and executing the computer program stored in the storage unit 320. The information processing unit 312 processes information specifying a control instruction and an operation target transmitted / received to / from the wireless communication unit 132 of the HMD 100 via the wireless communication unit 330. Based on the control instruction information processed by the information processing unit 312, the control processing unit 314 controls the control device 300 in response to the control instruction. In addition, the control apparatus 300 in 2nd Embodiment is corresponded to the control execution part in a claim.

  FIG. 9 is an explanatory diagram showing the flow of control processing. The processing from step S30 to step S36 in the control processing is from step S10 to step S18 excluding the processing for specifying the distance between the image display unit 20 and the operation target in the device confirmation processing (FIG. 4) of the first embodiment. The process is the same. Therefore, in 2nd Embodiment, the process from step S30 of a control process to step S36 is demonstrated easily.

  In the control process, when the outside scene SC is imaged by the camera 61 (step S30) and an operation target is detected in the captured image (step S32: YES), the image setting unit 165 displays an image associated with the operation target as an image. Display in the maximum display area PN.

  FIG. 10 is an explanatory diagram illustrating an example of a visual field VR visually recognized by the user when an operation target is detected. FIG. 10 shows the outside scene SC visually recognized by the user in the living room and the images IM4 and IM5 displayed in the maximum image display area PN. As shown in FIG. 10, the user stores the television remote controller RC1 that is the operation target set by the image determination unit 168 and the switch SW that is a room illumination switch, and is stored in the storage unit 320 and corresponds to the remote control RC1. The attached image IM4 and the image IM5 associated with the switch SW are visually recognized. The remote controller RC1 and the switch SW correspond to objects in the claims.

  When the image determination unit 168 detects a determination target (step S36: YES) with the images IM4 and IM5 displayed (step S34 in FIG. 9), the image setting unit 165 can detect the user's gesture. An image in the gesture detection mode indicating the state is displayed in the image display maximum area PN (step S38).

  FIG. 11 is an explanatory diagram illustrating an example of the visual field VR visually recognized by the user in the gesture detection mode. As shown in FIG. 11, the image determination unit 168 detects the right hand HD1 as a determination target, the image setting unit 165 displays the image IM6 associated with the remote controller RC1, and the arrow image associated with the switch SW. CS2 and image IM7 are displayed. The image IM6 is an image including a plurality of icons indicating the contents to be controlled when the position of the index finger FF1 of the right hand HD1 overlaps. For example, when an “ON / OFF” icon among the displayed icons is selected by a change in the position of the finger FF1, the control processing unit 314 performs control to switch on / off the power of the television TV1. . Further, an arrow image CS2 shown in FIG. 11 is an image showing a gesture to be performed by the user in order to execute control for selecting the switch SW. The image IM7 is an image showing the content of control executed when the gesture of the index finger FF1 along the arrow image CS2 is performed. That is, the image IM7 indicates that the switch SW is selected when the user's index finger FF1 moves so as to surround the switch SW along the arrow image CS2.

  When the image in the gesture detection mode is displayed in the image display maximum area PN by the image setting unit 165 (step S38 in FIG. 9), the image determination unit 168 controls the remote controller RC1 or the right hand HD1 for controlling the switch SW. Waiting for gesture detection (step S40). When the gesture of the right hand HD1 is not detected (step S40: NO), the image determination unit 168 continues to wait for the detection of the gesture of the right hand HD1 (step S40). When the gesture of the right hand HD1 is detected (step S40: YES), the image setting unit 165 executes control corresponding to the detected gesture that is the detected gesture (step S42). The image setting unit 165 determines whether or not there is a setting gesture that is a corresponding gesture for executing another control that is set next to the control that is executed corresponding to the detected gesture (step S44). . When it is determined that there is a setting gesture (step S44: YES), the image setting unit 165 displays an image in the gesture detection mode corresponding to the setting gesture in the image display maximum area PN (step S38).

  FIG. 12 is an explanatory diagram illustrating an example of the visual field VR visually recognized by the user when an image in the gesture detection mode corresponding to the setting gesture is displayed. FIG. 12 shows an image IM8 that is a gesture instruction image corresponding to the set gesture after the movement of the index finger FF1 along the arrow image CS2 (FIG. 11) is detected as a detected gesture. The image IM8 is an image showing control for turning on the switch SW when the index finger FF1 is moved upward, and turning off the switch SW when the index finger FF1 is moved downward.

  When the image setting unit 165 displays an image in the gesture detection mode in the maximum image display area PN (step S38 in FIG. 9) and a gesture for turning on or off the switch SW is detected (step S40: YES), the image setting is performed. The unit 165 executes control corresponding to the detected gesture (step S42). When there is a setting gesture corresponding to the control executed next to the executed control (step S44: YES), the processing after step S38 is performed. In the process of step S44, when there is no setting gesture corresponding to the control executed next to the executed control (step S44: NO), the image setting unit 165 has executed the control corresponding to the detected gesture. A post-control image indicating this is displayed in the maximum image display area PN (step S46).

  FIG. 13 is an explanatory diagram illustrating an example of a visual field VR visually recognized by the user when a post-control image is displayed. FIG. 13 shows a state in which a text image TX4, which is a post-control image, is displayed after a switch gesture as a detection gesture is turned off and a gesture is detected. As shown in FIG. 13, when the switch SW is turned off by the control processing unit 314, the text image TX4 of “turned off” is transmitted by the image setting unit 165 to notify the user to that effect. It is displayed in the area PN.

  When the text image TX4 of the post-control image is displayed in the image display maximum area PN (step S46 in FIG. 9), the image setting unit 165 displays the text image TX4 in the image display maximum area PN for a predetermined time. While the text image TX4 is displayed in the image display maximum area PN, the operation unit 135 waits for detection of a predetermined operation for ending the control process using the operation target detected in the captured image (step S48). ). When the operation unit 135 does not detect a predetermined operation for ending the control process while the text image TX4 is displayed in the image display maximum area PN (step S48: NO), the image setting unit 165 displays the text. After hiding the image TX4, the processing after step S32 is performed. In the process of step S48, when the operation unit 135 detects a predetermined operation for ending the control process (step S48: YES), the control unit 10 ends the control process. In the second embodiment, the control unit 10 can end the control process even when the control process is in progress by the operation unit 135 detecting a predetermined operation.

  As described above, in the HMD 100a according to the second embodiment, as illustrated in FIG. 12, the image setting unit 165 performs the switch SW that is executed in accordance with the direction in which the index finger FF1 of the right hand HD1 moves along the vertical direction. The image IM8 indicating the control of turning on and off is displayed in the image display maximum area PN. Therefore, in the HMD 100a of the second embodiment, the user can recognize the content of the control that is executed next in response to his / her gesture as visual information, which improves usability for the user.

C. Variation:
In addition, this invention is not limited to the said embodiment, It can implement in a various aspect in the range which does not deviate from the summary, For example, the following deformation | transformation is also possible.

C1. Modification 1:
In the above-described embodiment, the image setting unit 165 displays the image corresponding to the set operation target and the detected determination target and the gesture instruction image in the maximum image display area PN, but does not necessarily display these images. Also good. For example, the image setting unit 165 displays an image corresponding to the set operation target, and executes control corresponding to the detected gesture without displaying the image corresponding to the determination target or the gesture instruction image. Good.

  In the above embodiment, the distance measuring unit 166 measures the distance between the image display unit 20 and the temporary object using the infrared LED 64 and the TOF sensor 63, but it is not always necessary to measure the distance, and other methods are also available. The distance may be measured by For example, the image setting unit 165 may set all of the temporary targets detected by the image determination unit 168 as selectable operation targets regardless of the distance between the image display unit 20 and the operation target. The distance measuring unit 166 may measure the distance between the image display unit 20 and the operation target by comparing the images of the operation target captured by the stereo camera.

  In the above embodiment, the image determination unit 168 detects the index finger FF1 of the right hand HD1 as a determination target. However, the determination target to be detected can be variously modified. For example, the determination target may be the control unit 10 or a ballpoint pen or a touch pen for operating the track pad 14 of the control unit 10. Further, the determination target may be a rod-shaped instrument for instructing on the extension line of the index finger FF1, a specific light shape in the laser pointer, or a specific-shaped instrument having a light emitting unit. (For example, a flashlight). The presence / absence of detection of the determination target may be determined based on the presence / absence of light emission so that the penlight in the light emitting state is detected as the determination target and the penlight not in the light emission state is not detected as the determination target. When it is determined whether or not to be detected as a determination target depending on the presence or absence of light emission, more accurate detection is performed as the determination target. The determination target may be arbitrarily set by the user by operating the operation unit 135.

  In the first embodiment, the image setting unit 165 determines the confirmation action of the meter MT1 by a combination of the gesture and the determination sound detected after the gesture. However, the determination sound is not necessarily detected, and only the gesture is detected. The confirmation action of the meter MT1 may be determined. In addition, the image setting unit 165 determines the confirmation action of the meter MT1 by detecting the determination sound after the gesture, but the timing for detecting the determination sound is not limited to after the gesture, and may be before the gesture. The confirmation action may be determined by a determination sound detected while detecting the gesture.

C2. Modification 2:
In the second embodiment, the control for turning off the switch SW has been described. In this modification, the control when the icon displayed on the image IM6 corresponding to the remote controller RC1 is selected will be described. FIG. 14 is an explanatory diagram illustrating an example of a visual field VR visually recognized by the user when the icon of the image IM6 is selected. As shown in FIG. 14, the position of the index finger FF1 of the right hand HD1 overlaps the “ON / OFF” icon of the image IM6. In this case, the control processing unit 314 switches the power of the TV TV 1 on and off. In the example illustrated in FIG. 14, the power of the TV TV 1 was turned off before the position of the index finger FF1 overlapped with the icon. When it overlaps with the icon, the TV TV 1 is turned on. As described above, in this modified example, the gesture image to be performed by the user is not displayed, but the image setting unit 165 performs control for operating the television TV 1 by changing the position of the index finger FF1 after the image IM6 is displayed. A signal is transmitted to the control device 300.

  In the above embodiment, the image determination unit 168 detects whether or not the captured image captured by the camera 61 includes an image of the operation target or the determination target. However, as a method of detecting the operation target or the determination target, Can be variously modified. For example, an infrared sensor or an ultrasonic sensor may detect an operation target or a determination target. Further, a radar detector that detects a radar wave may detect an operation target or a determination target.

C3. Modification 3:
In the above embodiment, the operation unit 135 is formed in the control unit 10, but the mode of the operation unit 135 can be variously modified. For example, a mode in which a user interface that is the operation unit 135 is provided separately from the control unit 10 may be used. In this case, the operation unit 135 is separate from the control unit 10 in which the power supply 130 and the like are formed. Therefore, the operation unit 135 can be reduced in size and user operability is improved. In addition, a 10-axis sensor that detects the movement of the operation unit is formed in the operation unit 135, and various operations are performed based on the detected movement, so that the user can operate the HMD 100 sensuously.

  For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image generation unit may use LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. In the case of the laser retina projection type, the image display maximum area PN can be defined as an image area recognized by the user's eyes.

  Further, for example, the HMD 100 may be a head mounted display in which the optical image display unit covers only a part of the user's eyes, in other words, the optical image display unit does not completely cover the user's eyes. The HMD 100 may be a so-called monocular type head mounted display.

  FIG. 15 is an explanatory diagram showing an external configuration of the HMD 100 in a modified example. In the example of FIG. 15A, the difference from the HMD 100 shown in FIG. 1 is that the image display unit 20b includes a right optical image display unit 26b instead of the right optical image display unit 26, and the left optical image. Instead of the display unit 28, a left optical image display unit 28b is provided. The right optical image display unit 26b is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely above the right eye of the user when the HMD 100b is worn. Similarly, the left optical image display unit 28b is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely above the left eye of the user when the HMD 100b is worn. In the example of FIG. 15B, the difference from the HMD 100 shown in FIG. 1 is that the image display unit 20c includes a right optical image display unit 26c instead of the right optical image display unit 26, and the left optical image. Instead of the display unit 28, a left optical image display unit 28c is provided. The right optical image display unit 26c is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely below the right eye of the user when the head mounted display is mounted. The left optical image display unit 28c is formed smaller than the optical member of the above-described embodiment, and is disposed obliquely below the left eye of the user when the head mounted display is mounted. Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is also arbitrary, and the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit completely covers the user's eye. It can be realized as an HMD 100 in an uncovered mode.

  The earphone may be an ear-hook type or a headband type, or may be omitted. Further, for example, it may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet.

  The configuration of the HMD 100 in the above embodiment is merely an example and can be variously modified. For example, one of the direction key 16 and the track pad 14 provided in the control unit 10 may be omitted, in addition to the direction key 16 and the track pad 14, or in place of the direction key 16 and the track pad 14, etc. An operation interface may be provided. Moreover, the control part 10 is a structure which can connect input devices, such as a keyboard and a mouse | mouth, and is good also as what receives an input from a keyboard or a mouse | mouth.

  In addition, instead of the image display unit 20 worn like glasses, an image display unit of another method such as an image display unit worn like a hat may be employed as the image display unit. The earphones 32 and 34 can be omitted as appropriate.

  In the above-described embodiment, the HMD 100 may guide the image light representing the same image to the left and right eyes of the user to make the user visually recognize the two-dimensional image, or may display different images on the left and right eyes of the user. The image light to be represented may be guided to make the user visually recognize the three-dimensional image.

  In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. For example, in the above-described embodiment, the image processing unit 160 and the sound processing unit 170 are realized by the CPU 140 reading and executing a computer program, but these functional units may be realized by a hardware circuit. Good.

  In addition, when part or all of the functions of the present invention are realized by software, the software (computer program) can be provided in a form stored in a computer-readable recording medium. In the present invention, the “computer-readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but an internal storage device in a computer such as various RAMs and ROMs, a hard disk, etc. It also includes an external storage device fixed to the computer.

  Moreover, in the said embodiment, as shown in FIG. 1 and FIG. 2, the control part 10 and the image display part 20 are formed as a separate structure, However, about the structure of the control part 10 and the image display part 20, about. However, the present invention is not limited to this, and various modifications are possible. For example, all of the components formed in the control unit 10 may be formed inside the image display unit 20 or a part thereof may be formed. In addition, the power source 130 in the above embodiment may be formed independently and replaceable, or the configuration formed in the control unit 10 may be overlapped and formed in the image display unit 20. For example, the CPU 140 shown in FIG. 2 may be formed in both the control unit 10 and the image display unit 20, or a function performed by the CPU 140 formed in the control unit 10 and the CPU formed in the image display unit 20. May be configured separately.

  Moreover, the aspect of the wearable computer with which the control part 10 and the image display part 20 are integrated and attached to a user's clothes may be sufficient.

  The present invention is not limited to the above-described embodiments and modifications, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments and the modifications corresponding to the technical features in each form described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 10 ... Control part 11 ... Decision key 12 ... Illumination part 13 ... Display switching key 14 ... Trackpad 15 ... Luminance switching key 16 ... Direction key 17 ... Menu key 18 ... Power switch 20 ... Image display part 21 ... Right holding part 22 ... Right display drive unit 23 ... Left holding unit 24 ... Left display drive unit 26 ... Right optical image display unit 28 ... Left optical image display unit 30 ... Earphone plug 32 ... Right earphone 34 ... Left earphone 40 ... Connection unit 42 ... Right cord 44 ... Left code 46 ... Connecting member 48 ... Body code 51,52 ... Transmitting part 53,54 ... Receiving part 61 ... Camera (imaging part)
63 ... TOF sensor 64 ... Infrared LED
69 ... Microphone (voice acquisition unit)
100 ... HMD (head-mounted display device)
120: Storage unit 130 ... Power source 132 ... Wireless communication unit (second communication unit)
135: Operation unit 140 ... CPU
150 ... OS
160: Image processing unit 165: Image setting unit (control unit)
166 ... Distance measuring unit (distance specifying unit)
168 ... Image determination unit (detection unit)
170 ... voice processing unit (control unit)
180 ... interface 190 ... display control unit 201 ... right backlight control unit 202 ... left backlight control unit 211 ... right LCD control unit 212 ... left LCD control unit 221 ... right backlight 222 ... left backlight 241 ... right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate 300 ... Control device (control execution unit)
310 ... CPU
312 ... Information processing unit 314 ... Control processing unit 320 ... Storage unit (object specifying unit)
330 ... wireless communication unit (first communication unit)
340 ... Controlled part 500 ... Control system OA ... External device MC ... Control device SC ... Outside view RE ... Right eye LE ... Left eye EL ... End IL ... Illumination light PL ... Image light LM ... Lamp group PN ... Maximum image display area AP ... tip portion ER ... end portion VR ... field of view BT ... button group SW ... switch (object)
RC1 ... Remote control (object)
HD1 ... right hand FF1 ... index finger (specific object)
IM1, IM2, IM3, IM4, IM5, IM6, IM7, IM8 ... image CS1, CS2 ... arrow image MT1, MT2 ... meter (object)
TV1 ... TV TX1, TX2, TX3, TX4 ... Text image

Claims (8)

A transmissive head-mounted display device,
An image display unit that performs display based on image data and transmits an outside scene;
An imaging unit for imaging the outside scene;
A detection unit for detecting both a specific object and a selectable object from the captured outside scene;
The virtual image previously associated with the combination of the detected specific object and the detected object is displayed in accordance with the position of the object using the image display unit, and the detected specific A control unit that determines control of the target based on a relationship between a change in the position of the object and the detected position of the target, and executes control of the determined target;
A head-mounted display device comprising: The head-mounted display device according to claim 1 ,
The control unit, as the virtual image previously associated with the combination of the detected specific object and the detected target object, the position of the specific object necessary for executing the control of the target object A head-mounted display device that displays the virtual image showing a change in the image using the image display unit. The head-mounted display device according to claim 1 or 2 ,
The control unit, as associated in advance with the imaginary image of the detected object, the virtual image showing the contents of control of the object to be executed is displayed using the image display unit, the head-mounted Display device. The head-mounted display device according to any one of claims 1 to 3 ,
When the position after the change of the specific object overlaps the target object in the imaged outside scene, the control unit applies the target object that overlaps the position after the change of the specific object. A head-mounted display device that determines control of the object associated in advance. The head-mounted display device according to any one of claims 1 to 4 , further comprising:
A distance specifying unit for specifying a distance between the detected object and the image display unit;
The control unit is a head-mounted display device that sets, as the selectable target object, the target object whose detected distance is equal to or less than a threshold value. The head-mounted display device according to any one of claims 1 to 5 , further comprising:
It has an audio acquisition unit that acquires external audio,
The head-mounted display device, wherein the control unit determines control of the object based on a combination of a detected change in position of the specific object and the acquired sound. A control system including the head-mounted display device according to any one of claims 1 to 6 ,
An object specifying unit for specifying the object;
A first communication unit for transmitting information on the identified object;
A control execution unit,
The head-mounted display device is
A second communication unit that receives the transmitted information of the object and transmits the determined control information of the object;
The first communication unit receives the transmitted control information of the object,
The control execution unit executes control based on control of the object received by the first communication unit. A control method for a transmissive head-mounted display device, comprising: an image display unit that performs display based on image data and transmits an outside scene; and an imaging unit that images the outside scene.
Detecting both a specific object and a selectable object from the imaged outside scene;
The virtual image previously associated with the combination of the detected specific object and the detected object is displayed in accordance with the position of the object using the image display unit, and the detected specific And determining the control of the target based on the relationship between the change in the position of the object and the detected position of the target, and executing the control of the determined target.

Images