JP6638392B2 - Display device, display system, display device control method, and program - Google Patents

Description

  The present invention relates to a display device, a display system, a control method for a display device, and a program.

  2. Description of the Related Art Conventionally, a head mounted display (HMD), which is a head-mounted display device, is known (for example, see Patent Document 1). In Patent Literature 1, it is worn in front of a user and immerses the user in an image of a virtual world without receiving interference from the real world.

JP-A-2005-118332

By the way, some HMDs have a configuration in which a real space in the real world can be visually recognized. As an application of this type of HMD, for example, an HMD used to support work on an object in a real space has a use of displaying a text or an image. In such a case, it is troublesome for the user to perform an operation of touching a switch of the controller in order to operate the HMD, since it is necessary to open a hand for performing the operation. Also, the operation of touching the touch sensor disposed in front of the user as described in Patent Literature 1 is troublesome because it is necessary to open a hand for performing the operation. Further, the touch sensor described in Patent Literature 1 is disposed in a space in front of the eyes, and cannot be applied to an HMD having a configuration in which a real space in the real world can be visually recognized.
The present invention has been made in view of the above circumstances, and has as its object to enable a user wearing a head-mounted display device to easily operate the display device.

In order to achieve the above object, a display device of the present invention, in a state of being mounted on a user's head, a display unit that displays an image so that the user can visually recognize the real space, and a display unit around the display unit A detection unit that detects the approach of the indicator of the above, and a control unit that executes a process involving display on the display unit in accordance with the detection result of the detection unit, and the display unit that the detection unit detects A process executed by the control unit in association with two or more operations on the periphery is set, and the control unit performs the set process when the detection unit detects two or more operations on the periphery of the display unit. Is performed.
ADVANTAGE OF THE INVENTION According to this invention, a display device can be operated easily without contacting a display part by making a pointer approach a display part. For example, it is possible to improve convenience when performing an operation during a work. Further, erroneous operations can be prevented by executing processing associated with two or more operations on the periphery of the display unit.

Further, according to the present invention, in the display device, the display unit is a left-eye display unit positioned on the left eye side of the user in a state where the display unit is mounted on the head of the user, and on the right eye side. The display unit for the right eye is located, and the processing executed by the control unit is set in association with a combination of an operation on the display unit for the left eye and an operation on the display unit for the right eye detected by the detection unit. Be performed.
According to the present invention, an erroneous operation can be more reliably prevented.

In addition, in the display device according to the present invention, the detection unit may perform an operation of bringing an object closer to the display unit for the left eye and a contact operation, and an operation of bringing the object closer to the display unit for the right eye. Detecting each of the operations.
According to the present invention, the display device can be operated in various modes, and erroneous operations can be prevented.

Further, in the display device according to the present invention, in the display device, when the detection unit detects the approach at two positions, the control unit displays an operation control instruction image corresponding to the approach to the two positions, and In a state where the control instruction image is displayed, the movement of the indicator in the predetermined direction detected by the detection unit is associated with the movement of the operation control instruction image displayed by the display unit in the predetermined direction. And
According to the present invention, the operation control instruction image displayed on the display unit can be moved in a direction corresponding to the movement of the pointer in response to the operation of bringing the pointer closer to the two positions. Thus, operations such as a position instruction and a direction instruction can be easily performed with the two pointers.

In addition, in the display device according to the present invention, the control unit may be configured such that the detection unit causes the right-eye display unit and the left-eye display unit to move vertically or horizontally with respect to the head of the user. When an operation of moving an object in a direction is detected, a position input in a vertical direction or a horizontal direction in a display area of the display unit is received.
According to the present invention, position input can be performed without touching the display unit.

In the display device, the detection unit may be configured to detect contact with an outer surface of the display unit and approach of an object from outside the display unit.
In the display device, the control unit may be configured to rotate a display in a display area of the display unit when the detection unit detects an operation of rotating the object.
Further, in the display device, when an operation on two points is detected by the detection unit, and the distance between the detected two operation positions changes, the control unit responds to the change in the distance. The display in the display area of the display unit may be configured to be enlarged or reduced.

In addition, the present invention provides the display device, further comprising an imaging unit, wherein the control unit performs an operation of contacting a surface of the display unit for a predetermined time or more by the detection unit, or an object having a predetermined area or more of the display unit. It is characterized in that, when a covering operation is detected, the imaging by the imaging unit is executed.
According to the present invention, imaging can be easily executed by operating the display unit, and erroneous operation can be prevented.

Further, the present invention provides the display device, further comprising: an operation unit that receives an operation, wherein the control unit performs a normal operation mode in which processing is performed in accordance with the operation received by the operation unit, and a detection result of the detection unit. And a switching between a proximity operation mode in which a process involving display on the display unit is executed in response to the above.
According to the present invention, an erroneous operation can be prevented by switching between a proximity operation mode in which an operation based on approach is performed and a normal operation mode.

Further, according to the present invention, in the display device described above, the detection unit detects an approach of a pointer to the display unit or a holding unit that holds an optical component included in the display unit. .
ADVANTAGE OF THE INVENTION According to this invention, approach of the approach of the indicator to the periphery of a display part can be detected more reliably, and can be reflected on a display.

Further, according to the present invention, in the display device, the detection unit includes a proximity sensor arranged around the display unit, and the display and operation control are performed based on a movement of the indicator detected by the proximity sensor. Is made to correspond.
According to the present invention, the approach of the pointer to the periphery of the display unit can be more reliably detected and reflected on the display by the proximity sensor arranged around the display unit.

The present invention also provides the display device, wherein the control unit displays an operation control command image on the display unit when the proximity sensor detects the approach of the indicator to the periphery of the display unit by the proximity sensor. The operation of the operation unit is associated with the operation control instruction image.
According to the present invention, an operation is performed using, for example, an operation control instruction image functioning as a GUI (Graphical User Interface) by using the movement of a pointer detected by a proximity sensor arranged around a display unit. And operability can be improved.

In addition, the present invention is the display device, wherein the display unit is configured to be able to visually recognize the real object by transmitting external light, and displays an image so that the real object is visually recognized by being superimposed on the real object. It is characterized by.
ADVANTAGE OF THE INVENTION According to this invention, a real object and an image can be visually recognized overlapping with a user who is equipped with a display unit that allows a real object to be viewed through external light and allows the user to view an image. Then, an operation of bringing the indicator close to the display unit can be performed on the visually recognized image. Thus, the display related to the real object can be easily controlled by a simple operation of bringing the pointer close to the display unit.

Further, in order to achieve the above object, the present invention, in a state of being mounted on the user's head, a display unit that displays an image so that the user can visually recognize the real space, and an operation unit that receives an operation A method of controlling a display device, comprising: setting a process to be executed in association with two or more operations on the periphery of the display unit, detecting approach of a pointer to the periphery of the display unit, When two or more operations are detected, the set processing is executed.
By executing the control method of the present invention, the display device can be easily operated without touching the display unit, and for example, the convenience can be improved when the operation is performed during work. Further, erroneous operation can be prevented by executing processing associated with two or more operations on the display unit.

Further, in order to achieve the above object, the present invention, in a state of being mounted on the user's head, a display unit that displays an image so that the user can visually recognize the real space, and an operation unit that receives an operation A program that can be executed by a computer that controls a display device including: a process in which an indicator is approached to the periphery of the display unit when a process to be executed in association with two or more operations on the periphery of the display unit is set; Is a program for executing the set processing when two or more operations on the periphery of the display unit are detected.
When the computer executes the program of the present invention, the display device can be easily operated without touching the display portion, and for example, the convenience can be improved when the operation is performed during work. Further, erroneous operation can be prevented by executing processing associated with two or more operations on the display unit.

  The program according to the present invention can also be configured in the form of a recording medium or the like in which the program is recorded so as to be readable by the computer. Recording media include a flexible disk, a hard disk drive (HDD), a compact disk read only memory (CD-ROM), a digital versatile disk (DVD), a Blu-ray (registered trademark) disc, a magneto-optical disk, and a nonvolatile memory card. A computer-readable medium such as an internal storage device (semiconductor memory such as a random access memory (RAM) or a read only memory (ROM)) of an image display device and a universal serial bus (USB) memory. Various media are available.

FIG. 2 is an external view of the HMD of the first embodiment. FIG. 2 is an external view of the HMD of the first embodiment. FIG. 2 is a cross-sectional view of a main part illustrating a configuration of an HMD. FIG. 2 is a diagram illustrating a configuration of an optical system of an image display unit. FIG. 2 is a functional block diagram of each unit constituting the HMD. 5 is a flowchart illustrating the operation of the HMD. FIG. 7 is a diagram illustrating a specific example of an operation mode and display of the HMD. FIG. 7 is a diagram illustrating a specific example of an operation mode and display of the HMD. FIG. 7 is a diagram illustrating a specific example of an operation mode and display of the HMD. FIG. 7 is a diagram illustrating a specific example of an operation mode and display of the HMD. FIG. 7 is a diagram illustrating a specific example of an operation mode and display of the HMD. FIG. 9 is an external view of an HMD according to a second embodiment. FIG. 10 is an external view of an HMD according to a third embodiment. The external view of the HMD of 4th Embodiment. The external view of the HMD of 5th Embodiment.

[First Embodiment]
FIG. 1 and FIG. 2 are explanatory diagrams showing an external configuration of an HMD 100 according to an embodiment to which the present invention is applied.
The HMD 100 is a display device that allows a user to visually recognize a virtual image while being worn on the user's head, and has a glasses-like shape. FIG. 1 is a perspective view seen from the front side, and FIG. 2 is a perspective view seen from the back side. In FIG. 2, illustration of the right holding unit 31 and the left holding unit 32 is omitted for convenience of description.

  The frame 3 configuring the main body of the HMD 100 includes a right light guide 26 and a left light guide 28 that are positioned in front of the user's face and display an image. It has an eyeglass-shaped frame to hold. Further, the frame 3 has a right holding portion 31 and a left holding portion 32 that are in contact with the temporal region of the user. The HMD 100 has the same shape as a frame such as glasses for eyesight correction or sunglasses as a whole, and the frame 3 can have a shape similar to, for example, a spectacle frame.

The right holding unit 31 and the left holding unit 32 hold the frame 3 on the user's head like a temple (vine) of glasses. The frame 3 is provided with a right light guide 26 which is located in front of the right eye of the user when the user wears the HMD 100 and allows the right eye of the user to visually recognize an image. The frame 3 is provided with a left light guide 28 that is located in front of the left eye of the user when the HMD 100 is mounted and allows the left eye of the user to visually recognize an image. The right light guide section 26 and the left light guide section 28 are optical elements (optical components) having translucency, and are fitted and held in rims 3a, 3b formed on the frame 3 like lenses of glasses. You.
At the center position of the frame 3, nose pads 37 and 38 for pointing the frame 3 to the user's nose when the HMD 100 is mounted are provided. The central position where the nose pads 37 and 38 are provided is located in front of the user's eyebrows, and a camera 61 is provided at this position.

Frame ends 35 and 36 located at both left and right ends of the frame 3 incorporate various optical systems described later.
The camera 61 is a digital camera provided with an image sensor such as a CCD or a CMOS, an imaging lens, and the like, and may be configured as a stereo camera. The camera 61 captures an image of at least a part of the outside scene (real space) in the front direction of the HMD 100, in other words, in the direction of the user's field of view with the HMD 100 mounted. In other words, the camera 61 captures an image of a range or direction that overlaps the field of view of the user, and captures the direction in which the user gazes. Although the width of the angle of view of the camera 61 can be set as appropriate, in the present embodiment, as will be described later, it includes the outside world visually recognized by the user through the right light guide 26 and the left light guide 28. Furthermore, it is more preferable that the imaging range of the camera 61 is set so that the entire field of view of the user through the right light guide 26 and the left light guide 28 can be imaged.
The camera 61 performs imaging according to the control of the imaging control unit 161 (FIG. 5) included in the control unit 140, and outputs captured image data to the imaging control unit 161.

  The right display drive unit 22 and the left display drive unit 24 form image light. The image light formed by the right display drive unit 22 is guided to the right eye of the user by the right light guide unit 26, and the image light formed by the left display drive unit 24 is guided to the left eye of the user by the left light guide unit 28. I will For example, FIG. 1 shows a configuration example in which the right display drive unit 22 is housed in the frame end 35 and the left display drive unit 24 is housed in the frame end 36.

FIG. 3 is a cross-sectional view of a main part showing a configuration of the HMD 100, and shows a vertical cross section of the HMD 100 at a position including the right light guide 26 and the frame 3 supporting the periphery thereof.
The right light guide 26 is instructed by the frame 3 to be surrounded by the rim 3a. The rim 3a has a groove shape, and the right light guide 26 is fitted into the rim 3a.

FIG. 4 is a plan view of a main part showing the configuration of the optical system of the HMD 100. FIG. 4 illustrates the left eye LE and the right eye RE of the user for explanation.
The display driving units 22 and 24 include liquid crystal displays 241 and 242 (Liquid Crystal Displays, hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252, and the like.

  The right display drive unit 22 is configured symmetrically with the left display drive unit 24. The right display drive unit 22 includes a right backlight 221 having a light source such as an LED and a diffusion plate, a transmissive right LCD 241 disposed on an optical path of light emitted from the diffusion plate of the right backlight 221, and a right LCD 241. And a right projection optical system 251 having a lens group for guiding the image light L transmitted through the optical system. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The left display drive unit 24 includes a left backlight 222 having a light source such as an LED and a diffuser, a transmissive left LCD 242 disposed on an optical path of light emitted from the diffuser of the left backlight 222, and a left LCD 242. And a left projection optical system 252 including a lens group for guiding the image light L transmitted through the optical system. The left LCD 242 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

The image light emitted by the right display drive unit 22 and the image light emitted by the left display drive unit 24 enter the right light guide unit 26 and the left light guide unit 28, respectively.
The right light guide section 26 and the left light guide section 28 are formed of a light transmissive resin or the like, and guide image light output from the display driving sections 22 and 24 to the user's eyes.

The left projection optical system 252 includes a collimator lens that converts the image light L emitted from the left LCD 242 into a parallel light flux. The image light L converted into a parallel light beam by the collimator lens is incident on the left light guide unit 28. The left light guide 28 is a prism on which a plurality of reflection surfaces for reflecting the image light L are formed. The image light L is guided to the left eye LE through a plurality of reflections inside the left light guide 28. I will A half mirror 262A (reflection surface) located in front of the left eye LE is formed in the left light guide unit 28.
The image light L reflected by the half mirror 262A is emitted from the left light guiding unit 28 toward the left eye LE, and the image light L forms an image on the retina of the left eye LE, and allows the user to visually recognize the image.

The right projection optical system 251 has a collimating lens that converts the image light L emitted from the right LCD 241 into a parallel light flux. The image light L converted into a parallel light beam by the collimator lens is incident on the right light guide 26. The right light guide 26 is a prism on which a plurality of reflecting surfaces for reflecting the image light L are formed. The image light L is guided to the right eye RE through a plurality of reflections inside the right light guide 26. I will In the right light guide 26, a half mirror 261A (reflection surface) located in front of the right eye RE is formed.
The image light L reflected by the half mirror 261A is emitted from the right light guide unit 26 toward the right eye RE, and the image light L forms an image on the retina of the right eye RE, and allows the user to visually recognize the image.

The image light L reflected by the half mirror 261A and the external light OL transmitted through the right light guide 26 are incident on the right eye RE of the user. The image light L reflected by the half mirror 262A and the external light OL transmitted through the left light guide 28 are incident on the left eye LE. As described above, the HMD 100 superimposes the image light L of the internally processed image and the external light OL on the user's eyes, and watermarks the right light guide 26 and the left light guide 28 for the user. The outside scene is seen, and an image by the image light L is visually recognized on the outside scene. Thus, the HMD 100 functions as a see-through display device.
In the present embodiment, the right light guide section 26 and the left light guide section 28 are not configured to form a real image, but form a virtual image and allow a user to visually recognize an image. In the present embodiment, this is expressed as “display an image”. In other words, the HMD 100 may be any as long as it emits the image light L constituting the image irrespective of whether it is a real image or a virtual image toward the user, and allows the user to recognize the image.

  Note that the left projection optical system 252 and the left light guide unit 28 are collectively referred to as a “left display unit”, and the right projection optical system 251 and the right light guide unit 26 are collectively referred to as a “right display unit”. Can also. The configuration of the right display unit and the left display unit is not limited to the above example, and any method can be used as long as a virtual image is formed in front of the user's eyes using image light, for example, using a diffraction grating. Alternatively, a transflective film may be used.

  As shown in FIG. 2, the right light guide unit 26 and the left light guide unit 28 allow the right and left eyes of the user to visually recognize a rectangular image. The rectangular right display section 301 corresponding to the right eye is a rectangular image display area formed by the half mirror 261A, and is image light formed by the right projection optical system 251. The left display section 302 is a rectangular image display area formed by the half mirror 262A, and is image light formed by the right projection optical system 251.

  As described above, while the HMD 100 displays an image, the user sees the half mirror 261A as a rectangular area located inside the right display unit 301, and the half mirror 262A is located inside the left display unit 302. Appears as a substantially square area. Since the entirety of the half mirrors 261A and 262A transmit the external light as described above, the user can visually recognize the entirety of the right display unit 301 and the left display unit 302 so as to overlap the outside scene.

  The camera 61 is arranged at the center position on the front surface of the frame 3 as shown in FIG. In other words, the imaging direction of the camera 61, that is, the angle of view, is directed toward the front of the user, and is directed toward the line of sight when the user views the real space (outside scenery). The camera 61 may include a so-called wide-angle lens as an imaging lens, and may be configured to capture a wide angle of view. The wide-angle lens may include a lens called an ultra-wide-angle lens or a quasi-wide-angle lens, may be a single focus lens or a zoom lens, and has a configuration in which the camera 61 includes a lens group including a plurality of lenses. There may be.

  In the following description, as shown in FIG. 1, a direction based on the frame 3 will be described using X, Y, and Z orthogonal coordinates. The X direction is a direction that approaches and separates from the front surface of the frame 3 including the right light guide unit 26 and the left light guide unit 28, and corresponds to an extension direction of the right holding unit 31 and the left holding unit 32. The X direction can be said to be a depth direction with respect to the user's field of view. The Y direction is the height direction of the HMD 100, more specifically, the height direction of the frame 3, and is the vertical direction of the field of view of the user when the HMD 100 is mounted. The Z direction corresponds to the width direction of the frame 3, more specifically, the direction in which the right light guide unit 26 and the left light guide unit 28 are arranged, and the right and left eyes of the user when the HMD 100 is mounted. It is the direction along the substantially horizontal line that connects.

The HMD 100 includes an external sensor 400 that detects an input operation from the outside, and touchpads 401 and 402.
The external sensor 400 includes a right sensor unit 410 and a left sensor unit 420.
The touch pads 401 and 402 are arranged facing the side of the HMD 100 when the HMD 100 is mounted. More specifically, the touch pad 401 is arranged on the outer surface of the frame 3 in the right holder 31, and the touch pad 402 is arranged on the outer surface of the frame 3 in the left holder 32. The touchpads 401 and 402 may have a long shape or a rectangular shape according to the shapes of the right holding portion 31 and the left holding portion 32, respectively. With the HMD 100 mounted, the user can operate the touchpad 401 with his right hand and operate the touchpad 402 with his left hand, for example.
The touchpads 401 and 402 are operation units (operation reception units) that detect a contact operation and specify an operation position, and are, for example, a capacitance-type or pressure-sensitive touchpad.

A right sensor unit 410 and a left sensor unit 420 are arranged on the front side of the frame 3.
The right sensor unit 410 includes sensors 411, 412, and 413. The sensors 411, 412, 413 are arranged around the right light guide 26. More specifically, the sensor 411 is located above the right light guide 26, is arranged to extend in the Z direction, and has a horizontally long detection area. The sensor 412 is arranged on the side of the right light guide 26 so as to extend in the Y direction, and has a vertically long detection area. The sensor 413 is disposed below the right light guide 26 so as to extend in the Z direction, and has a horizontally long detection area.
Each sensor of the right sensor unit 410 is preferably arranged so as not to overlap with the right light guide unit 26 so as not to block the external light OL passing through the right light guide unit 26. When the sensors 411, 412, and 413 can transmit the external light OL, the sensors 411, 412, and 413 may overlap the right light guide unit 26.

The left sensor unit 420 includes sensors 421, 422, and 423. The sensors 421, 422, 423 are arranged around the left light guide 28. Specifically, the sensor 421 is located above the left light guide unit 28, is arranged to extend in the Z direction, and has a horizontally long detection area. The sensor 422 is arranged on the side of the left light guide 28 so as to extend in the Y direction, and has a vertically long detection area. The sensor 423 is disposed below the left light guide 28 so as to extend in the Z direction, and has a horizontally long detection area. It is preferable that the sensors constituting the left sensor unit 420 are arranged so as not to overlap with the left light guide unit 28 so as not to block the external light OL passing through the left light guide unit 28. When the sensors 421, 422, and 423 can transmit the external light OL, the sensors 421, 422, and 423 may overlap the left light guide unit 28.
The right sensor unit 410 forms a detection unit corresponding to the periphery of the right light guide unit 26, and the left sensor unit 420 configures a detection unit corresponding to the periphery of the left light guide unit 28.

  The sensors 411, 412, 413, 421, 422, and 423 are proximity sensors that detect the approach of an object, and are configured by, for example, capacitance type sensors. The detection ranges of the sensors 411, 412, 413, 421, 422, and 423 extend in front of the HMD 100. That is, the right sensor unit 410 and the left sensor unit 420 detect an object approaching from the front of the HMD 100 when approaching the detection range from outside the detection range. Here, an object in the real space detected by the right sensor unit 410 and the left sensor unit 420 is defined as an operating tool (pointer).

  The proximity sensor is, for example, an operating body that does not contact the proximity sensor, and is preferably capable of detecting an operating body existing within 1 cm from the proximity sensor. An operating tool that contacts the proximity sensor may be detected. In addition, it is preferable that erroneous detection in a state where the HMD 100 is mounted can be prevented or suppressed. For example, as a function of suppressing erroneous detection in a state where the HMD 100 is mounted, a configuration that can execute sensitivity adjustment, sensitivity customization, and the like may be performed. In addition, in the mounted state of the HMD 100, a proximity sensor in which directivity is set in a detection direction, which does not detect the approach of the operating body between the right light guide unit 26, the left light guide unit 28, and the user's face, is used. Is also good.

As shown in FIG. 3, the sensor 411 located above the right light guide 26 can detect an operating tool approaching from the front of the frame 3 as indicated by an arrow A11. Further, as shown by an arrow A13, a configuration may be adopted in which an operating tool approaching from behind the frame 3 can be detected. Further, an operating tool approaching from above can be detected as indicated by an arrow A12. That is, the sensor 411 can detect an object (operating body) approaching in the X direction and the Y direction. The same applies to the sensor 421.
The sensor 413 located below the right light guide 26 can detect an operating tool approaching from the front of the frame 3 as indicated by an arrow A31. Further, as shown by an arrow A33, a configuration may be adopted in which an operating tool approaching from behind the frame 3 can be detected. Further, an operating tool approaching from below can be detected as indicated by an arrow A32. That is, the sensor 413 can detect an object approaching in the X direction and the Y direction. The same applies to the sensor 423.

  In addition, the sensors 412 and 422 can detect an object approaching in the X direction and the Z direction. Further, each of the sensors 412 and 422 detects the movement of the operating tool in the X direction or the Y direction in the detectable range of the sensors 412 and 422.

  Further, each of the sensors 411 and 421 detects the movement of the operating tool in the X direction or the Z direction in the detectable range of the sensor 411. Further, each of the sensors 413 and 423 detects the movement of the operating tool in the X direction or the Z direction within the detectable range of the sensors 413 and 423.

  The HMD 100 can detect the approach of an object based on the detection value of each sensor of the right sensor unit 410 and the detection value of each sensor of the left sensor unit 420. Further, by analyzing the detection value of each sensor, the movement of the object located within the detection range can be detected.

FIG. 5 is a functional block diagram of each unit constituting the HMD 100.
The HMD 100 roughly includes a processing unit 10 and an image display unit 20.
The image display unit 20 includes a right display drive unit 22 and a left display drive unit 24, and supplies image light to the right light guide unit 26 and the left light guide unit 28. The image display unit 20 includes components arranged on the frame 3, and specifically includes a camera 61, a nine-axis sensor 66, an external sensor 400, and touchpads 401 and 402. For example, FIG. 1 shows a configuration example in which the right display drive unit 22 is housed in the frame end 35 and the left display drive unit 24 is housed in the frame end 36. Other parts constituting the image display unit 20 may be arranged at the frame ends 35 and 36 of the frame 3 or at a part including the front surface of the frame 3.

  The right display drive unit 22, left display drive unit 24, camera 61, 9-axis sensor 66, external sensor 400, and touch pads 401 and 402 included in the image display unit 20 are connected to the interface 25.

The interface 25 is connected to the control unit 140 included in the processing unit 10. The control unit 140 functions as an image processing unit 160, an imaging control unit 161, an input detection unit 162, an input analysis unit 163, a processing execution unit 164, a communication control unit 170, and a display control unit 190, as described later. The interface 25 connects the components of the control unit 140 and the components of the processing unit 10 to each other, and transmits and receives various data and signals.
For example, the interface 25 outputs a control signal output by the display control unit 190 to the right display drive unit 22 and the left display drive unit 24. Further, the interface 25 outputs a control signal transmitted from the display control unit 190 to the corresponding right backlight control unit 201 or left backlight control unit 202.
The interface 25 connects the camera 61, the nine-axis sensor 66, the external sensor 400, and the touchpads 401 and 402 to the control unit 140.

The right display drive unit 22 includes the right backlight 221, the right LCD 241, and the right projection optical system 251 described above. Further, the right display drive unit 22 includes a right backlight (BL) control unit 201 that controls the right backlight (BL) 221 and a right LCD control unit 211 that drives the right LCD 241.
The right backlight control unit 201 drives the right backlight 221 according to a control signal transmitted by the display control unit 190. The right LCD control section 211 drives the right LCD 241 based on the signal transmitted by the image processing section 160 and the signal transmitted by the display control section 190.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display drive unit 24 includes the above-described left backlight 222, left LCD 242, and left projection optical system 252. In addition, the left display drive unit 24 includes a left backlight control unit 202 that drives a left backlight 222 and a left LCD control unit 212 that drives a left LCD 242.

The left backlight control unit 202 drives the left backlight 222 according to a control signal transmitted by the display control unit 190. The left LCD control unit 212 drives the left LCD 242 based on the signal transmitted by the image processing unit 160 and the signal transmitted by the display control unit 190.
Note that the right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as a right “image light generation unit”. Similarly, the left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are collectively referred to as a left “image light generation unit”.

  The camera 61 performs imaging according to control data input from the imaging control unit 161, and outputs captured image data to the imaging control unit 161 via the interface 25.

  The nine-axis sensor 66 is a motion sensor (inertial sensor) that detects acceleration (three axes), angular velocity (three axes), and geomagnetism (three axes). The 9-axis sensor 66 may be a sensor unit integrating a plurality of sensors. The nine-axis sensor 66 is connected to the control unit 140 via the interface 25, performs detection at a predetermined cycle (sampling frequency) under the control of the control unit 140, and outputs a detected value to the control unit 140. The control unit 140 can detect the movement of the user's head based on the detection value of the nine-axis sensor 66 with the HMD 100 mounted on the user's head.

The external sensor 400 illustrated in FIG. 5 includes a right sensor unit 410 and a left sensor unit 420, and each of the right sensor unit 410 and the left sensor unit 420 may be connected to the interface 25. For example, the sensors 411, 412, and 413 of the right sensor unit 410 may be unitized and connected to the interface 25. Further, the sensors 421, 422, 423 of the left sensor unit 420 may be unitized and connected to the interface 25. Alternatively, each of the sensors 411, 412, 413, 421, 422, and 423 may be connected to the interface 25.
The external sensor 400 performs detection at a predetermined cycle (sampling frequency) under the control of the input detection unit 162, and outputs a detection value to the input detection unit 162.

The touchpads 401 and 402 are connected to the interface 25, respectively. The touchpads 401 and 402 perform detection at a predetermined cycle (sampling frequency) according to the control of the input detection unit 162, and output a detection value to the input detection unit 162.
Here, the touch pads 401 and 402 may include a detection circuit (not shown) for analyzing a detection value, and the detection circuit may be connected to the interface 25. In this case, the touchpads 401 and 402 perform detection under the control of a detection circuit, and the detection circuit detects a touch operation with the touchpads 401 and 402, specifies an operation position, and inputs data indicating the operation position to an input detection unit. 162. Further, the input detection unit 162 may detect a touch operation based on a detection value input from the touch pads 401 and 402, and specify an operation position.

The processing unit 10 controls the HMD 100. The processing unit 10 includes an operation unit 111 that receives a user operation. The operation unit 111 includes, for example, a power switch (not shown) for instructing on / off of the power of the HMD 100, and other various switches.
The respective units constituting the processing unit 10 are housed in, for example, frame ends 35 and 36 of the frame 3. Further, all or a part of the processing unit 10 may be housed in the right holding unit 31 or the left holding unit 32, or may be housed in a housing configured separately from the frame 3. In this case, the case and the image display unit 20 inside the frame 3 may be connected by wire using a cable or may be connected by wireless communication.

The processing unit 10 includes a control unit 140, an input information acquisition unit 110, and a storage unit 120.
The input information acquisition unit 110 is connected to the operation unit 111. The input information acquisition unit 110 receives a user operation based on a signal input from the operation unit 111. The input information acquisition unit 110 outputs data indicating the operation content of the operation unit 111 to the control unit 140.

  The processing unit 10 includes a power supply unit 113. The power supply unit 113 includes, for example, a primary battery, a secondary battery, or a capacitor, and power is supplied from the power supply unit 113 to each unit of the processing unit 10 and the image display unit 20. The power supply state from the power supply unit 113 is controlled by the control unit 140 according to the operation of a power switch (not shown) and the execution status of the program executed by the control unit 140.

The storage unit 120 is a nonvolatile storage device, and stores various computer programs and data related to these programs. The storage unit 120 may store data of a still image or a moving image displayed by the right display driving unit 22 and the left display driving unit 24.
The storage unit 120 stores the setting data 121. The setting data 121 includes various setting values used by the control unit 140. The setting value included in the setting data 121 may be a value previously input by operating the operation unit 111, or may be set from an external device (not shown) or another device (not shown) via the communication unit 117. The value may be received and stored.
The storage unit 120 stores the input analysis data 122 and the image data 123.

  The control unit 140 includes a CPU (not shown) for executing a program, a RAM (not shown) for temporarily storing a program and data executed by the CPU, and a non-volatile storage of a basic control program and data executed by the CPU. ROM (not shown). The control unit 140 reads and executes a computer program stored in the storage unit 120, and executes an operating system (OS) 150, an image processing unit 160, an imaging control unit 161, an input detection unit 162, an input analysis unit 163, and a processing execution unit 164. , The communication control unit 170 and the display control unit 190.

  The imaging control unit 161 controls the camera 61 to execute imaging, generates captured image data, and temporarily stores the image data in the storage unit 120. When the camera 61 is configured as a camera unit including a circuit that generates captured image data, the imaging control unit 161 acquires the captured image data from the camera 61 and temporarily stores it in the storage unit 120.

The image processing unit 160 generates a signal for image display based on the image data displayed by the right display driving unit 22 and the left display driving unit 24, and transmits the signal to the right display driving unit 22 and the left display driving unit 24. The signal generated by the image processing unit 160 may be a vertical synchronization signal, a horizontal synchronization signal, a clock signal, an analog image signal, or the like.
Further, the image processing unit 160 may perform a resolution conversion process for converting the resolution of the image data into a resolution suitable for the right display driving unit 22 and the left display driving unit 24, as necessary. Further, the image processing unit 160 performs image adjustment processing for adjusting luminance and saturation of image data, 2D / 3D conversion processing for creating 2D image data from 3D image data, or generating 3D image data from 2D image data. May be executed. When the image processing is performed, the image processing unit 160 generates and transmits a signal for displaying an image based on the processed image data.

  The display control unit 190 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24, and generates the image light by the right display drive unit 22 and the left display drive unit 24 based on the control signal. Control the injection. Specifically, the right LCD control unit 211 controls the drive ON / OFF of the right LCD 241, and the right backlight control unit 201 controls the drive ON / OFF of the right backlight 221. The display control unit 190 controls ON / OFF of driving of the left LCD 242 by the left LCD control unit 212 and ON / OFF of driving of the left backlight 222 by the left backlight control unit 202.

  The control unit 140 displays an image based on content data (not shown) stored in the storage unit 120, an image of a menu screen for setting the HMD 100, and the like. The content data stored in the storage unit 120 can include text data, still image data, moving image data, audio data, and the like. Further, the control unit 140 may cause the image display unit 20 to display a captured image captured by the camera 61 or a display image generated from the captured image.

The control unit 140 changes the display mode of the image being displayed according to a user operation detected by the external sensor 400 and the touchpads 401 and 402.
In addition, the control unit 140 causes the image display unit 20 to display an image by superimposing on the real space visually recognized by the user with the external light OL transmitted through the right light guide unit 26 and the left light guide unit 28 (FIG. 1). Is also good. In this case, the control unit 140 detects an object (real object) in the real space based on the captured image of the camera 61, and specifies a position where the user visually recognizes the detected object. The control unit 140 obtains a display position on the right LCD 241 and the left LCD 242 corresponding to a position where the user visually recognizes the object, and displays a text or an image at the display position. As a result, texts and images displayed on the image display unit 20 are displayed at positions corresponding to the object in the real space, and function as an image (hereinafter, referred to as an AR image) having an AR (Augmented Reality) effect. I do.
Further, the HMD 100 may be connected to various external devices (not shown) serving as a content supply source. In this case, the communication unit 117 may be used as an interface for connecting an external device, or another interface may be provided. For example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, and an interface for a memory card may be provided. Examples of the external device include an image supply device that supplies an image to the HMD 100, such as a personal computer (PC), a mobile phone terminal, and a portable game machine.

  The input detection unit 162 operates the external sensor 400 and the touchpads 401 and 402 to acquire a detection value. The input detection unit 162 detects an in-plane operation position of the touch pad 401, that is, an operation position in the X direction and the Y direction, as indicated by reference numeral A1 in FIG. Further, the input detection unit 162 detects an operation position in the plane of the touch pad 402, that is, an operation position in the X direction and the Y direction. Further, the input detection unit 162 may acquire the respective detection values of the sensors 411, 412, 413, 421, 422, and 423, or collectively collect the respective detection values of the right sensor unit 410 and the left sensor unit 420. May be obtained.

The input analysis unit 163 detects an operation on the HMD 100 by analyzing a detection value acquired by the input detection unit 162.
Specifically, the input analysis unit 163 detects an object located within the detection range of the external sensor 400 based on the detection values of the sensors 411, 412, 413 of the right sensor unit 410. Further, the input analysis unit 163 detects an operation on the left eye side display unit of the HMD 100 based on the detection values of the sensors 421, 422, 423 of the left sensor unit 420. Accordingly, the input analysis unit 163 detects that an operation tool including a user's hand or an operation tool (for example, a glove, a pointing stick, a writing tool, or the like) used by the user approaches or comes into contact with the frame 3. , And detects operations by these operating tools.
Here, the input analysis unit 163 may individually detect an operation on the right-eye display unit and an operation on the left-eye display unit of the HMD 100. Alternatively, an operation on the entire HMD 100 may be detected based on a detection result of an operation on the right-eye display unit of the HMD 100 and a detection result of an operation on the left-eye display unit. For example, when both the sensor 411 and the sensor 421 detect the operation tool, the input analysis unit 163 may be configured to be able to specify that the operation is performed by one operation tool.

In addition, the input analysis unit 163 may analyze a detection value of the external sensor 400 to detect a motion of an object located within a detection range of the external sensor 400.
The input analysis unit 163 may detect the movement of the object within the detection range using, for example, an XYZ orthogonal coordinate system that is virtually set by obtaining a change in the detection value of each sensor of the external sensor 400. . In this case, the input analysis unit 163 calculates the motion of the object in the directions indicated by X, Y, and Z in FIG. 1, for example. The X direction, the Y direction, and the Z direction are as described with reference to FIG.
The input analysis unit 163 calculates a moving speed and / or a moving amount in each of the X direction, the Y direction, and the Z direction with respect to the movement of the object within the detection range of the external sensor 400, and calculates the calculated moving speed and / or the moving amount. Is generated. Here, the input analysis unit 163 may generate data indicating whether the movement in each direction is the forward direction or the backward direction.

  The input analysis unit 163 generates data indicating presence or absence of an operation, an operation position, a locus of movement of the operation position, and the like for each of the touch pads 401 and 402 based on the detection results of the touch pads 401 and 402. Further, the input analysis unit 163 may detect, as one operation, an operation of touching the touch pads 401 and 402 a plurality of times based on a temporal change in the position of the touch operation detected by the touch pads 401 and 402. Specifically, an operation in which the user hits the touch pad 401 with the operating body a plurality of times (tap) may be detected as one operation. In this case, the input analysis unit 163 may detect two tap operations as one double tap operation, or may detect three tap operations as one triple tap operation. The same applies to the touch pad 402.

  In addition, when the operation tool that has touched the touch pad 401 or the touch pad 402 moves while maintaining the contact state, the input analysis unit 163 may obtain a trajectory of the movement. In this case, the input analysis unit 163 calculates the motion of the object in the directions indicated by X and Y in FIG. 1, for example. The input analysis unit 163 generates, for each of the touchpads 401 and 402, data representing the operation position and the trajectory of the operation position as components in the X direction and the Y direction. For the trajectory of the operation, data including the moving speed and the moving amount in the X direction and the Y direction may be generated, and data indicating whether the movement in each of the X and Y directions is the forward direction or the reverse direction may be generated. Is also good.

The input analysis unit 163 reads out and uses the input analysis data 122 stored in the storage unit 120 in a process of analyzing the detection values of the external sensor 400 and the touch pads 401 and 402 to detect an operation. The input analysis data 122 is data for obtaining the position and movement of the object from the detection value of the external sensor 400. Specifically, data that associates the detection value of the sensor of the external sensor 400, the change over time of the detection value, the combination of the detection values of the sensors 411, 412, 413, 421, 422, and 423 with the position and the movement of the object. is there. The input analysis data 122 may be in any form, for example, an arithmetic expression, a parameter, a database storing a plurality of parameters, an LUT (LookUp Table) including a plurality of parameters and converting a detection value into other data. it can. Further, the input analysis data 122 may be a combination of a program and data for the input analysis unit 163 to process the detection value data. Further, the input analysis data 122 may be data individually prepared for each of the sensors 411, 412, 413, 421, 422, and 423. The data may be individually prepared for each of the right sensor unit 410 and the left sensor unit 420, and is used when the detection values of the sensors 411, 412, 413, 421, 422, and 423 are integrally processed. It may be data.
In addition, the input analysis data 122 may include parameters used for a process of obtaining an operation position of a contact operation and a locus of the operation position from detection values of the touch pads 401 and 402.

  As described above, the input analysis unit 163 can detect a plurality of operations on the external sensor 400 or the touch pads 401 and 402 as one operation or a series of operations. Therefore, one operation including a plurality of touch operations and an operation of moving the operation tool in a specific mode can be detected. Specifically, a double tap or triple tap operation on the touch pads 401 and 402, and a gesture operation of moving an operation tool can be detected as one operation. Therefore, various operations can be performed on the HMD 100.

  The process execution unit 164 executes a process corresponding to the operation detected by the input analysis unit 163. For example, when an operation detected by the input analysis unit 163 is performed, the process execution unit 164 performs a process of displaying a menu image based on the image data 123 stored in the storage unit 120. Here, the image data 123 is one of the content data described above, and may include image data of various images displayed under the control of the control unit 140 in addition to the menu image.

The processing execution unit 164 acquires data generated by the input analysis unit 163 and specifies an operation performed by the external sensor 400 and the touchpads 401 and 402. Therefore, the set processing can be executed in response to a complicated operation including a plurality of operations performed on the external sensor 400 and the touch pads 401 and 402.
The processing executed by the processing execution unit 164 can be specified by the setting data 121, for example. In this case, the setting data 121 includes data that associates an operation detected by the input analysis unit 163 with a process executed by the process execution unit 164.

  The control unit 140 switches between the normal operation mode and the proximity operation mode as the operation mode of the HMD 100 and executes the operation mode. The normal operation mode is an operation mode in which the HMD 100 operates according to the operation of the touch pads 401 and 402. In the normal operation mode, the control unit 140 does not execute a process in response to an operation on the external sensor 400.

  In the normal operation mode, for example, the control unit 140 does not allow the input detection unit 162 to perform detection by the external sensor 400. Alternatively, the input analysis unit 163 does not perform the process of analyzing the detection value obtained by the input detection unit 162. Further, the process execution unit 164 does not execute the process corresponding to the operation related to the external sensor 400 among the operations analyzed by the input analysis unit 163. Any of these operation states may be executed. However, as described above, when the input detection unit 162 limits the detection of the external sensor 400, there is an advantage that power consumption can be suppressed. Further, when the execution of the processing by the input analysis unit 163 or the processing execution unit 164 is restricted as described above, there is an advantage that the switching from the normal operation mode to the proximity operation mode described below can be quickly executed.

The proximity operation mode is an operation mode in which an operation is detected by the external sensor 400. That is, this is an operation mode in which the user can operate the operating tool by approaching the operating body to the external sensor 400.
In the proximity operation mode, an object located in the detection range of the right sensor unit 410 and the left sensor unit 420 is detected, and an operation is detected by the operation of the input detection unit 162 and the input analysis unit 163 described above.

  The normal operation mode is an operation mode for preventing a malfunction due to an erroneous operation or an unintended operation by restricting a process corresponding to the operation of the external sensor 400. For example, it is possible to prevent the control unit 140 from changing the display due to an approach of an object not intended by the user. In the proximity operation mode, an operation of bringing an object closer to the external sensor 400 is possible, and various operations on the HMD 100 can be performed more easily. The operation of switching between the normal operation mode and the proximity operation mode will be described later with reference to a flowchart.

  The detection range in which the external sensor 400 detects the approach of the object is arbitrary, but may be any range that can detect the approach of the object to the periphery of the display unit of the HMD 100, for example. Here, the display unit is a front portion of the frame 3 on which the image display unit 20 is disposed, a right light guide unit 26 and a left light guide unit 28 for displaying an image, or a right light guide unit 26 and a left light guide unit 28. Refers to the front part of the frame 3 holding the frame. As shown in FIGS. 1 and 2, the external sensor 400 includes a right sensor unit 410 and a left sensor unit 420 that can detect approach and contact of an object with the front part (front part) of the frame 3. Thus, the approach can be detected in the proximity operation mode.

In the proximity operation mode, the presence or absence of an object in the detection range of each sensor is detected based on the detection value of each sensor of the external sensor 400 by the processing of the input detection unit 162 described later. Further, the input detection unit 162 may detect the approach and separation of the object within the detection range, and may detect the intrusion and separation of the object into the detection range.
The distance at which the external sensor 400 detects the approach of an object, that is, the detection range is, specifically, a range in which the sensors 411, 412, 413 and the sensors 421, 422, 423 detect an object that is not in contact. It can be represented by the distance from the sensor. This distance may be fixed or variable. Further, the sensors for detecting the approach in the approach operation mode may include the touch pads 401 and 402.

  For example, the detection range of each sensor of the external sensor 400 may be 1 cm to 2 cm or about 2 cm to 5 cm from a contact position (for example, the surface of the sensor or the surface of the frame 3). In the proximity operation mode, the detection range may be changed under the control of the input detection unit 162. A specific method of changing the detection range is, for example, a method of switching processing of the input detection unit 162 that analyzes a detection value output from each sensor of the external sensor 400. Further, a method of switching a setting value of a circuit for driving each sensor of the external sensor 400 or a detection condition such as a current, a voltage, and a pulse frequency for driving each sensor may be adopted.

The GPS 115 and the communication unit 117 are connected to the control unit 140.
The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and calculates the current position of the processing unit 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. Also, the GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time measured by the control unit 140.

  The communication unit 117 executes wireless data communication conforming to standards such as wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), and Bluetooth (registered trademark).

  Further, the HMD 100 may include a microphone (not shown) for collecting sound. Further, a headphone or a speaker for outputting sound may be provided. In this case, the control unit 140 has a function of acquiring data of a sound collected by the microphone and a function of outputting a sound signal based on the sound data. The microphone is connected to, for example, a circuit (not shown) that outputs a sound signal of the collected sound to the control unit 140. Further, for example, the speaker or the headphone is connected to an amplifier (not shown) for amplifying the audio signal output from the control unit 140, and outputs audio based on the amplified audio signal. In this case, the control unit 140 can make the user listen to the voice based on the voice data included in the content data. Further, processing such as changing the display can be executed based on the sound collected by the microphone. As a processing unit that performs these processes, the control unit 140 may include an audio processing unit.

FIG. 6 is a flowchart showing the operation of the HMD 100.
After the power of the HMD 100 is turned on, the control unit 140 starts operating in the normal operation mode (step S11).
When the input detection unit 162 detects an operation on the touchpads 401 and 402 (step S12), the input analysis unit 163 analyzes the operation and specifies the operation state (step S13). Here, the process execution unit 164 determines whether or not the operation state specified by the input analysis unit 163 corresponds to an operation for instructing switching of the operation mode (Step S14).
If the operation does not correspond to an operation for instructing switching of the operation mode (step S14; NO), the process execution unit 164 refers to the setting data 121 and relates to the process corresponding to the operation state specified by the input analysis unit 163 in step S13. The setting is obtained (step S15). The process execution unit 164 executes the set process (Step S16), and thereafter, the control unit 140 determines whether to end the operation (Step S17). For example, when it is determined that the operation is to be ended, for example, when the power switch is operated (step S17; YES), the control unit 140 ends this processing. If the operation is not to be ended (step S17; NO), the process returns to step S12.

  When the process execution unit 164 determines that the operation state specified by the input analysis unit 163 corresponds to the operation for instructing the switching of the operation mode (Step S14; YES), the process execution unit 164 shifts to the proximity operation mode. (Step S18). In step S18, the processing execution unit 164 shifts to a state where processing can be executed in accordance with an operation on the external sensor 400. Specifically, the detection operation of the external sensor 400 under the control of the input detection unit 162 is started, or the analysis by the input analysis unit 163 is started.

Next, when the input detection unit 162 detects an operation on the external sensor 400 or the touch pads 401 and 402 (Step S19), the input analysis unit 163 analyzes the operation and specifies an operation state (Step S20). Here, the process execution unit 164 determines whether the operation state specified by the input analysis unit 163 corresponds to an operation for instructing switching of the operation mode (Step S21).
If the operation does not correspond to the operation for instructing the switching of the operation mode (step S21; NO), the process execution unit 164 refers to the setting data 121 and relates to the process corresponding to the operation state specified by the input analysis unit 163 in step S20. The setting is obtained (step S22). The process execution unit 164 executes the set process (step S23), and thereafter, the control unit 140 determines whether to end the operation (step S24). When it is determined that the operation is to be ended (Step S24; YES), the control unit 140 ends this processing. If the operation is not to be ended (step S24; NO), the process returns to step S19.

  When the process execution unit 164 determines that the operation state specified by the input analysis unit 163 corresponds to the operation for instructing the switching of the operation mode (Step S21; YES), the process execution unit 164 shifts to the normal operation mode. (Step S25), and the process returns to Step S12.

The operation for instructing the switching of the operation mode is, for example, a double tap operation on the touch pad 401 or the touch pad 402. The operation of instructing the switching of the operation mode may be a common operation in both the switching from the normal operation mode to the proximity operation mode and the switching from the proximity operation mode to the normal operation mode. In this case, in order to make the operation detectable in both the normal operation mode and the proximity operation mode, it is preferable that the operation be an operation on the touch pads 401 and 402.
In addition, in the normal operation mode, the process corresponding to the operation of the external sensor 400 is not performed, but the operation mode may be switched by detecting only the operation of the external sensor 400 instructing the switching of the operation mode.

Here, a specific example of the operation in which the processing execution unit 164 changes the display mode in response to the operation of the external sensor 400 will be described.
FIG. 7 is a diagram illustrating a manner of operation on the HMD 100 and a specific example of display. FIG. 7A shows an example of the operation, and FIG. 7B shows a display example corresponding to the operation of FIG. 7A. FIG. 7C shows another example of the operation, and FIG. 7D shows a display example corresponding to the operation of FIG. 7C.

FIG. 7A shows a state in which the two operating bodies are brought close to each other on the side of the HMD 100. More specifically, the two operating bodies RH and LH are moved close to the HMD 100 in a state where they are separated to the left and right. It shows the state that it was turned on. The operating tool RH is, for example, the right hand of the user, and the operating tool LH is, for example, the left hand of the user. The operation bodies RH and LH are not limited to the side of the HMD 100, and may be in front of the right light guide 26 and the left light guide 28. FIG. 7A shows a view of the HMD 100 viewed from the back side, that is, from the side of the face of the user in the mounted state, for convenience of understanding.
7A and 7B, the detection range D of the external sensor 400 is located in a space extending in front of the right light guide unit 26 and the left light guide unit 28, and up, down, left, and right.

  The operation in FIG. 7A indicates an operation in which the operating tools RH and LH are moved closer to the right light guide unit 26 and the left light guide unit 28 and moved upward U or downward D, respectively. In this case, the input analysis unit 163 detects the operating tool RH by the right sensor unit 410 of the external sensor 400 and detects the operating tool LH by the left sensor unit 420. When the right sensor unit 410 and the left sensor unit 420 each detect one object, the process execution unit 164 displays a Y-direction guide GY (operation control instruction image) for position input as a set operation. . In FIG. 7B, a Y-direction guide GY is displayed in the display area VA in the visual field VR of the user. Here, the display area VA indicates a maximum range in which a user can visually recognize a virtual image formed by the right light guide section 26 and the left light guide section 28. The display area VA is actually an area in which a virtual image viewed by the right eye of the user and a virtual image viewed by the left eye are combined by the visual function of the user. Are represented as one rectangle in the sense of.

The Y-direction guide GY is a guide for performing an operation of inputting a position in the vertical direction, that is, the Y direction, and is, for example, a straight line extending horizontally in the display area VA.
The processing execution unit 164 displays the Y-direction guide GY in the display area VA, and follows the movement in the upward direction U when the input analysis unit 163 detects the upward movement U of the operation tool RH and the operation tool LH. To move the Y-direction guide GY upward U. When the input analysis unit 163 detects the downward movement D of the operating tool RH and the operating tool LH, it moves the Y-direction guide GY downward D so as to follow the downward movement D. The correspondence between the movement amount of the operating tool RH and the operating tool LH detected by the input analysis unit 163 and the movement amount of the Y-direction guide GY is set in the setting data 121 in advance.

  As shown in FIGS. 7A and 7B, in the HMD 100, the user positions the right hand on the right eye side, positions the left hand on the left eye side, and approaches the frame 3, thereby inputting the position in the Y direction. Can be performed. When the user moves the operating tools RH and LH in a predetermined manner after moving the Y-direction guide GY to a desired position, the position in the Y direction is determined. The predetermined mode means, for example, that the operating tools RH and LH are stationary for a predetermined time, and both the operating tools RH and LH are translated in the horizontal direction, that is, the X direction (the right direction R or the left direction L in FIG. 7C). And so on. When the operating tools RH and LH are in contact with the frame 3 while the operating tools RH and LH are moved in the upward direction U or the downward direction D, the operating tools RH and LH are separated from the frame 3 to obtain Y. The position in the direction may be determined.

The operation in FIG. 7C indicates an operation in which the operating tools RH and LH are moved close to the frame 3 and moved in the right direction R or the left direction L. In this case, the input analysis unit 163 detects the operating tool RH by the right sensor unit 410 of the external sensor 400 and detects the operating tool LH by the left sensor unit 420. The processing execution unit 164 is set when the right sensor unit 410 and the left sensor unit 420 each detect one object after the position in the Y direction is determined by the operation illustrated in FIG. As an operation, an X-direction guide GX (operation control instruction image) for position input is displayed.
In FIG. 7D, an X-direction guide GX is displayed in the display area VA in the user's visual field VR. The X-direction guide GX is a guide for performing an operation of inputting a position in the horizontal direction, that is, the X direction, and is, for example, a straight line that extends in the vertical direction in the display area VA.
The processing execution unit 164 displays the X-direction guide GX in the display area VA, and follows the movement in the right direction R when the input analysis unit 163 detects the movement of the operation tool RH and the operation tool LH in the right direction R. The X direction guide GX is moved to the right direction R so that When the input analysis unit 163 detects the movement of the operating tool RH and the operating tool LH in the left direction L, the X direction guide GX is moved in the left direction L so as to follow the movement in the left direction L. The correspondence between the movement amounts of the operating tool RH and the operating tool LH detected by the input analysis unit 163 and the movement amount of the X-direction guide GX is set in the setting data 121 in advance.

  As shown in FIGS. 7C and 7D, in the HMD 100, the user positions the right hand on the right eye side, positions the left hand on the left eye side, and approaches the frame 3, thereby inputting the position in the X direction. Can be performed. After moving the X-direction guide GX to a desired position, when the user moves the operating tools RH and LH in a predetermined manner, the position in the X direction is determined. As in the case of the position input using the Y-direction guide GY, for example, the operating tools RH and LH are kept stationary for a predetermined time, and both the operating tools RH and LH are in the vertical direction, that is, in the Y direction (upward U or downward direction). D), and the like. When the operating tools RH and LH are in contact with the frame 3 while the operating tools RH and LH are moving in the right direction R or the left direction L, the operating tools RH and LH are separated from the frame 3 so that X The position in the direction may be determined.

The examples shown in FIGS. 7A to 7D can be used for a position input operation in which the user inputs or specifies a position in the visual field VR. The position input based on the positions of the X-direction guide GX and the Y-direction guide GY is detected, for example, by the input analysis unit 163 as a position in the visual field VR. In this case, the input analysis unit 163 may perform a process of converting the detected position into coordinates of a display position on the right LCD 241 and the left LCD 242.
In addition, a trigger for displaying the Y-direction guide GY and the X-direction guide GX and starting the position input may be a trigger that detects whether the external sensor 400 detects a contact between two objects or that detects a contact between the two objects. The contact of one of the objects may be released. That is, the processing execution unit 164 may execute the processing in accordance with a combination of the operation of bringing the operating tool into contact with the external sensor 400 and the operation of bringing the operating tool close to the detection range of the external sensor 400. When displaying the Y-direction guide GY and the X-direction guide GX, a text or an image for guiding the operation method may be displayed under the control of the processing execution unit 164.

  Further, the operation shown in FIGS. 7A to 7D may be performed with one hand. For example, there is a case where the input analysis unit 163 detects an approach at two positions with either the right sensor unit 410 or the left sensor unit 420 of the external sensor 400. In this case, as the set operation, the process execution unit 164 displays a Y-direction guide GY (operation control instruction image) for position input. For example, this corresponds to a case where each of the index finger and the middle finger of the right hand of the user is detected as an operation tool.

  As shown in FIG. 7B, the process execution unit 164 moves the two operating bodies (the index finger and the middle finger of the right hand) up and down (Y direction in FIG. 1) while the Y direction guide GY is displayed. Is detected, and the Y-direction guide GY is moved up and down in response to this operation. The correspondence between the movement amounts of the two operating bodies detected by the input analysis unit 163 and the movement amounts of the Y-direction guide GY is set in the setting data 121 in advance. Then, when the input analysis unit 163 detects that one of the two operating bodies is separated from the frame 3 by a predetermined distance or more, the position in the Y direction is determined.

Further, when the input analysis unit 163 detects the approach at two positions with either the right sensor unit 410 or the left sensor unit 420 of the external sensor 400, the processing execution unit 164 sets the position input X-direction guide. GX (operation control command image) is displayed. For example, this corresponds to a case where each of the index finger and the middle finger of the right hand of the user is detected as an operation tool.
The process execution unit 164 moves the two operating bodies (the index finger and the middle finger of the right hand) right and left (Z direction in FIG. 1) with the X direction guide GX displayed, as shown in FIG. 7D. Is detected, and the X-direction guide GX is moved right and left in response to this operation. The correspondence between the movement amount of the two operating bodies detected by the input analysis unit 163 and the movement amount of the X-direction guide GX is set in the setting data 121 in advance. Then, when the input analysis unit 163 detects that one of the two operating bodies is separated from the frame 3 by a predetermined distance or more, the position in the X direction is determined.
This operation may be applied to position input in the depth direction of the user's field of view through the frame 3 when the two operating bodies are moved in the X direction in FIG.

  FIG. 8 is a diagram showing a manner of operation on the HMD 100 and a specific example of display. FIG. 8A shows an example of the operation, and FIGS. 8B and 8C show display examples corresponding to the operation of FIG. 8A.

FIG. 8A shows a state in which the two operating bodies are brought close to each other in front of the HMD 100. More specifically, in a state in which the two operating bodies RH and LH are separated to the left and right, the right light guide unit 26 is provided. , A state where it is brought close to the front surface of the left light guide section 28. In the example of FIG. 8A, the operating tool RH approaches the sensor 413, and the operating tool LH approaches the sensor 423. FIG. 8A shows a front view of the HMD 100, that is, a view of the user's face in a mounted state from the outside for the sake of convenience of understanding.
As indicated by reference numeral D in FIG. 8A, the detection range D of the external sensor 400 is located in a space extending in front of the right light guide section 26 and the left light guide section 28, and up, down, left and right.

  The operation of FIG. 8A indicates an operation of moving the operating tool RH along the sensor 413 and moving the operating tool LH along the sensor 423. In this case, the input analysis unit 163 detects the operating tool RH and its movement by the sensor 413 of the right sensor unit 410, and detects the operating tool LH and its movement by the sensor 423 of the left sensor unit 420.

The processing execution unit 164 detects two objects by sensors provided in the external sensor 400 (six in the present embodiment), and detects an operation in which the two objects approach or separate from each other. Enlarge or reduce the display.
8B and 8C, the display image V1 is displayed in the visual field VR of the user. The processing execution unit 164 enlarges the image being displayed when the input analysis unit 163 detects an operation of separating the two objects as an operation of the external sensor 400. For example, the process execution unit 164 changes the display magnification of the display image V1 from the display state of FIG. 8C and shifts to the display state of FIG. In addition, when the input analysis unit 163 detects an operation in which the two objects approach each other as an operation of the external sensor 400, the process execution unit 164 reduces the image being displayed. For example, the processing execution unit 164 changes the display magnification of the display image V1 from the display state of FIG. 8B and shifts to the display state of FIG.

  The correspondence between the change amount of the distance between the two objects detected by the external sensor 400 and the change amount of the display magnification or the display size of the display image V1 is set in the setting data 121 in advance.

  As shown in FIGS. 8A to 8C, in the HMD 100, the user moves the hand closer to the frame 3 so as to detect it as at least two objects, and perform an operation of approaching or separating the two objects. Thus, the display size of the display image V1 can be changed. This operation is not limited to the case where one sensor continues to detect the object. For example, an object may be detected across the sensors 411, 412, and 413. In this case, the input analysis unit 163 analyzes a detection value that continuously detects the object with an adjacent sensor, and outputs one object. It is only necessary to detect an operation in which the object moves across different sensors. In addition, the condition may be that the right sensor unit 410 and the left sensor unit 420 detect two objects one by one, or the condition for the sensor that detects the object may not be set. For example, when the input analysis unit 163 is configured to be able to detect two objects by one sensor, the approach / separation may be detected for two objects detected by one sensor. Therefore, for example, the display size of the display area VA may be changed by using one hand of the user, bringing two fingers close to the frame 3, and moving the two fingers closer or further away.

  When the external sensor 400 detects two operating bodies as shown in FIG. 8A, the display image V1 may be rotated in response to a rotation operation of these two operating bodies. In this case, the input analysis unit 163 may determine that the rotation has been performed when the detection positions of the two operating bodies relatively move and the movement of one detection position is larger than the movement of the other detection position. The rotation angle for rotating the display image V1 may be determined according to the degree of rotation or the rotation speed of the operating tool. Further, for example, when the operating position does not substantially move and the operating body rotates a plurality of times, the rotation angle may be determined based on the number of rotations.

In addition, the following example is given as an operation on the display image V1 in the display area VA.
1. Enlargement / reduction rotation: The external sensor 400 detects contact or proximity at two points, and makes any one point contact one of the sensors of the right sensor unit 410 while rotating the frame 3 clockwise. Here, one of the sensors detects contact or proximity at another point. In response to this operation, the processing execution unit 164 rotates the display image V1 while enlarging it. An operation of rotating the display image V1 while reducing it by rotating the display counterclockwise may be performed.
2. Enlargement / reduction rotation: When the external sensor 400 detects contact or proximity at two points and enlarges / reduces the display image V1, another operation body is positioned outside the visual field VR, and the rotation angle of this operation body Alternatively, the display image V1 is rotated by a rotation amount according to the speed.
3. In the detection range of the external sensor 400, the display image V1 is enlarged according to the operation of moving the operating tool close to the external sensor 400, and the display image V1 is reduced according to the operation of separating the operating tool H from the external sensor 400. You may. The operation of moving the operating tool toward or away from the external sensor 400 may be combined with the operation of bringing another operating tool into contact with the sensor of the external sensor 400.

  FIG. 9 is a diagram illustrating another specific example of an operation mode and display of the HMD 100. FIGS. 9A and 9B show an example of an input operation using the virtual keyboard VKB.

  FIGS. 9A and 9B show a state in which the virtual keyboard VKB is called by operating the external sensor 400 or the touch pads 401 and 402. The virtual keyboard VKB is a keyboard-shaped image including a plurality of input keys, and is displayed by the image display unit 20 in the display area VA. When the user performs a position input operation in the display area VA in a state where the virtual keyboard VKB is displayed and visible, the pointer VP is displayed at the input position. Here, when the user performs a determination operation, a key at a position overlapping the pointer VP is selected, and a character is input. As the operation of specifying the position of the pointer VP, for example, the position input operation described with reference to FIGS.

  FIG. 9B shows an example in which the application screen VAP is displayed together with the virtual keyboard VKB. The application screen VAP is, for example, a screen for editing a document by an application program executed by the control unit 140. The application screen VAP is a screen corresponding to a process executed by the user in the application program. When the application program enables the operation by the GUI, for example, an operation of selecting an icon arranged in the icon arrangement unit IA is performed.

In the example of FIG. 9B, an operation of selecting a key of the virtual keyboard VKB and an operation of selecting an icon arranged in the icon arrangement unit IA of the application screen VAP are performed. All of these can use the position input operation. In addition, the first state of operating the virtual keyboard VKB and the second state of operating the application screen VAP can be switched by an operation using the external sensor 400.
For example, the first state and the second state may be switched by an operation of covering the right sensor unit 410 or the left sensor unit 420 with an operating body such as a user's hand. In this case, the input analysis unit 163 determines the right sensor unit based on the detection value at which each sensor of the right sensor unit 410 detects the object almost simultaneously or the detection value at which each sensor of the left sensor unit 420 detects the object almost simultaneously. An operation for covering 410 or the left sensor unit 420 is detected. Further, the first state and the second state may be switched by a double tap operation on the touch pads 401 and 402.

In the example shown in FIGS. 9A and 9B, the processing execution unit 164 enlarges or reduces the display size of the virtual keyboard VKB by the operation described with reference to FIGS. 8A to 8C. You may. Further, a process of rotating the virtual keyboard VKB, a change in the key arrangement in the virtual keyboard VKB, a change in the type of keyboard (Japanese keyboard, English keyboard, numeric keypad, etc.) may be performed.
In addition, after a character is input using the virtual keyboard VKB in FIG. 9A, the input character or character string may be transferred to an application program for processing. Examples of the operation of selecting an application program to be transferred and instructing execution of transfer include an operation of bringing the operating tool closer to the external sensor 400, an operation of moving the operating tool in a detection range of the external sensor 400, and the like.
Further, at the start of the display of the virtual keyboard VKB, at the time of displaying or switching between the virtual keyboard VKB and the application screen VAP, an image or text for guiding the operation may be displayed in a pop-up display or a balloon.

  FIG. 10 is a diagram illustrating another specific example of the operation mode and display on the HMD 100, and illustrates an example of an input operation using the virtual remote controller VRC.

  FIG. 10 shows a state where the virtual remote controller VRC is called by operating the external sensor 400 or the touchpads 401 and 402. The virtual remote controller VRC is an image that includes a plurality of input keys and imitates a remote controller that operates a home appliance or an AV (Audio Visual) device. The virtual remote controller VRC is provided with a plurality of operation switches like a remote controller in a real space. With the virtual remote controller VRC displayed, the operation switch of the virtual remote controller VRC can be selected by the above-described position input operation.

  Further, the display size of the virtual remote controller VRC can be enlarged / reduced by the operation of instructing the enlargement / reduction of the display image V1 described above. In addition, the display angle of the virtual remote controller VRC can be changed, that is, rotated, by the operation of instructing the rotation of the display image V1 described above.

  FIG. 10 shows a direction instruction guide VAG. The direction instruction guide VAG is an image for performing four direction instruction operations corresponding to up, down, left, and right. By the above-described position input operation on the direction instruction guide VAG, any one of up, down, left and right methods can be instructed.

  In the state of FIG. 10, a circular pointer VP is displayed at a position designated by the position input operation, and guides the position input operation with respect to the virtual remote controller VRC and the direction instruction guide VAG.

The display of the virtual remote controller VRC and the direction instruction guide VAG can be started by an operation similar to the above-described operation of displaying the virtual keyboard VKB or by a menu operation described later.
When the operation switch of the virtual remote controller VRC is selected by a position input operation, the process execution unit 164 controls the communication unit 117 to transmit a signal to an external device to be operated by the virtual remote controller VRC. The external device to be operated is, for example, a household appliance or an AV device as described above, but is not limited as long as the device can be controlled or operated by a wireless signal. Further, the communication unit 117 may transmit a wireless signal such as a wireless LAN or Bluetooth according to the control of the process execution unit 164. When the communication unit 117 includes an IR transmitter (not shown) that transmits an infrared signal, the communication unit 117 may transmit an infrared signal to a device to be operated under the control of the processing execution unit 164.

In the HMD 100, the camera 61 can capture an image of the user's line of sight. In this case, execution of imaging by the camera 61 can be instructed by an operation using the external sensor 400.
Specifically, in response to an operation of bringing the operating tool into contact with the external sensor 400 for a predetermined time or more, or an operation of covering the right sensor unit 410 or the left sensor unit 420, the process execution unit 164 instructs the focus adjustment of the camera 61. I do. The input analysis unit 163 may detect an operation of covering the right sensor unit 410 or the left sensor unit 420 when a predetermined number or more of the sensors included in the external sensor 400 detect contact or proximity at substantially the same time. After this operation, when the same operation is performed, the shutter of the camera 61 (execution of photographing) may be performed. Further, the zoom adjustment of the camera 61 may be performed by the above-described operation of enlarging or reducing the display image V1.

  FIG. 11 is a diagram showing a mode of operation on the HMD and a specific example of display, and particularly shows an example in which an operation in the Z direction is used. FIG. 11A shows the visual field VR of the user, and FIG. 11B particularly shows the operation guide area GA displayed in the display area VA. The screen including the operation guide area GA in FIG. 11A corresponds to the operation screen and the guidance screen, and the operation guide area GA particularly corresponds to the guidance screen.

  In the example of FIG. 11A, an operation guide area GA for performing a display for guiding the operation is arranged in the visual field VR. The operation guide area GA is an area for displaying operation icons A1 to A4, a menu button B1, a menu list B2, a guide display B3, and the like. The operation icons A1 to A4, the menu button B1, the menu list B2, the guide display B3, and the like correspond to an operation control instruction image. The operation guide area GA is arranged in the display area VA as a colored area or an area having a frame so that the user can visually recognize the operation guide area GA. The menu list B2 corresponds to a menu screen.

  FIG. 11B shows the operation guide area GA. The operation guide area GA illustrated in FIG. 11B is a rectangle having four sides of an upper end US, a lower end GS, a right end RS, and a left end LS. The upper end US, the lower end GS, the right end RS, and the left end LS are each trapezoidal, and form an image with a sense of depth. To the user, it looks like a rectangular frame extending toward the back (front) of the field of view (field of view). The operation guide area GA may be displayed as a stereoscopic image by the display control unit 190 displaying an image having parallax by the right display driving unit 22 and the left display driving unit 24. Further, as a planar image, as shown in FIG. 11B, the upper end portion US, the lower end portion GS, the right end portion RS, and the left end portion LS may be displayed as a trapezoidal image having a sense of depth.

  In the operation guide area GA, a direction that appears to move in the depth direction is shown as a Z direction in FIG. The horizontal direction in the operation guide area GA is defined as the X direction, and the vertical direction in the operation guide area GA is defined as the Y direction.

  As shown in FIG. 11A, operation icons A1 to A4 are displayed in the operation guide area GA. The operation icons A1 to A4 are arranged at four corners of the operation guide area GA, and correspond to operations on the four corners of the external sensor 400.

For example, the operation icons A1 and A2 displayed at the lower right corner of the operation guide area GA correspond to the lower right corner of the detection range of the external sensor 400. An operation of touching or approaching the operating tool to the lower right corner of the detection range of the external sensor 400 is detected as an operation of selecting the operation icons A1 and A2. The operation icons A1 and A2 move in the X, Y, and Z directions with the subsequent movement of the operation tool. A mode in which the operation icons A1 and A2 are operated with two fingers of one hand of the user is also possible.
The operation icon A3 corresponds to the lower left corner of the detection area of the external sensor 400. An operation of touching or approaching the operating tool to the lower left corner of the detection range of the external sensor 400 is detected as an operation of selecting the operation icon A3. The operation icon A3 moves in the X, Y, and Z directions with the subsequent movement of the operation tool.
The operation icons A4 and A5 correspond to the upper right corner of the detection area of the external sensor 400. An operation of touching or approaching the operating tool to the upper left and right corners of the detection range of the external sensor 400 is detected as an operation of selecting the operation icons A4 and A5. The operation icons A4 and A5 move in the X, Y, and Z directions with the subsequent movement of the operation tool. A mode in which the operation icons A4 and A5 are operated with two fingers of one hand of the user is also possible.
Further, the guide display B3 is, for example, an icon for performing a determination instruction, and corresponds to the lower left corner of the detection area of the external sensor 400. An operation of touching or approaching the operating tool to the lower left corner of the detection range of the external sensor 400 is detected as an operation of selecting the guide display B3.

  The menu list B2 is a list displayed by operating the menu button B1, and is a list of setting items and the like. Items displayed in the menu list B2 can be designated by the above-described position input operation.

  In the present embodiment, the detection range in which the external sensor 400 detects an object is, as shown by a symbol D in FIGS. 7A, 7B, and 8A, a right light guide section 26 and a left light guide section. The space extends in front of and around the portion 28. The operation guide area GA displayed in the display area VA corresponds to the detection range D. Therefore, the right half of the operation guide area GA is associated with the right half of the detection range D, and the left half of the operation guide area GA is associated with the left half of the detection range D. The detection of an object in the right half of the detection range D is actually performed by the sensors 411, 412, and 413 of the right sensor unit 410. The detection of an object in the left half of the detection range D is performed by the sensors 421, 422, and 423 of the left sensor unit 420. That is, the right half of the operation guide area GA is associated with the detection result of the right sensor unit 410, and the left half of the operation guide area GA is associated with the left sensor unit 420.

  As described above, the display area VA is recognized by synthesizing an image visually recognized by both eyes of the user by the right light guide unit 26 and the left light guide unit 28 by the visual function of the user. Therefore, in the right light guide 26, the left half of the operation guide area GA is located on the left half of the image displayed by the right light guide 26, but is associated with the left sensor 420. Similarly, in the left light guide section 28, the right half of the operation guide area GA is located on the right half of the image displayed by the left light guide section 28, but is associated with the right sensor section 410. As described above, the operation guide area GA is associated with the entire external sensor 400 and the positional relationship so as to correspond to the detection range D that the user is conscious of. Therefore, the user is conscious of the entire frame 3 as a range in which the operating tool approaches and touches, and intuitively operates the operation icons A1 to A5 displayed in the operation guide area GA, the menu button B1, the guide display B3, and the like. Is possible.

  Also, during the display shown in FIG. 11A, the HMD 100 can execute the normal operation mode and perform operations on the operation icons A1 to A5, the menu button B1, the menu list B2, and the guide display B3. While the operation guide area GA is being displayed, a position designation operation in the display area VA is performed by operating the touch pads 401 and 402, and the operation icons A1 to A5, the menu button B1, the menu list B2, and the guide display B3 can be selected. After the selection operation, the selection can be confirmed by performing a tap operation or a pressing operation on the touch pads 401 and 402.

  In the above embodiment, the size or width of the detection range of the external sensor 400 in the proximity operation mode may be switched according to the image displayed by the process execution unit 164, that is, the form of the user interface. The size of the detection range of the sensor may be switched to, for example, three stages of contact, proximity, and medium proximity. For example, in the display states of FIGS. 7B and 7D, the detection range is set to the middle and near, and in the display states of FIGS. 8B to 8C, the detection range is set to the contact state. In the state, the detection range may be close. For example, when the detection range is set to contact, the input detection unit 162 detects contact with each sensor, and when set to close, the detection range is set to within 2 cm from the sensor surface, and when set to medium proximity. May have a detection range within 5 cm from the sensor surface. In addition, the input detection unit 162 may be configured to distinguish between an operation that contacts the sensor surface and an operation that does not contact the sensor surface in the state where the sensor is set to the proximity and the middle proximity.

As described above, the HMD 100 according to the first embodiment to which the present invention is applied has a right side as a display unit that displays an image so that a user can visually recognize a real space in a state where the HMD 100 is mounted on the head of the user. The light guide unit 26, the left light guide unit 287, or the image display unit 20 is provided. The HMD 100 includes an external sensor 400 that detects the approach of the operating tool to the vicinity of the right light guide unit 26 and the left light guide unit 28. When the external sensor 400 detects two or more operations around the right light guide unit 26 and the left light guide unit 28, the control unit 140 executes a process set in association with the two or more operations.
According to the configuration of the HMD 100 and the control method of the HMD 100, the user can easily operate the HMD 100 without touching the frame 3, and for example, to improve the convenience in the case of performing the operation during the work. Can be. Further, since the processing to be executed is associated with two or more operations on the frame 3, erroneous operations can be prevented.

  The image display unit 20 includes a left light guide unit 28 serving as a left-eye display unit positioned on the left eye side of the user when mounted on the head of the user, and a right eye unit positioned on the right eye side. A right light guide unit 26 as a display unit is provided. The control unit 140 sets a process to be executed by the control unit 140 in association with a combination of an operation on the left light guide unit 28 and an operation on the right light guide unit 26 detected by the external sensor 400. Thereby, since the process according to the combination of the operations on the right side and the left side of the HMD 100 is performed, the risk of erroneous operation is small, and therefore, a sophisticated and complicated process can be assigned to the proximity operation. Thereby, the HMD 100 can be operated more easily.

  In addition, since the external sensor 400 detects the approach and contact operation of the object to the left light guide unit 28 and the approach and contact operation of the object to the right light guide unit 26, the user operates the HMD 100 in various modes. Operation can be prevented.

  As shown in FIGS. 7A to 7D, the control unit 140 operates the external sensor 400 to move an object in both the right light guide unit 26 and the left light guide unit 28 in the Z direction and the Y direction. Is detected, a position input in the vertical or horizontal direction in the display area of the image display unit 20 is accepted. Thereby, the position input can be performed by the operation of moving the object closer to the image display unit 20, and the HMD 100 can be more easily operated.

7A to 7D, the external sensor 400 detects the display position of the image displayed by the right light guide unit 26 and the left light guide unit 28 under the control of the control unit 140. The position does not have to match the operation position of the operation. In the examples of FIGS. 7A to 7D, the X-direction guide GX and the Y-direction guide GY move in accordance with the directions in which the operating tools RH and LH move. In this example, the display position of the guide may be different from the detection position where the operation tools RH and LH are detected. In other words, the control unit 140 accepts an input corresponding to a position different from the detected operation position in response to an operation at two or more positions detected by the external sensor 400.
In addition, the display position of the guide may not be a position corresponding to or coincident with the detection position, but may be a position of the guide at the time of detecting the operating tools RH and LH, or a position that is easily visible to the user. In this case, operability such as an input device for inputting a relative position such as a mouse can be realized.

  The external sensor 400 detects the approach of an object to the left light guide unit 28 (display unit for the left eye) and the approach of an object to the right light guide unit 26 (display unit for the right eye). The HMD 100 can be operated without touching the.

The external sensor 400 is configured to detect contact with the outer surfaces of the right light guide unit 26 and the left light guide unit 28 and the approach of an object from outside the right light guide unit 26 and the left light guide unit 28. Is also good.
The control unit 140 may be configured to rotate the display in the display area of the image display unit 20 when the rotation operation is detected by the external sensor 400.
When an operation on two points is detected by the external sensor 400 and the distance between the detected operation positions of the two points changes, the control unit 140 responds to the change in the distance in the display area of the image display unit 20. The display may be configured to be enlarged or reduced.

  Further, an operation of contacting the surfaces of the right light guide unit 26 and the left light guide unit 28 for a predetermined time or more by the external sensor 400, or an operation of covering an area of the right light guide unit 26 and the left light guide unit 28 with a predetermined area or more with an object. When it is detected, the control unit 140 causes the camera 61 to execute imaging. Thereby, the imaging can be easily performed by the operation on the frame 3, and the erroneous operation can be prevented.

  The external sensor 400 may be configured to detect the approach of a pointer to a holding unit that holds the right light guide unit 26 and the left light guide unit 28, which are optical components constituting the display unit. This makes it possible to more reliably detect the approach of the operating tool to the vicinity of the right light guide unit 26 and the left light guide unit 28 and reflect the approach on the display.

  In addition, the HMD 100 includes an external sensor 400 that detects the approach of the operating tool to the vicinity of the right light guide unit 26 and the left light guide unit 28, and touch pads 401 and 402 that receive an operation. The control unit 140 includes a normal operation mode in which processing is performed in response to an operation received by the touchpads 401 and 402, and a proximity mode in which processing including display on the image display unit 20 is performed in response to a detection result of the external sensor 400. The operation mode is switched and executed. For this reason, for example, it is possible to improve convenience when performing an operation during work. By switching between the proximity operation mode for performing an operation by approach and the normal operation mode, erroneous operation can be prevented.

  The control unit 140 may cause the image display unit 20 to display, for example, a screen including the menu list B2 as a menu screen for operation. In this case, in the normal operation mode, the control unit 140 can process an operation accepted by the touchpads 401 and 402 as an operation of selecting or confirming the menu list B2. In the proximity operation mode, the control unit 140 can perform selection or determination processing in the operation guide area GA including the menu list B2 in accordance with the approach or separation of an object detected by the external sensor 400. This makes it possible to easily operate the HMD 100 using the menu screen in both the normal operation mode and the proximity operation mode.

  The control unit 140 displays an operation guide area GA as a guide screen including information on an operation detected by the external sensor 400 by the right light guide unit 26 and the left light guide unit 28 as illustrated in FIG. May be. In this case, the information displayed on the left half of the operation guide area GA corresponds to an operation on the left portion of the left eye display unit, and the information displayed on the right half of the operation guide area GA is the right side of the right eye display unit. It is associated with an operation on a part. Thus, the user can intuitively perform the operation while viewing the guide screen by utilizing the shape of the frame 3 mounted on the user's head.

  The control unit 140 displays the operation screen in a form that can be visually recognized by both the right and left eyes of the user by the right light guide unit 26 and the left light guide unit 28. Then, as shown in FIG. 11A, the control unit 140 changes the display of the left half of the operation screen in response to the operation on the left portion of the left-eye display unit. The display on the right half of the operation screen is changed in response to the operation on the right part of the display unit for the right eye. Thus, the display can be changed in a manner suitable for the user's feeling by utilizing the shape of the image display unit 20 mounted on the user's head.

  The control unit 140 can change the display corresponding to the operation of moving the object in the vertical direction, that is, the Y direction, and the operation of moving the object in the horizontal direction, that is, the Z direction, based on the user's head detected by the external sensor 400. . The control unit 140 changes the display of the image display unit 20 in response to an operation of the object approaching and separating from the image display unit 20 detected by the external sensor 400, that is, an operation in the X direction. Accordingly, since the operation of moving the object toward and away from the image display unit 20 and the operation of moving the object up and down and left and right are detected, various operations on the HMD 100 can be performed more easily.

  Further, the external sensor 400 includes sensors 411, 412, 413, 421, 422, and 423, which are proximity sensors disposed around the right light guide 26 and the left light guide 28. The control unit 140 associates the display with the operation control based on the movement of the operation tool detected by the proximity sensor. Accordingly, the proximity sensor disposed around the right light guide unit 26 and the left light guide unit 28 more reliably detects the approach of the operating body to the vicinity of the right light guide unit 26 and the left light guide unit 28, and displays the display. Can be reflected.

  Further, when the proximity sensor detects the approach of the operating tool to the vicinity of the right light guide unit 26 and the left light guide unit 28 by the proximity sensor, for example, as illustrated in FIG. An operation control command image is displayed by the light unit 26 and the left light guide unit 28. The control unit 140 may associate the operation by the touchpads 401 and 402 with the operation control instruction image. In this case, the operation is performed using an operation control command image functioning as a GUI, for example, by using the movement of the operating body detected by the proximity sensor disposed around the right light guide unit 26 and the left light guide unit 28. It is possible to improve operability.

  The control unit 140 may display, for example, a Y-direction guide GY and an X-direction guide GX as operation control instruction images illustrated in FIG. 7 when the approach to the two positions is detected by the proximity sensor. That is, when an operation of two operating bodies such as two fingers is detected, a Y-direction guide GY or an X-direction guide GX is displayed as an operation control instruction image corresponding to these two operating bodies. Then, even if the movement in the predetermined direction of the Y-direction guide GY or the X-direction guide GX is associated with the movement in the predetermined direction of the operating tool detected in a state where the operation control instruction images corresponding to these two positions are displayed. Good. In this case, the operation control instruction image can be moved in a direction corresponding to the movement of the operating tool by performing an operation of bringing the operating tool closer to the proximity sensor at two positions. In addition, this operation can be switched and executed as an operation different from a case where one operating body such as a hand or one finger is approached. Thus, operations such as a position instruction and a direction instruction can be easily performed with the two operating bodies.

  In addition, the control unit 140 may cause the right light guide unit 26 and the left light guide unit 28 to display a display for guiding the arrangement state of the proximity sensor with respect to the right light guide unit 26 and the left light guide unit 28. For example, an image showing the position, number, detection range, and the like of the sensors 411, 412, 413, 421, 422, and 423 disposed around the right light guide unit 26 and the left light guide unit 28 is displayed on the right guide. The display may be performed by the light unit 26 and the left light guide unit 28. In this case, the operation of approaching the operation body can be performed more reliably.

  Further, the right light guide 26 and the left light guide 28 are configured so that the real object can be visually recognized by transmitting the external light OL, and display an image so that the real object is visually recognized overlapping with the real object. Accordingly, a user wearing the right light guide unit 26 and the left light guide unit 28 that can visually recognize the real object by transmitting the external light OL and visually recognize the image can see the real object and the image overlapping with each other. Is done. Then, an operation of bringing the operating body close to the right light guide unit 26 and the left light guide unit 28 can be performed on the visually recognized image. Thus, the display related to the real object can be easily controlled by a simple operation of bringing the operating tool closer to the right light guide unit 26 and the left light guide unit 28.

  In the above-described embodiment, the display of the operation control instruction images such as the Y-direction guide GY and the X-direction guide GX displayed by the right light guide unit 26 and the left light guide unit 28 may be changed according to the external environment. Good. For example, the brightness (luminance and illuminance of external light) around the HMD 100 is detected by the camera 61, and the display brightness and the color arrangement of the right light guide unit 26 and the left light guide unit 28 are automatically adjusted according to the detected brightness. May be. For the purpose of suppressing the power consumption of the HMD 100, an energy saving mode or the like may be executed, and the operation control instruction image may be displayed on only one of the right light guide unit 26 and the left light guide unit 28. In this case, the input analysis unit 163 may detect the approach of the operating tool only on the display side of the right sensor unit 410 or the left sensor unit 420. Alternatively, a marker for specifying an object in the real space, which is a target on which the AR image is superimposed, may be used, and this marker may be controlled by the proximity sensor. Alternatively, a gaze sensor for detecting the gaze of the user may be provided, and the object may be specified in combination with the detection result of the gaze sensor.

Further, the processing unit 10 may be configured to be able to display a GUI that conforms to the configuration of the external sensor 400 in the frame 3, or may be configured to be able to provide this GUI by an application program. In this case, when the camera 61 captures an image of a mirror image of the user wearing the HMD 100 or a marker corresponding to the HMD 100, the control unit 140 specifies the model number of the HMD 100 based on the captured image data. The control unit 140 may acquire data of a GUI (for example, the operation control command image described above) that matches the model number of the specified HMD 100, and display the GUI. In this case, the GUI data and the data for specifying the model number and the like of the HMD 100 may be stored in the storage unit 120 in advance by the HMD 100 or may be acquired by communicating with an external device by the communication unit 117 or the like. Good.
In addition, the control unit 140 may execute a process corresponding to the operation of the operation tool detected by the external sensor 400 and the operation control instruction image, and in this case, may display a GUI or the like that indicates the operation content. .

[Second embodiment]
FIG. 12 is an external view showing a configuration of an HMD 100A according to a second embodiment to which the present invention is applied.
The HMD 100A shown in FIG. 12 has a line-of-sight sensor 65 at a position facing the user's eye in the HMD 100 of the first embodiment. The line-of-sight sensors 65 are provided as a pair at the center position between the right light guide 26 and the left light guide 28 so as to correspond to the right eye and the left eye of the user. The line-of-sight sensor 65 includes, for example, a pair of cameras that respectively image the right eye and the left eye of the user. The line-of-sight sensor 65 captures an image under the control of the control unit 140, and the control unit 140 detects the reflected light or the image of the pupil on the eyeball surface of the right eye RE and the left eye LE from the captured image data, and specifies the gaze direction. .

Thereby, for example, in the position input operation in the normal operation mode, the control unit 140 may specify the line of sight of the user, and may detect the position of the line of sight as an input. Further, in the proximity operation mode, the control unit 140 may accept a determination operation based on the detection result of the eye gaze sensor 65. For example, when a menu item or the like in the menu list B2 is selected based on the position or movement of an object detected by the external sensor 400, the eye-gaze sensor 65 detects the user's blink and determines the blink as a selection. May be detected.
Regarding other operations, the HMD 100A can execute the same operation as the HMD 100.

[Third embodiment]
FIG. 13 is an external view showing a configuration of an HMD 100B according to a third embodiment to which the present invention is applied.
The HMD 100B shown in FIG. 13 does not have the touch pads 401 and 402 in the HMD 100 according to the first embodiment. The HMD 100B includes a direction key 67 and a decision key 68 below the camera 61.

In the HMD 100B, direction operations in four directions, up, down, left and right, can be performed by operating the direction keys 67, and position input can be performed using these direction instructions. The direction keys 67 can be used in the normal operation mode, like the touchpads 401 and 402.
The confirmation key 68 located at the center of the direction key 67 functions as a key for instructing confirmation by a pressing operation.
The control unit 140 changes the display mode according to the operation of the direction key 67 and the confirmation key 68 in the normal operation mode.
Regarding other operations, the HMD 100B can execute the same operation as the HMD 100.

[Fourth embodiment]
FIG. 14 is an external view showing a configuration of an HMD 100C according to a fourth embodiment to which the present invention is applied.
The HMD 100C illustrated in FIG. 14 includes sensors 451 to 456 and 462 to 466 instead of the sensors 411, 412, 413, 421, 422, and 423 included in the HMD 100 according to the first embodiment.

  The sensor 451 is located above the camera 61 and is arranged on the upper end surface of the frame 3 or on the upper part of the front surface. The sensors 452, 453, 454, 455, and 456 are arranged around the right light guide 26. In addition, the sensors 462, 463, 464, 465, and 466 are arranged around the left light guide unit 28.

  The sensors 451 to 456 and 462 to 466 can be used as sensors constituting the external sensor 400. Each of the sensors 451 to 456 and 462 to 466 can detect approach and contact of an object, similarly to the sensors 411 to 413 and 421 to 423, and can be configured by, for example, a capacitive proximity sensor.

  The sensors 451 to 456 and 462 to 466 are preferably arranged so as not to block the external light OL passing through the right light guide 26 and the left light guide 28, respectively. Further, the intervals and arrangement of the sensors 451 to 456 and 462 to 466 are arbitrary, and can be appropriately set according to the characteristics of each sensor. As a result, the same detection range D as in the first embodiment can be realized. Is preferred.

  The detection range D when the external sensor 400 is configured by the sensors 451 to 456 and 462 to 466 can be substantially the same as the example described in the first embodiment. Further, the functions and operations of the HMD 100C can be the same as those of the HMD 100.

As described above, the number and arrangement of the proximity sensors in the HMD are arbitrary, and more sensors may be used. In the HMDs 100, 100A, 100B, and 100C, a sensor that detects approach of an object and a sensor that detects contact with an object may be arranged as separate sensors.
In any of these cases, the input analysis unit 163 appropriately analyzes the detection value of the sensor, so that the operation performed by the user using the operating tool can be appropriately detected.

[Fifth Embodiment]
FIG. 15 is an external view showing a configuration of an HMD 100D according to a fifth embodiment to which the present invention is applied.
The HMD 100D is another mode of a display device that allows a user to visually recognize a virtual image while being worn on the user's head, and has a spectacle shape. FIG. 15 is a perspective view seen from the front side.

The frame 300 constituting the main body of the HMD 100D includes a transmissive display section 337 located in front of the user's face, a right storage section 335 and a left storage section 336 that instruct the transparent display section 337 from the side, and a side of the user. It has a right holding part 331 and a left holding part 332 that are in contact with the head.
The transmissive display section 337 has a plate shape as a whole, and a nose contact portion 338 which is a concave portion capable of contacting the nose of the user is formed at the center of the lower end of the transmissive display section 337. The transmissive display unit 337 is located in front of both eyes of the user when the user wears the HMD 100D, and allows the user's eyes to visually recognize an image.

  The right storage section 335 is fixed to a side end of the transmissive display section 337, and supports the right holding section 331. A case 333 is attached inside the right housing section 335. The left storage section 336 is fixed to a side end of the transmissive display section 337 and supports the left holding section 332. A case 334 is mounted inside the left storage section 336. The frame 300 included in the HMD 100D of the present embodiment has a structure in which the right holding unit 331, the right housing unit 335, the transmissive display unit 337, the left housing unit 336, and the left holding unit 332 are fixed to each other. The directions called right and left in the configuration of the HMD 100D are right and left of the user with the HMD 100D mounted. The same applies to the names of the right eye side and the left eye side.

The right holding unit 331 and the left holding unit 332 hold the frame 300 on the user's head like a temple of glasses. The frame 300 has a right transmission part 301A and a left transmission part 302A that are located in front of the user and allow the user to visually recognize an image. The right transmission part 301A is located in front of the right eye of the user, and the left transmission part 302A is located in front of the left eye of the user. The central position where the nose pad 338 is provided in the transmissive display unit 337 is located in front of the user's eyebrows, and the camera 61 is provided at this position.
The right transmissive unit 301A corresponds to the right display unit 301 and transmits the external light OL to allow the right eye of the user to visually recognize the outside scenery, and guides the image light L to the right eye of the user to visually recognize the image. . The left transmissive portion 302A corresponds to the left display portion 302 and transmits the external light OL to allow the left eye of the user to visually recognize the outside scenery, and guides the image light L to the left eye of the user to visually recognize the image. .

  The configurations of the camera 61, the external sensor 400, and the touchpads 401 and 402 are the same as those in the first embodiment except for a detailed shape and the like.

The frame 300 includes a right display drive unit 22 and a left display drive unit 24, like the image display unit 20 included in the frame 3 (FIG. 1), and includes a camera 61, a 9-axis sensor 66, an external sensor 400, a touch pad 401, The image display unit 21 including the image display unit 402 is provided.
The case 333 contains, for example, the right display drive unit 22, and the case 334 contains, for example, the left display drive unit 24.
Further, other parts constituting the image display unit 21 may be arranged in the right housing unit 335, the left housing unit 336, and the like of the frame 300.
In the frame 300, each unit constituting the processing unit 10 (FIG. 4) is arranged.

On the front side of the transmissive display section 337, a right sensor section 410 and a left sensor section 420 are arranged. The sensors 411, 412, 413 constituting the right sensor unit 410 are arranged around the right transmission unit 301A. More specifically, the sensor 411 is disposed above the right transmission part 301A, the sensor 412 is disposed on the right holding part 331 side of the right transmission part 301A, and the sensor 413 is disposed below the right transmission part 301A.
The sensor 421 included in the left sensor unit 420 is disposed above the left transmission unit 302A, the sensor 422 is disposed on the left holding unit 332 side of the left transmission unit 302A, and the sensor 423 is disposed below the left transmission unit 302A. Is done.

  The right sensor unit 410 constitutes a detection unit corresponding to a portion located on the right eye side of the user in the transmissive display unit 337, and the left sensor unit 420 is located on the left eye side of the user in the transmissive display unit 337. Corresponding to the part.

  In this configuration, each of the sensors constituting the right sensor unit 410 and the left sensor unit 420 is disposed on a transmissive display unit 337 that includes a right transmissive unit 301A and a left transmissive unit 302A and is integrally configured in appearance. With this configuration, it is possible to efficiently detect an operation of approaching the operating tool from the front of the frame 300.

  The present invention is not limited to the configuration of each of the above embodiments, but can be implemented in various modes without departing from the spirit of the invention.

  In each of the above embodiments, the configuration in which the user perceives the outside scene through the display unit is not limited to the configuration in which the right light guide unit 26 and the left light guide unit 28 transmit outside light. For example, the present invention can be applied to a display device that displays an image in a state where the outside scene cannot be visually recognized. More specifically, the present invention is applied to a display device that displays a captured image of the camera 61, an image or CG generated based on the captured image, video stored in advance, video based on video data input from the outside, or the like. Can be applied. As this type of display device, a so-called closed type display device in which the outside scene is not visible can be included. Also, a display device that displays video data or an analog video signal input from the outside without performing processing such as AR display, MR display, or VR display is, of course, included as an application target of the present invention.

  Further, in the above embodiment, the configuration in which the approach of the operating tool is detected by each of the proximity sensors included in the external sensor 400 has been described, but the present invention is not limited to this. For example, a camera (not shown) that captures images from the inside to the outside such as the rims 3a and 3b in the frame 3 may be provided, and the approach of the operating tool may be detected based on a captured image of this camera.

  Further, for example, instead of the frame 3, another type of image display unit such as an image display unit worn like a hat may be adopted, and a display for displaying an image corresponding to the left eye of the user. And a display unit that displays an image corresponding to the right eye of the user. Further, the display device of the present invention may be configured as, for example, a head-mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a personal protective equipment such as a helmet. In this case, a portion that positions the user with respect to the body and a portion that is positioned with respect to the portion can be used as the mounting portion.

  Further, in the above-described embodiment, the configuration has been described in which the image display units 20 and 21 and the processing unit 10 are housed in the frames 3 and 300. However, the processing unit 10 is separated from the frames 3 and 300 and connected via a connection cable. It may be configured to be connected. In this case, a notebook computer, a tablet computer, or a desktop computer may be used as the processing unit 10. Further, as the processing unit 10, a portable electronic device including a game machine, a mobile phone, a smartphone, a portable media player, and other dedicated devices may be used. Further, the processing unit 10 may be configured separately from the frames 3 and 300, and may transmit and receive various signals between the processing unit 10 and the image display units 20 and 21 by wireless communication.

  In addition, for example, as a configuration for generating image light in the image display units 20 and 21, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be employed. Further, as a configuration for generating image light, LCOS (Liquid Crystal on Silicon, LCoS is a registered trademark), a digital micromirror device, or the like can be used.

  The “display unit” in the present invention corresponds to a configuration that emits image light, and it has been described that the HMDs 100 to 100D emit image light as “display”. For example, in the above-described embodiment, the image light is generated by the left and right image light generation units, and the image light is emitted by the right light guide unit 26 and the left light guide unit 28 toward each of the right eye and the left eye of the user. Then, the configuration in which the image light is incident on each of the right eye and the left eye of the user is illustrated. The configuration of the “display section” is not limited to this. That is, any device that irradiates image light may be used. For example, in the configuration of the present embodiment, image light is emitted toward the user's eyes by the “right light guide unit” and the “left light guide unit” having the half mirrors 261A and 262A. As a configuration for generating image light, a right backlight 221 and a left backlight 222 and a right LCD 241 and a left LCD 242 are provided. The “display unit” does not require these components.

  For example, the image light generated by a mechanism built in one or both of the right display drive unit 22 and the left display drive unit 24 is provided on the user side of the frames 3 and 300, that is, on the side facing the user's eyes. The light may be reflected by the reflection mechanism and emitted to the user's eye. Here, as the reflection mechanism, for example, a scanning optical system using a MEMS (Micro Electro Mechanical Systems) mirror can be adopted. That is, a configuration may be adopted in which a scanning optical system having a MEMS mirror that scans the light emitted from the image light generating unit is provided, and light scanned by this scanning optical system is directly incident on the user's eyes. Further, an optical member on which a virtual image is formed by light scanned by the scanning optical system may be provided on the frame 3. This optical member forms a virtual image by the scanning light scanned by the MEMS mirror. In this case, a virtual image is formed on the virtual image forming surface by scanning the light with the MEMS mirror, and the image is visually recognized (recognized) by the user capturing the virtual image with his eyes. The optical component in this case may be one that guides light through multiple reflections, such as the right light guide section 26 and the left light guide section 28 in the above embodiment, and utilizes a half mirror surface. Is also good.

Further, the scanning optical system is not limited to the configuration including the MEMS mirror. The mechanism for generating image light may be a laser light source that emits laser light. For example, the present invention can be applied to a laser retinal projection type head mounted display. That is, the light emitting unit includes a laser light source and an optical system that guides the laser light source to the user's eye, and scans the retina by irradiating the laser light to the user's eye to form an image on the retina. Alternatively, a configuration may be employed in which a user visually recognizes an image.
Further, a configuration may be employed in which the image light is guided to the user's eyes by using a diffraction grating instead of the virtual image forming surface that receives the scanned light. That is, the present invention is not limited to a device that guides image light inside the optical member, and may have only a function of guiding image light by refracting and / or reflecting it toward the user's eyes.

In the configuration including the scanning optical system having the MEMS or the like, the position at which the user visually recognizes the image, that is, the display position of the image can be changed by changing the mounting angle of the scanning optical system in the frames 3 and 300. Therefore, in the process of changing the display position in each of the above embodiments, an operation of changing the angle of the scanning optical system may be performed instead of the operation of changing the display position of the image on the right LCD 241 and the left LCD 242.
In each of the above embodiments, a configuration in which an image is visually recognized by a user's eye may be a configuration in which image light is guided to the eye using a hologram element.

  The optical system that guides the image light to the user's eyes may be configured to include an optical member that transmits external light that enters the apparatus from the outside and that the image light enters the user's eyes together with the image light. Also, an optical member that is located in front of the user's eyes and overlaps part or all of the user's field of view may be used.

  In each of the above embodiments, the configuration in which the virtual images are formed by the half mirrors 261A and 262A on a part of the right light guide part 26 and the left light guide part 28 located in front of the user's eyes has been exemplified. The present invention is not limited to this, and an image may be displayed on a display area having an area occupying the entire surface or the majority of the right light guide section 26 and the left light guide section 28. In this case, the operation of changing the display position of the image may include a process of reducing the image.

  Further, the optical element of the present invention is not limited to the right light guide section 26 and the left light guide section 28 having the half mirrors 261A and 262A, and may be any optical component that allows image light to enter the user's eyes. Specifically, a diffraction grating, a prism, and a holographic display unit may be used.

  In addition, at least a part of each functional block illustrated in FIG. 5 or the like may be realized by hardware, or may be configured to be realized by cooperation of hardware and software. The present invention is not limited to a configuration in which independent hardware resources are arranged. Further, the program executed by the control unit 140 may be stored in the storage unit 120 or another storage device (not shown) in the processing unit 10, or the program stored in an external device may be stored in the communication unit 117 or the like. It is good also as a structure which acquires and runs through. Further, among the components formed in the processing unit 10, only the operation unit 111 may be formed as a single user interface (UI).

  3, 300: frame, 10: processing unit, 20, 21: image display unit, 26: right light guide unit (display unit, display unit for right eye), 28: left light guide unit (display unit, display for left eye) ), 61 ... camera, 65 ... line-of-sight sensor, 66 ... 9-axis sensor, 67 ... direction key, 68 ... fix key, 100, 100A, 100B, 100C, 100D ... HMD (display device), 120 ... storage unit, 121 ... setting data, 122 ... input analysis data, 123 ... image data, 140 ... control unit, 160 ... image processing unit, 161 ... imaging control unit, 162 ... input detection unit, 163 ... input analysis unit, 164 ... processing execution unit, 170 communication control unit, 190 display control unit, 241 right LCD, 242 left LCD, 301 right display unit, 302 left display unit, 400 external sensor (detection unit), 401, 402 touch pad ( Operation ), 410: right sensor section, 411, 412, 413 ... sensor, 420: left sensor section, 421, 422, 423 ... sensor, 451, 452, 453, 454, 455, 456 ... sensor, 462, 463, 464 465, 466: sensor, B2: menu list, D: detection range, GA: operation guide area, GS: lower end, GX: X direction guide, GY: Y direction guide, V1: display image, VA: display area.

Claims (16)

A display unit that displays an image so that the user can visually recognize the real space while being attached to the user's head,
A detection unit that detects the approach of the pointer to the periphery of the display unit,
A control unit that executes a process involving display on the display unit, in response to a detection result of the detection unit,
With
A process executed by the control unit is set in association with two or more operations on the periphery of the display unit detected by the detection unit, and the control unit detects two or more operations on the periphery of the display unit by the detection unit If it is, executes processing that is set,
An operation unit that accepts contact operations is provided,
The control unit includes a normal operation mode in which processing is performed in response to a contact operation received by the operation unit, and a proximity operation mode in which processing including display on the display unit is performed in response to a detection result of the detection unit. And switch to execute
In the normal operation mode, the control unit does not execute a process corresponding to the detection result of the detection unit,
A display device characterized by the above-mentioned. The display unit includes a display unit for the left eye located on the left eye side of the user in a state where the display unit is mounted on the head of the user, and a display unit for the right eye located on the right eye side,
The processing executed by the control unit is set in association with a combination of an operation on the display unit for the left eye and an operation on the display unit for the right eye detected by the detection unit,
The display device according to claim 1, wherein: The detection unit detects the operation and the contact operation of approaching the object to the display unit for the left eye, and detects the operation and the contact operation of approaching the object to the display unit for the right eye,
The display device according to claim 2, wherein: The control unit, when the detection unit detects the approach at two positions, displays an operation control instruction image corresponding to the approach to the two positions,
In a state where the operation control instruction image is displayed, associating the movement of the indicator in the predetermined direction detected by the detection unit with the movement of the operation control instruction image displayed by the display unit in the predetermined direction,
The display device according to claim 2, wherein: The control unit, when the detection unit detects, in both the right-eye display unit and the left-eye display unit, an operation in which an object moves vertically or horizontally with respect to the head of the user. Receiving a vertical or horizontal position input in a display area of the display unit;
The display device according to any one of claims 2 to 4, wherein: The detection unit is configured to detect contact with an outer surface of the display unit and approach of an object from outside the display unit,
The display device according to claim 1, wherein: The control unit, when detecting the operation of rotating the object by the detection unit, to rotate the display in the display area of the display unit,
The display device according to claim 1, wherein: The control unit, when an operation on two points is detected by the detection unit, and, when a distance between the detected operation positions of the two points changes, a display in a display area of the display unit corresponding to the change in the distance. Scaling up or down,
The display device according to claim 1, wherein: Equipped with an imaging unit,
The control unit causes the imaging unit to execute imaging when the detection unit detects an operation of touching the surface of the display unit for a predetermined time or more or an operation of covering an area of the display unit with a predetermined area or more with an object. thing,
The display device according to any one of claims 1 to 8, wherein: The detection unit, the display unit, or to detect the approach of a pointer to a holding unit that holds an optical component constituting the display unit,
Display device according to claim 1, wherein 9 of the. The detection unit has a proximity sensor arranged around the display unit, and associates the display and operation control based on the movement of the indicator detected by the proximity sensor,
Display device according to claim 1, wherein 9 of the. The control unit, when the proximity sensor detects the approach of the indicator to the periphery of the display unit, causes the display unit to display an operation control instruction image, and performs a contact operation by the operation unit and the operation control instruction image. And
The display device according to claim 11, wherein: The display unit is visibly configure by Limi object that transmits external light, to display an image to be visually recognized to overlap with the real object,
Display device according to claim 1, wherein 12 of the.   The control unit determines whether or not the contact operation on the operation unit corresponds to an operation for instructing switching of the operation mode. Switching between the operation mode and the proximity operation mode for execution;
  The display device according to claim 1, wherein: A display unit that displays an image so that the user can visually recognize the real space while being mounted on the user's head, a detection unit that detects approach of a pointer around the display unit, and a contact operation. A control method of a display device comprising: an operation unit for receiving;
A process to be executed in association with two or more operations on the periphery of the display unit detected by the detection unit is set,
Detecting the approach of the pointer to the periphery of the display unit,
If two or more operations to the peripheral the display unit by the detection unit is detected, executes processing that is set,
Switching between a normal operation mode in which processing is performed in response to a contact operation received by the operation unit and a proximity operation mode in which processing accompanied by display on the display unit is performed in response to a detection result of the detection unit Run,
In the normal operation mode, do not execute a process corresponding to the detection result of the detection unit,
A method for controlling a display device, comprising: A display unit that displays an image so that the user can visually recognize the real space while being mounted on the user's head, a detection unit that detects approach of a pointer around the display unit, and a contact operation. A computer-executable program that controls a display device including an operation unit that accepts,
When a process to be executed in association with two or more operations on the periphery of the display unit detected by the detection unit is set,
Detecting the approach of the pointer to the periphery of the display unit,
When two or more operations on the periphery of the display unit are detected by the detection unit, executes a set process ;
Switching between a normal operation mode in which processing is performed in response to a contact operation received by the operation unit and a proximity operation mode in which processing accompanied by display on the display unit is performed in response to a detection result of the detection unit Run,
In the normal operation mode, because the program does not execute the processing corresponding to the detection result of the detecting unit.

Images