JP2017182413A - Head-mounted type display device, control method for the same, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve operability.SOLUTION: A head-mounted type display device comprises a display section. The head-mounted type display device further comprises: a related image display section causing the display section to display an image; and a processing section executing processing related to the image displayed. The processing section comprises: a polyhedron image display section causing the display section to display a polyhedron image in which instructions are allocated to each surface of a polyhedron; a polyhedron direction switching section receiving operation with respect to the polyhedron image and switching a direction in which the polyhedron is displayed; and an instruction execution section executing the instructions allocated to a surface of a predetermined direction in the polyhedron image.SELECTED DRAWING: Figure 11

Description

  The present invention relates to a head-mounted display device, a method for controlling the head-mounted display device, and a computer program.

  In recent years, head-mounted display devices that can display images in front of the user's eyes are becoming popular. A conventional head-mounted display device includes a controller having a touch pad separately from the eyeglass portion to be mounted on the head, and uses this controller to obtain a command (instruction) from a user. Specifically, a button and a cursor are displayed on the glasses portion, the cursor is moved onto the button with a touch pad, and an operation for performing a tap is accepted, whereby a command set for the button can be input.

  The operation of moving the cursor on the button and tapping is complicated. Therefore, Patent Document 1 describes a configuration in which a button can be tapped with a hand put out in a see-through display area.

Japanese Patent Laying-Open No. 2015-519673 JP 2013-542514 A

  However, in the head-mounted display device described in Patent Document 1, even if the button can be tapped by hand, it is necessary to touch the fingertip on the button with high accuracy, and a desired one of many buttons can be selected. Because it was necessary to select a button, the operability could not be improved sufficiently. In addition, in conventional head-mounted display devices, it has been desired to improve search accuracy, to make the device configuration compact, to reduce costs, to save resources, to facilitate manufacturing, and the like.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to one aspect of the present invention, a head-mounted display device including a display unit is provided. The head-mounted display device includes a display control unit that displays an image on the display unit, and a processing unit that executes a process related to the displayed image. The processing unit switches a display direction of the polyhedron in response to an operation on the polyhedron image and a polyhedron image display unit that displays on the display unit a polyhedron image in which an instruction is assigned to each surface of the polyhedron. A polyhedron orientation switching unit; and a command execution unit that executes the command assigned to a plane in a predetermined direction in the polyhedron image. According to the head-mounted display device of this form, the operation of the polyhedron image is changed so that the orientation of the polyhedron is switched by the polyhedron direction switching unit, and the command assigned to the surface of the predetermined orientation in the polyhedron image is It is executed by the execution unit. For this reason, the user of the head-mounted display device can execute processing related to the image only by performing an operation on the polyhedral image. Therefore, the head-mounted display device of this form can improve the operability for the user.

(2) In the head-mounted display device of the above aspect, the display unit is configured to visually recognize an outside scene, and the processing unit uses the related image related to the visually recognized outside scene as the image. You may make it display on a display part. According to this form of the head-mounted display device, the user can execute processing related to the outside scene that can be visually recognized by the display unit.

(3) In the head-mounted display device of the above aspect, the related image includes identification information for identifying a plurality of stores included in the visually recognized outside scene, and the processing executed by the processing unit includes: It may be a store input process for designating one store from the plurality of stores. According to the head-mounted display device of this aspect, the process of designating one store from among a plurality of stores included in the visually recognized outside scene can be performed with good operability.

(4) In the head-mounted display device of the above aspect, the related image includes a support image for supporting a work related to the visually recognized outside scene, and the processing executed by the processing unit includes It may be a process of sequentially switching support images. According to the head-mounted display device of this form, it is possible to perform the process of sequentially switching the support images for the work related to the visually recognized outside scene with good operability.

(5) In the head-mounted display device of the above aspect, the polyhedron image may be a regular hexahedron 3D image, and the surface in the predetermined direction may be a surface facing a user. According to the head-mounted display device of this form, the user can perform processing by a simple and intuitive operation such as rotating a regular hexahedron. Therefore, the operability can be further improved.

(6) In the head-mounted display device of the above aspect, the operation on the polyhedral image may be an operation of flicking with a fingertip. According to the head-mounted display device of this form, the user can perform processing only by performing a simple operation such as flicking. Therefore, the operability can be further improved.

(7) According to another aspect of the present invention, a head-mounted display device including a display unit is provided. The head-mounted display device includes a display control unit that displays an image on the display unit, and a processing unit that executes a process related to the displayed image. The processing unit switches a display direction of the three-dimensional image in response to an operation on the three-dimensional image display unit and a three-dimensional image display unit that displays a three-dimensional image in which a command is assigned to each surface of the three-dimensional image. A stereoscopic direction switching unit; and a command execution unit that executes the command assigned to a plane in a predetermined direction in the stereoscopic image. According to the head-mounted display device of this aspect, by receiving an operation on the stereoscopic image, the stereoscopic orientation is switched by the stereoscopic orientation switching unit, and the command assigned to the surface of the predetermined orientation in the stereoscopic image is the command It is executed by the execution unit. For this reason, the user of the head-mounted display device can execute processing relating to the image only by performing an operation on the stereoscopic image. Therefore, the head-mounted display device of this form can improve the operability for the user.

  The present invention can also be realized in various forms other than the head-mounted display device. For example, the present invention can be realized by a method for controlling the head-mounted display device, a computer program for realizing the function of each component included in the head-mounted display device, a recording medium on which the computer program is recorded, and the like.

It is explanatory drawing which shows schematic structure of the information processing system in 1st Embodiment of this invention. It is a principal part top view which shows the structure of the optical system with which an image display part is provided. It is a figure which shows the principal part structure of the image display part seen from the user. It is a figure for demonstrating the angle of view of a camera. It is a block diagram which shows the structure of HMD functionally. It is a block diagram which shows the structure of a control apparatus functionally. It is explanatory drawing which shows an example of the augmented reality display by HMD. It is a block diagram which shows the structure of a store server functionally. It is a flowchart which shows a shop input process routine. It is explanatory drawing which shows the data set transmitted to HMD from a store server. It is explanatory drawing which shows an example of a user's visual field after the process of step S160 was performed. It is explanatory drawing which shows an example of operation which rotates a cubic switch. It is explanatory drawing which shows the cubic switch after rotation. It is explanatory drawing which shows the see-through display area and interface area which are produced | generated by the work assistance process part of 2nd Embodiment. It is explanatory drawing which shows the modification of a switch. It is explanatory drawing which shows the modification of a switch. It is explanatory drawing which shows the modification of a cubic switch. It is a principal part top view which shows the structure of the optical system with which the image display part of a modification is provided.

A. First embodiment:
A-1. Overall configuration of the information processing system:
FIG. 1 is an explanatory diagram showing a schematic configuration of an information processing system according to the first embodiment of the present invention. The information processing system 1 includes a head-mounted display device 100 and a store server 300. The head-mounted display device 100 is connected to the Internet INT by wireless communication via the communication carrier BS. The store server 300 is connected to the Internet INT by wired communication. As a result, the head-mounted display device 100 and the store server 300 are connected to each other via the Internet INT. The communication carrier BS includes a transmission / reception antenna, a radio base station, and an exchange station.

  The head-mounted display device 100 searches a store database, which will be described later, stored in the store server 300. Hereinafter, the head-mounted display device 100 and the store server 300 will be described in detail.

A-2. Configuration of head mounted display device:
The head-mounted display device 100 is a display device that is mounted on a user's head, and is also referred to as a head mounted display (HMD). The HMD 100 is a see-through type (transmission type) head-mounted display device in which an image floats in an external environment visually recognized through a glass.

  The HMD 100 includes an image display unit 20 that allows a user to visually recognize an image, and a control device (controller) 10 that controls the image display unit 20.

  The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 in a main body having a right holding unit 21, a left holding unit 23, and a front frame 27. With.

  Each of the right holding unit 21 and the left holding unit 23 extends rearward from both end portions of the front frame 27 and holds the image display unit 20 on the user's head like a temple of glasses. Here, of both ends of the front frame 27, an end located on the right side of the user in the mounted state of the image display unit 20 is defined as an end ER, and an end located on the left side of the user is defined as an end EL. To do. The right holding unit 21 extends from the end ER of the front frame 27 to a position corresponding to the right side of the user when the image display unit 20 is worn. The left holding part 23 is provided to extend from the end EL of the front frame 27 to a position corresponding to the left side of the user when the image display part 20 is worn.

  The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is positioned in front of the right eye of the user when the image display unit 20 is mounted, and causes the right eye to visually recognize the image. The left light guide plate 28 is positioned in front of the left eye of the user when the image display unit 20 is mounted, and causes the left eye to visually recognize the image.

  The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other. This connection position corresponds to the position between the eyebrows of the user when the image display unit 20 is mounted. The front frame 27 may be provided with a nose pad portion that comes into contact with the user's nose when the image display unit 20 is mounted at a connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the user's head by the nose pad, the right holding unit 21 and the left holding unit 23. Further, a belt that is in contact with the back of the user's head when the image display unit 20 is mounted may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 can be firmly held on the user's head by the belt.

  The right display unit 22 displays an image by the right light guide plate 26. The right display unit 22 is provided in the right holding unit 21 and is located in the vicinity of the right side of the user when the image display unit 20 is worn. The left display unit 24 displays an image by the left light guide plate 28. The left display unit 24 is provided in the left holding unit 23 and is located in the vicinity of the user's left head when the image display unit 20 is worn. The right display unit 22 and the left display unit 24 are also collectively referred to as “display driving unit”.

  The right light guide plate 26 and the left light guide plate 28 of the present embodiment are optical units (for example, prisms) formed of a light transmissive resin or the like, and use image light output from the right display unit 22 and the left display unit 24. Lead to the eyes of the person. A dimming plate may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The light control plate is a thin plate-like optical element having different transmittance depending on the wavelength region of light, and functions as a so-called wavelength filter. For example, the light control plate is disposed so as to cover the surface of the front frame 27 (the surface opposite to the surface facing the user's eyes). By appropriately selecting the optical characteristics of the light control plate, the transmittance of light in an arbitrary wavelength region such as visible light, infrared light, and ultraviolet light can be adjusted. The amount of external light incident on the light plate 28 and transmitted through the right light guide plate 26 and the left light guide plate 28 can be adjusted.

  The image display unit 20 guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, respectively, and an image (augmented reality (AR) image) is generated by the image light. The user visually recognizes (this is also referred to as “display an image”). When external light enters the user's eyes through the right light guide plate 26 and the left light guide plate 28 from the front of the user, image light constituting the image and external light are incident on the user's eyes. To do. For this reason, the visibility of the image for the user is affected by the intensity of external light.

  For this reason, for example, by attaching a light control plate to the front frame 27 and appropriately selecting or adjusting the optical characteristics of the light control plate, it is possible to adjust the ease of visual recognition of the image. In a typical example, it is possible to select a dimming plate having a light transmittance that allows a user wearing the HMD 100 to visually recognize at least the outside scenery. Use of the light control plate can be expected to protect the right light guide plate 26 and the left light guide plate 28, and to suppress damage to the right light guide plate 26 and the left light guide plate 28, adhesion of dirt and the like. The light control plate may be detachable from the front frame 27 or each of the right light guide plate 26 and the left light guide plate 28. In addition, a plurality of types of light control plates may be exchanged and removable, or the light control plates may be omitted.

  The camera 61 is disposed on the front frame 27 of the image display unit 20. The camera 61 is provided on the front surface of the front frame 27 at a position that does not block outside light that passes through the right light guide plate 26 and the left light guide plate 28. In the example of FIG. 1, the camera 61 is disposed on the end ER side of the front frame 27. The camera 61 may be disposed on the end EL side of the front frame 27 or may be disposed at a connection portion between the right light guide plate 26 and the left light guide plate 28.

  The camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like. The camera 61 of this embodiment is a monocular camera, but a stereo camera may be adopted. The camera 61 captures at least a part of the outside scene (real space) in the front side direction of the HMD 100, in other words, in the visual field direction visually recognized by the user when the image display unit 20 is mounted. In other words, the camera 61 images a range or direction that overlaps the user's field of view, and images a direction that the user visually recognizes. The angle of view of the camera 61 can be set as appropriate. In the present embodiment, the width of the angle of view of the camera 61 is set so that the entire field of view of the user that the user can see through the right light guide plate 26 and the left light guide plate 28 is imaged. The camera 61 executes imaging in accordance with the control of the control function unit 150 (FIG. 5), and outputs the obtained imaging data to the control function unit 150.

  The HMD 100 may include a distance measuring sensor that detects a distance to a measurement object positioned in a preset measurement direction. The distance measuring sensor can be disposed, for example, at a connection portion between the right light guide plate 26 and the left light guide plate 28 of the front frame 27. The measurement direction of the distance measuring sensor can be the front side direction of the MD 100 (the direction overlapping the imaging direction of the camera 61). The distance measuring sensor can be composed of, for example, a light emitting unit such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured by the light source. In this case, the distance is obtained by triangular distance measurement processing or distance measurement processing based on a time difference. The distance measuring sensor may be configured by, for example, a transmission unit that emits ultrasonic waves and a reception unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance is obtained by distance measurement processing based on the time difference. The distance measuring sensor is controlled by the control function unit 150 and outputs the detection result to the control function unit 150 in the same manner as the camera 61.

  FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. For convenience of explanation, FIG. 2 shows a user's right eye RE and left eye LE. As shown in FIG. 2, the right display unit 22 and the left display unit 24 are configured symmetrically.

  As a configuration for allowing the right eye RE to visually recognize an image (AR image), the right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 and a right optical system 251. The OLED unit 221 emits image light. The right optical system 251 includes a lens group and the like, and guides the image light L emitted from the OLED unit 221 to the right light guide plate 26.

  The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-luminous display panel configured by light emitting elements that emit light by organic electroluminescence and emit color lights of R (red), G (green), and B (blue). In the OLED panel 223, a plurality of pixels each having a unit including one R, G, and B element as one pixel are arranged in a matrix.

  The OLED drive circuit 225 selects and energizes the light emitting elements included in the OLED panel 223 under the control of the control function unit 150 (FIG. 5), and causes the light emitting elements to emit light. The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back side of the light emitting surface by bonding or the like. The OLED drive circuit 225 may be configured by a semiconductor device that drives the OLED panel 223, for example, and may be mounted on a substrate that is fixed to the back surface of the OLED panel 223. A temperature sensor 217 (FIG. 5) described later is mounted on this board. Note that the OLED panel 223 may employ a configuration in which light emitting elements that emit white light are arranged in a matrix, and color filters corresponding to the colors R, G, and B are stacked. Further, an OLED panel 223 having a WRGB configuration including a light emitting element that emits W (white) light in addition to the light emitting elements that respectively emit R, G, and B color light may be employed.

  The right optical system 251 includes a collimator lens that converts the image light L emitted from the OLED panel 223 into a parallel light flux. The image light L converted into a parallel light beam by the collimator lens enters the right light guide plate 26. A plurality of reflecting surfaces that reflect the image light L are formed in an optical path that guides light inside the right light guide plate 26. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 (reflection surface) located in front of the right eye RE is formed on the right light guide plate 26. The image light L is reflected by the half mirror 261 and then emitted from the right light guide plate 26 to the right eye RE. The image light L forms an image with the retina of the right eye RE, thereby allowing the user to visually recognize the image.

  As a configuration for causing the left eye LE to visually recognize an image (AR image), the left display unit 24 includes an OLED unit 241 and a left optical system 252. The OLED unit 241 emits image light. The left optical system 252 includes a lens group and the like, and guides the image light L emitted from the OLED unit 241 to the left light guide plate 28. The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. Details of each part are the same as those of the OLED unit 221, the OLED panel 223, and the OLED drive circuit 225. A temperature sensor 239 is mounted on the substrate fixed to the back surface of the OLED panel 243. The details of the left optical system 252 are the same as those of the right optical system 251.

  According to the configuration described above, the HMD 100 can function as a see-through display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 enter the right eye RE of the user. The image light L reflected by the half mirror 281 and the external light OL transmitted through the left light guide plate 28 enter the left eye LE of the user. As described above, the HMD 100 causes the image light L of the image processed inside and the external light OL to overlap and enter the user's eyes. As a result, for the user, the outside scene (real world) can be seen through the right light guide plate 26 and the left light guide plate 28, and the image (AR image) by the image light L is visually recognized so as to overlap the outside scene.

  The half mirror 261 and the half mirror 281 function as an “image extraction unit” that reflects the image light output from the right display unit 22 and the left display unit 24 and extracts an image. Further, the right optical system 251 and the right light guide plate 26 are collectively referred to as a “right light guide”, and the left optical system 252 and the left light guide plate 28 are also collectively referred to as a “left light guide”. The configurations of the right light guide unit and the left light guide unit are not limited to the above-described examples, and any method can be used as long as an image is formed in front of the user's eyes using image light. For example, a diffraction grating or a transflective film may be used for the right light guide and the left light guide.

  In FIG. 1, the control device 10 and the image display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to a connector provided in the lower part of the control device 10, and is connected to various circuits inside the image display unit 20 from the tip of the left holding unit 23. The connection cable 40 includes a metal cable or an optical fiber cable that transmits digital data. The connection cable 40 may further include a metal cable that transmits analog data. A connector 46 is provided in the middle of the connection cable 40.

  The connector 46 is a jack for connecting a stereo mini-plug, and the connector 46 and the control device 10 are connected by a line for transmitting an analog audio signal, for example. In the example of the present embodiment shown in FIG. 1, a right earphone 32 and a left earphone 34 constituting stereo headphones and a headset 30 having a microphone 63 are connected to the connector 46.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collection unit of the microphone 63 faces the line of sight of the user. The microphone 63 collects sound and outputs a sound signal to the sound interface 182 (FIG. 5). The microphone 63 may be a monaural microphone or a stereo microphone, and may be a directional microphone or an omnidirectional microphone.

  The control device 10 is a device for controlling the HMD 100. The control device 10 includes a lighting unit 12, a touch pad 14, a direction key 16, a determination key 17, and a power switch 18. The lighting unit 12 notifies the operation state of the HMD 100 (for example, ON / OFF of the power supply) by its light emission mode. For example, an LED (Light Emitting Diode) can be used as the lighting unit 12.

  The touch pad 14 detects a contact operation on the operation surface of the touch pad 14 and outputs a signal corresponding to the detected content. As the touch pad 14, various touch pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The determination key 17 detects a pressing operation and outputs a signal for determining the content operated in the control device 10. The power switch 18 switches the power state of the HMD 100 by detecting a slide operation of the switch.

  FIG. 3 is a diagram illustrating a main configuration of the image display unit 20 as viewed from the user. In FIG. 3, the connection cable 40, the right earphone 32, and the left earphone 34 are not shown. In the state of FIG. 3, the back sides of the right light guide plate 26 and the left light guide plate 28 can be visually recognized, the half mirror 261 for irradiating the right eye RE with image light, and the image light for irradiating the left eye LE with image light. The half mirror 281 can be visually recognized as a substantially rectangular area. The user views the outside scene through the entire left and right light guide plates 26 and 28 including the half mirrors 261 and 281 and visually recognizes a rectangular display image at the position of the half mirrors 261 and 281.

  FIG. 4 is a diagram for explaining the angle of view of the camera 61. In FIG. 4, the camera 61 and the user's right eye RE and left eye LE are schematically shown in plan view, and the angle of view (imaging range) of the camera 61 is indicated by θ. Note that the angle of view θ of the camera 61 extends in the horizontal direction as shown in the figure, and also in the vertical direction as in a general digital camera.

  As described above, the camera 61 is arranged at the right end of the image display unit 20 and images the direction of the user's line of sight (that is, the front of the user). For this reason, the optical axis of the camera 61 is a direction including the line-of-sight directions of the right eye RE and the left eye LE. The outside scene that the user can visually recognize in a state where the HMD 100 is worn is not always at infinity. For example, when the user gazes at the object OB with both eyes, the user's line of sight is directed toward the object OB as indicated by reference numerals RD and LD in the figure. In this case, the distance from the user to the object OB is often about 30 cm to 10 m, and more often 1 m to 4 m. Therefore, for the HMD 100, an upper limit and a lower limit of the distance from the user to the object OB during normal use may be determined. This standard may be obtained in advance and preset in the HMD 100, or may be set by the user. The optical axis and the angle of view of the camera 61 are set so that the object OB is included in the angle of view when the distance to the object OB during normal use corresponds to the set upper and lower limits. It is preferred that

  In general, a human viewing angle is approximately 200 degrees in the horizontal direction and approximately 125 degrees in the vertical direction. Among them, the effective visual field with excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. A stable gaze field in which a gaze point that a person gazes at appears quickly and stably is set to 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. In this case, when the gazing point is the object OB (FIG. 4), the effective visual field is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction around the lines of sight RD and LD. Further, the stable focus is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. The actual field of view that the user permeates through the image display unit 20 and permeates through the right light guide plate 26 and the left light guide plate 28 is referred to as a real field of view (FOV). The real field of view is narrower than the viewing angle and stable focus field, but wider than the effective field of view.

  The angle of view θ of the camera 61 of the present embodiment is set so that a wider range than the user's visual field can be imaged. The angle of view θ of the camera 61 is preferably set so that at least a range wider than the effective visual field of the user can be captured, and more preferably set so that a range wider than the actual visual field can be captured. It is more preferable that the angle of view θ of the camera 61 is set so as to be able to image a wider range than the stable viewing field of the user, and is set so as to be able to image a range wider than the viewing angle of both eyes of the user. Most preferred. For this reason, the camera 61 may include a so-called wide-angle lens as an imaging lens so that a wide angle of view can be captured. The wide-angle lens may include a lens called an ultra-wide-angle lens or a quasi-wide-angle lens. The camera 61 may include a single focus lens, a zoom lens, or a lens group including a plurality of lenses.

  FIG. 5 is a block diagram functionally showing the configuration of the HMD 100. The control device 10 includes a main processor 140 that controls the HMD 100 by executing a program, a storage unit, an input / output unit, sensors, an interface, and a power supply unit 130. The storage unit, input / output unit, sensors, interface, and power supply unit 130 are connected to the main processor 140. The main processor 140 is mounted on the controller board 120 built in the control device 10.

  The storage unit includes a memory 118 and a nonvolatile storage unit 121. The memory 118 constitutes a work area for temporarily storing a computer program executed by the main processor 140 and data to be processed. The nonvolatile storage unit 121 is configured by a flash memory or an eMMC (embedded Multi Media Card). The nonvolatile storage unit 121 stores a computer program executed by the main processor 140 and various data processed by the main processor 140. In the present embodiment, these storage units are mounted on the controller board 120.

  The input / output unit includes a touch pad 14 and an operation unit 110. The operation unit 110 includes a direction key 16, a determination key 17, and a power switch 18 provided in the control device 10. The main processor 140 controls these input / output units and acquires signals output from the input / output units.

  The sensors include a six-axis sensor 111, a magnetic sensor 113, and a GPS (Global Positioning System) receiver 115. The 6-axis sensor 111 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 111 may employ an IMU (Inertial Measurement Unit) in which these sensors are modularized. The magnetic sensor 113 is, for example, a triaxial geomagnetic sensor. The GPS receiver 115 includes a GPS antenna (not shown), receives a radio signal transmitted from a GPS satellite, and detects coordinates of the current position of the control device 10. These sensors (six-axis sensor 111, magnetic sensor 113, GPS receiver 115) output detection values to the main processor 140 according to a sampling frequency designated in advance. The timing at which each sensor outputs a detection value may be in accordance with an instruction from the main processor 140.

  The interface includes a wireless communication unit 117, an audio codec 180, an external connector 184, an external memory interface 186, a USB (Universal Serial Bus) connector 188, a sensor hub 192, an FPGA 194, and an interface 196. ing. These function as an interface with the outside. The wireless communication unit 117 performs wireless communication between the HMD 100 and an external device. The wireless communication unit 117 includes an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like (not shown), or is configured as a device in which these are integrated. The wireless communication unit 117 performs wireless communication complying with a standard such as a wireless LAN including Bluetooth (registered trademark) and Wi-Fi (registered trademark), for example.

  The audio codec 180 is connected to the audio interface 182 and encodes / decodes an audio signal input / output via the audio interface 182. The audio interface 182 is an interface for inputting and outputting audio signals. The audio codec 180 may include an A / D converter that converts analog audio signals to digital audio data, and a D / A converter that performs the reverse conversion. The HMD 100 according to the present embodiment outputs sound from the right earphone 32 and the left earphone 34 and collects sound by the microphone 63. The audio codec 180 converts the digital audio data output from the main processor 140 into an analog audio signal, and outputs the analog audio signal via the audio interface 182. The audio codec 180 converts an analog audio signal input to the audio interface 182 into digital audio data and outputs the digital audio data to the main processor 140.

  The external connector 184 is a connector for connecting an external device (for example, a personal computer, a smart phone, a game device, etc.) that communicates with the main processor 140 to the main processor 140. The external device connected to the external connector 184 can be a content supply source, and can be used for debugging a computer program executed by the main processor 140 and collecting an operation log of the HMD 100. The external connector 184 can employ various modes. As the external connector 184, for example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, a memory card interface, or an interface corresponding to a wireless connection such as a wireless LAN interface or a Bluetooth interface can be adopted.

  The external memory interface 186 is an interface to which a portable memory device can be connected. The external memory interface 186 includes, for example, a memory card slot in which a card-type recording medium is mounted and data is read and written, and an interface circuit. The size, shape, standard, etc. of the card type recording medium can be selected as appropriate. The USB connector 188 is an interface that can connect a memory device, a smartphone, a personal computer, or the like that conforms to the USB standard. The USB connector 188 includes, for example, a connector conforming to the USB standard and an interface circuit. The size, shape, USB standard version, etc. of the USB connector 188 can be selected as appropriate.

  Further, the HMD 100 includes a vibrator 19. The vibrator 19 includes a motor (not shown), an eccentric rotor, and the like, and utters vibrations under the control of the main processor 140. The HMD 100 causes the vibrator 19 to generate a vibration with a predetermined vibration pattern when, for example, an operation on the operation unit 110 is detected or when the power of the HMD 100 is turned on / off.

  The sensor hub 192 and the FPGA 194 are connected to the image display unit 20 via an interface (I / F) 196. The sensor hub 192 acquires detection values of various sensors included in the image display unit 20 and outputs them to the main processor 140. The FPGA 194 executes processing of data transmitted and received between the main processor 140 and each unit of the image display unit 20 and transmission via the interface 196. The interface 196 is connected to each of the right display unit 22 and the left display unit 24 of the image display unit 20. In the example of the present embodiment, the connection cable 40 is connected to the left holding unit 23, the wiring connected to the connection cable 40 is laid inside the image display unit 20, and each of the right display unit 22 and the left display unit 24 is It is connected to the interface 196 of the control device 10.

  The power supply unit 130 includes a battery 132 and a power supply control circuit 134. The power supply unit 130 supplies power for operating the control device 10. The battery 132 is a rechargeable battery. The power control circuit 134 detects the remaining capacity of the battery 132 and controls the charging of the OS 143. The power supply control circuit 134 is connected to the main processor 140 and outputs a detected value of the remaining capacity of the battery 132 and a detected value of the voltage of the battery 132 to the main processor 140. Note that power may be supplied from the control device 10 to the image display unit 20 based on the power supplied by the power supply unit 130. The power supply state from the power supply unit 130 to each unit of the control device 10 and the image display unit 20 may be configured to be controllable by the main processor 140.

  The right display unit 22 includes a display unit substrate 210, an OLED unit 221, a camera 61, an illuminance sensor 65, an LED indicator 67, and a temperature sensor 217. An interface (I / F) 211 connected to the interface 196, a receiving unit (Rx) 213, and an EEPROM (Electrically Erasable Programmable Read-Only Memory) 215 are mounted on the display unit substrate 210. The receiving unit 213 receives data input from the control device 10 via the interface 211. When receiving image data of an image to be displayed on the OLED unit 221, the receiving unit 213 outputs the received image data to the OLED drive circuit 225 (FIG. 2).

  The EEPROM 215 stores various data in a form that can be read by the main processor 140. The EEPROM 215 stores, for example, data on the light emission characteristics and display characteristics of the OLED units 221 and 241 of the image display unit 20, data on the sensor characteristics of the right display unit 22 or the left display unit 24, and the like. Specifically, for example, parameters relating to gamma correction of the OLED units 221 and 241 and data for compensating detection values of temperature sensors 217 and 239 described later are stored. These data are generated by the factory inspection of the HMD 100 and are written in the EEPROM 215. After shipment, the main processor 140 reads the data in the EEPROM 215 and uses it for various processes.

  The camera 61 executes imaging in accordance with a signal input via the interface 211 and outputs captured image data or a signal representing the imaging result to the control device 10. As shown in FIG. 1, the illuminance sensor 65 is provided at an end ER of the front frame 27 and is arranged to receive external light from the front of the user wearing the image display unit 20. The illuminance sensor 65 outputs a detection value corresponding to the amount of received light (received light intensity). As shown in FIG. 1, the LED indicator 67 is disposed near the camera 61 at the end ER of the front frame 27. The LED indicator 67 is lit during execution of imaging by the camera 61 to notify that imaging is in progress.

  The temperature sensor 217 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor 217 is mounted on the back side of the OLED panel 223 (FIG. 3). For example, the temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223. The temperature sensor 217 may be incorporated in the OLED panel 223 or the OLED drive circuit 225. For example, when the OLED panel 223 is mounted as an Si-OLED as an integrated circuit on an integrated semiconductor chip together with the OLED drive circuit 225, the temperature sensor 217 may be mounted on the semiconductor chip.

  The left display unit 24 includes a display unit substrate 230, an OLED unit 241, and a temperature sensor 239. On the display unit substrate 230, an interface (I / F) 231 connected to the interface 196, a receiving unit (Rx) 233, a six-axis sensor 235, and a magnetic sensor 237 are mounted. The receiving unit 233 receives data input from the control device 10 via the interface 231. When receiving image data of an image to be displayed on the OLED unit 241, the receiving unit 233 outputs the received image data to the OLED drive circuit 245 (FIG. 2).

  The 6-axis sensor 235 is a motion sensor (inertia sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 235 may employ an IMU in which the above sensors are modularized. The magnetic sensor 237 is, for example, a triaxial geomagnetic sensor. The temperature sensor 239 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature. The temperature sensor 239 is mounted on the back side of the OLED panel 243 (FIG. 3). The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243. The temperature sensor 239 may be incorporated in the OLED panel 243 or the OLED drive circuit 245. Details are the same as those of the temperature sensor 217.

  The camera 61, the illuminance sensor 65, and the temperature sensor 217 of the right display unit 22, and the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 of the left display unit 24 are connected to the sensor hub 192 of the control device 10. The sensor hub 192 sets and initializes the sampling period of each sensor according to the control of the main processor 140. The sensor hub 192 executes energization to each sensor, transmission of control data, acquisition of a detection value, and the like in accordance with the sampling period of each sensor. The sensor hub 192 outputs detection values of the sensors included in the right display unit 22 and the left display unit 24 to the main processor 140 at a preset timing. The sensor hub 192 may have a cache function that temporarily holds the detection value of each sensor. The sensor hub 192 may be provided with a conversion function (for example, a conversion function to a unified format) of the signal format and data format of the detection value of each sensor. The sensor hub 192 turns on or off the LED indicator 67 by starting and stopping energization of the LED indicator 67 according to the control of the main processor 140.

  FIG. 6 is a block diagram functionally showing the configuration of the control device 10. Functionally, the control device 10 includes a storage function unit 122 and a control function unit 150. The storage function unit 122 is a logical storage unit configured by the nonvolatile storage unit 121 (FIG. 5). The storage function unit 122 may be configured to use the EEPROM 215 or the memory 118 in combination with the nonvolatile storage unit 121 instead of the configuration using only the storage function unit 122. The control function unit 150 is configured when the main processor 140 executes a computer program, that is, when hardware and software cooperate.

  The storage function unit 122 stores various data used for processing in the control function unit 150. Specifically, setting data 123 and content data 124 are stored in the storage function unit 122 of the present embodiment. The setting data 123 includes various setting values related to the operation of the HMD 100. For example, the setting data 123 includes parameters, determinants, arithmetic expressions, LUTs (Look Up Tables), and the like when the control function unit 150 controls the HMD 100.

  The content data 124 includes content data (image data, video data, audio data, etc.) including images and video displayed by the image display unit 20 under the control of the control function unit 150. The content data 124 may include interactive content data. Bi-directional content refers to a user's operation acquired by the operation unit 110, processing corresponding to the acquired operation content is executed by the control function unit 150, and content corresponding to the processing content is stored in the image display unit 20. Means the type of content to display. In this case, the content data may include image data of a menu screen for acquiring a user's operation, data defining a process corresponding to an item included in the menu screen, and the like.

  The control function unit 150 executes various processes using the data stored in the storage function unit 122, whereby the OS 143, the image processing unit 145, the display control unit 147, the imaging control unit 149, and the input / output control unit 151. The functions as the communication control unit 153 and the store input processing unit 155 are executed. In the present embodiment, each functional unit other than the OS 143 is configured as a computer program executed on the OS 143.

  The image processing unit 145 generates a signal to be transmitted to the right display unit 22 and the left display unit 24 based on image / video image data displayed by the image display unit 20. The signal generated by the image processing unit 145 may be a vertical synchronization signal, a horizontal synchronization signal, a clock signal, an analog image signal, or the like. The image processing unit 145 may be configured by hardware (for example, DSP (Digital Signal Processor)) different from the main processor 140 in addition to the configuration realized by the main processor 140 executing a computer program.

  Note that the image processing unit 145 may execute resolution conversion processing, image adjustment processing, 2D / 3D conversion processing, and the like as necessary. The resolution conversion process is a process for converting the resolution of the image data into a resolution suitable for the right display unit 22 and the left display unit 24. The image adjustment process is a process for adjusting the brightness and saturation of image data. The 2D / 3D conversion process is a process of generating 2D image data from 3D image data or generating 3D image data from 2D image data. When these processes are executed, the image processing unit 145 generates a signal for displaying an image based on the processed image data, and transmits the signal to the image display unit 20 via the connection cable 40.

  The display control unit 147 generates a control signal for controlling the right display unit 22 and the left display unit 24, and controls the generation and emission of image light by each of the right display unit 22 and the left display unit 24 based on the control signal. To do. Specifically, the display control unit 147 controls the OLED drive circuits 225 and 245 to cause the OLED panels 223 and 243 to display an image. The display control unit 147 controls the timing at which the OLED drive circuits 225 and 245 draw on the OLED panels 223 and 243 and controls the luminance of the OLED panels 223 and 243 based on the signal output from the image processing unit 145.

  The imaging control unit 149 controls the camera 61 to execute imaging, generates captured image data, and temporarily stores it in the storage function unit 122. When the camera 61 is configured as a camera unit including a circuit that generates captured image data, the imaging control unit 149 acquires captured image data from the camera 61 and temporarily stores the captured image data in the storage function unit 122.

  The input / output control unit 151 appropriately controls the touch pad 14 (FIG. 1), the direction key 16, and the determination key 17, and acquires an input command therefrom. The acquired command is output to the OS 143 or a computer program operating on the OS 143 together with the OS 143. The communication control unit 153 controls the wireless communication unit 117 to perform wireless communication with the store server 300, for example.

  The store input processing unit 155 is a function realized in accordance with an application program that runs on the OS 143. The store input processing unit 155 obtains store identification information desired by the user in cooperation with the input / output control unit 151, the image processing unit 145, and the display control unit 147. Specifically, the store input processing unit 155 performs a function of presenting a plurality of candidate stores sent from the store server 300 to the user, and an input of one store selected by the user from the plurality of candidate stores. And a function to accept. The latter function of accepting store input is realized by the polyhedron image display unit 155a, the polyhedron orientation switching unit 155b, and the instruction execution unit 155c. Details of the respective units 155a to 155c will be described later. The image processing unit 145 and the display control unit 147 correspond to the “display control unit” in one embodiment of the present invention described in the “Summary of Invention” column, and the store input processing unit 155 corresponds to the “processing unit”. .

  FIG. 7 is an explanatory diagram illustrating an example of augmented reality display by the HMD 100. FIG. 7 illustrates the user's field of view VR. As described above, the image light guided to both eyes of the user of the HMD 100 forms an image on the retina of the user, so that the user visually recognizes the image AI as augmented reality (AR). In the example of FIG. 7, the image AI is a menu screen of the OS of the HMD 100. The menu screen includes, for example, icons for starting application programs such as “message”, “phone”, “camera”, “browser”, and “store guide”. The right and left light guide plates 26 and 28 transmit light from the outside scene SC, so that the user visually recognizes the outside scene SC. Thus, the user of the HMD of the present embodiment can view the image AI so that the portion of the visual field VR where the image VI is displayed overlaps the outside scene SC. Further, only the outside scene SC can be seen in the portion of the visual field VR where the image AI is not displayed.

A-3. Store server configuration:
FIG. 8 is a block diagram functionally showing the configuration of the store server 300. As illustrated in FIG. 4, the store server 300 includes a CPU 310, a storage unit 320, a ROM 330, a RAM 340, and a communication interface (I / F) 350, and each unit is connected to each other via a bus 360. .

  The CPU 310 controls each unit of the store server 300 by developing and executing a computer program stored in the storage unit 320 or the ROM 330 on the RAM 340. In addition, the CPU 310 also functions as a peripheral store information providing unit 312 and a store guide unit 314. The neighboring store information providing unit 312 provides the HMD 100 with stores included in the range of the user's field of view VR1 as candidate stores. The store guide unit 314 receives store identification information desired by the user from the HMD 100, acquires guide information about the store desired by the user from the store database 322, and provides the acquired store information to the HMD 100. In the present embodiment, information useful for using the store, such as information indicating a route to the store and a phone number of the store, is provided as the guide information.

  The storage unit 320 includes a ROM, a RAM, a DRAM, a hard disk, and the like. The storage unit 320 stores various computer programs including an operating system (OS). The storage unit 320 stores the store database 322 and the map database 324 described above. Various stores are recorded in the map database 324.

A-4. About store input processing:
FIG. 9 is a flowchart showing a store input processing routine. This store input processing routine corresponds to the store input processing unit 155 (FIG. 6). The store input processing routine is a processing routine according to a predetermined computer program (application program) stored in the nonvolatile storage unit 121 (FIG. 5), and is executed by the main processor 140 of the HMD 100. The store input processing routine is started in response to the instruction of the “store guide” icon on the menu screen illustrated in FIG. 7 by the direction key 16 (FIG. 1) and the enter key 17 (FIG. 1).

  When the process is started, the main processor 140 of the HMD 100 first acquires position information indicating the current position of the HMD 100 from the GPS receiver 115 (step S110). Next, the main processor 140 acquires information (gaze direction information) for specifying the gaze direction of the user wearing the image display unit 20 (step S120). Specifically, the line-of-sight direction information is acquired by specifying the orientation of the image display unit 20 based on the geomagnetism detected by the magnetic sensor 113. Subsequently, the main processor 140 transmits the acquired position information and line-of-sight direction information to the store server 300 (step S130).

  When the process of step S130 is performed, the following process is performed on the store server 300 side. The store server 300 first acquires position information and line-of-sight information, and sets a virtual virtual field of view of the user from these information. Specifically, at the current position of the HMD 100, a range (for example, 90 degrees left and right, 60 degrees up and down) set in advance around the user's line-of-sight direction is set as the virtual visual field. Note that the “virtual field of view” here does not necessarily match the field of view (real field of view) that the user can actually see through the right and left light guide plates 26, 28, but a range that does not exceed the actual field of view is desirable. . Subsequently, the store server 300 searches the stores included in the virtual view by checking the map database 324. Here, the distance from the HMD 100 is set to several hundred meters, for example, and stores included in the virtual field of view are searched. Thereafter, the store server 300 sets the searched store as a candidate store, and acquires information about each candidate store from the store database 322. Thereafter, the store server 300 transmits a data set indicating information about each acquired candidate store to the HMD 100.

  FIG. 10 is an explanatory diagram showing a data set DS transmitted from the store server 300 to the HMD 100. The data set DS records information about each candidate store in the form of a record RC. In each record RC, data of “store name”, “view position”, and “detailed description” are recorded. The “store name” data is the name of the candidate store, and the “view position” data is information indicating the position of the candidate store in the virtual field of view described above. The “detailed description” data indicates detailed information on the candidate stores, and specifically includes store types (for example, convenience stores, restaurants, taverns, etc.), handled products, and the like.

  In step S140 following step S130 of FIG. 9, the main processor 140 receives the above-described data set from the store server 300. Next, the main processor 140 controls the image processing unit 145 (FIG. 6) based on the received data set, and performs display processing for displaying the see-through display area on the display control unit 147 (FIG. 6) (step S150). . Subsequently, the main processor 140 controls the image processing unit 145 (FIG. 6) based on the received data set, and performs display processing for displaying the information display area on the display control unit 147 (FIG. 6) (step S160). ). The processing in step S160 corresponds to the polyhedron image display unit 155a (FIG. 6).

  FIG. 11 is an explanatory diagram illustrating an example of the user's field of view VR1 after the process of step S160 is performed. As shown in the drawing, the visual field VR1 includes a see-through display area P1 and an interface area P2.

  In the see-through display area P1, balloons A1 to A6 of store names as AR images are displayed so as to overlap the outside scene SC. The store names represented in the balloons A1 to A6 are based on the “store name” data of the received data set DS. The display positions of the balloons A1 to A6 correspond to the positions where the candidate stores exist, and are determined based on the “view position” data of the received data set DS. Specifically, after setting the same virtual visual field as the virtual visual field set by the store server 300 in the visual field VR1, the position in the set virtual visual field is obtained from the “view position” data. The display positions of the balloons A1 to A6 are determined.

  The interface area P2 is an area for exchanging information with the operator, and is connected to one side (for example, the right side) of the see-through display area P1. Specifically, in the interface area P2, one or more (six in the illustration of FIG. 11) store names displayed in the see-through display area P1 are displayed in a list, and a user selected from the list display is displayed. Receive the store name of the store you want. In the interface area P2, the amount of transmission of the outside scene is set so that a distant outside scene does not pass through and a nearby outside scene (for example, a fingertip) can be seen through.

  The interface area P2 shown in FIG. 11 includes a list display field LT and a cubic switch SW.

  The list display field LT includes [store name] buttons B1 corresponding to the store names displayed in the see-through display area P1 arranged in the vertical direction, and a [detail] button B2 on the right side of each [store name] button. Are provided. That is, for each store, a [store name] button B1 and a [details] button B2 are provided in pairs. The [Store Name] button B1 is a switch for instructing the store name to the control function unit 150 (FIG. 6) of the HMD 10. [Details] button B2 is a switch for instructing display of detailed information of the store. One of all the buttons B1 and B2 in the list display column LT is in an active state, and the remaining buttons B1 and B2 are in an inactive state. The “active state” is a state that is a target of operation by the user. In the illustrated example, the [store name] button B1 of “XX store” has a white background and is in an active state, and the remaining buttons B1 and B2 are in an inactive state. In the illustrated example, the [Detail] button B2 of “XX store” and the [Store name] button B1 and the [Detail] button B2 of each store other than “XX store” are in an inactive state. The list display column LT corresponds to the “image” in one embodiment of the present invention described in the “Summary of Invention” column.

  The cubic switch SW functions as a GUI (graphical user interface) for sequentially switching the buttons B1 and B2 in the active state and determining a store to be input. In the present embodiment, the cubic switch SW is constituted by a cubic (regular hexahedron) -shaped 3D image. A “3D image” is a three-dimensional (stereoscopic) image. In this embodiment, the cubic image is indicated by three-dimensional shape data having a depth, and the angle (angle) can be freely changed or a perspective can be given on the two-dimensional screen. . One instruction is assigned to each surface SF of the cube.

  In the present embodiment, the instructions assigned to the six surfaces SF are “up”, “down”, “right”, “left”, “decision”, and “deletion”. “Up” is an instruction to switch the buttons B1 and B2 in the active state upward. “Down” is an instruction to switch the buttons B1 and B2 in the active state downward. “Right” is a command to switch the buttons B1 and B2 in the active state to the right side. “Left” is an instruction to switch the buttons B1 and B2 in the active state to the left side.

  “Determine” is a command for executing the function set for the buttons B1 and B2 in the active state. That is, when the button in the active state is the [Store Name] button B1, the input of the store name is accepted, and when the button in the active state is the [Detail] button B2, details about the corresponding store are received. Display information. “Delete” is an instruction to delete stores corresponding to the buttons B1 and B2 in the active state from the candidate stores. Since the cubic switch SW is a regular hexahedron 3D image, a part (for example, three) of the six faces can be seen and the other faces are hidden behind.

  As described above, the interface area P2 allows the user to operate the cubic switch SW with the fingertip by inserting the fingertip of the hand into the field of view because the nearby outside scene is transparent and visible. be able to.

  FIG. 12 is an explanatory diagram illustrating an example of an operation of rotating the cubic switch SW. The user of the HMD 100 can rotate the cubic switch SW by placing (fitting) a fingertip FG on one surface SF of the cubic switch SW as shown in the figure and flicking (sliding) it as indicated by an arrow FL. . That is, the display direction of the cube as the cubic switch SW can be switched.

  FIG. 13 is an explanatory diagram showing the cubic switch SW after rotation. As a result of the operation as shown in FIG. 12, the cubic switch SW is switched to the state shown in FIG. 13, that is, the direction in which the surface labeled “up” is displayed. The state of FIG. 13 is a state in which the surface SF is switched by one in the flick direction from the state of FIG. 12, but the cubic switch SW increases the distance of the flick to increase the surface SF by two. It is also possible to switch between three.

  The front surface SF of the cubic switch SW can be tapped by the user. The “front surface” is a surface facing the user, and corresponds to the surface SF marked “up” in the example of FIG. The “tap” is an operation of moving the fingertip FG forward (front side as viewed from the user) so that the fingertip FG of the hand is applied (overlapped) to the image element and pressed. By tapping the front surface SF of the cubic switch SW, it is possible to cause the HMD 100 to execute a command assigned to the front surface SF.

  For example, in the display state of the interface area P2 shown in FIG. 11, when the front surface SF of the cubic switch SW, that is, the surface SF marked “down” is tapped, the button in the active state is “ [Store name] button B1 marked with “OO store” is switched to [Store name] button B1 marked with “ΔΔ store”.

  For example, in the display state of the interface area P2 shown in FIG. 11, the surface SF marked “OK” is switched to the front side by flicking the cubic switch SW downward, and then the front surface SF is tapped. In such a case, “determine” is executed for the [store name] button B1 labeled “XX store”, which is the button in the active state. Specifically, the control function unit 150 (FIG. 6) of the HMD 10 receives an input of the store name “XX store”.

  For example, in the display state of the interface area P2 shown in FIG. 11, the surface SF labeled “right” is switched to the front side by flicking the cubic switch SW to the right, and then the front surface SF is tapped. In this case, the button in the active state is switched from the [store name] button B1 marked “XX store” to the “detail” button B2 on the right side. After that, by flicking the cubic switch SW downward, the surface SF marked “OK” is switched to the front side, and when the front surface SF is tapped, the button “○○ which is in the active state” “Determine” is executed for the “Detail” button B2 of “Store”. As a result, detailed information about the corresponding store “XX store” is displayed in the interface area P2. The detailed information is based on the “detailed description” data (see FIG. 10) of the data set received from the customer server in step S140.

  Returning to the flowchart of FIG. The above-described operation using the cubic switch SW is realized by the processing from steps S170 to S196.

  In step S170, the main processor 140 controls the camera 61 to execute imaging, and detects the movement of the user's fingertip FG from the obtained captured image. Next, the main processor 140 determines from the detected movement of the fingertip FG whether or not the above-described flick or tap operation has been performed on the cubic switch SW (step S180). If the operation has not been performed, step S170 is performed. The process is returned to and the operation is waited for. If it is determined in step S180 that an operation has been performed on the cubic switch SW, it is determined whether the operation is a flick or a tap (step S190).

  If it is determined in step S190 that the flick has occurred, the main processor 140 switches the direction in which the cubic switch SW is displayed based on the flick direction (step S192). That is, the cubic switch SW is changed to an image rotated in the flicked direction. The processes in steps S170 to S192 correspond to the polyhedral orientation switching unit 155b (FIG. 6). After execution of step S192, the process returns to step S170 to wait for the operator's next operation.

  On the other hand, if it is determined in step S190 that it is a tap, the main processor 140 executes the instruction assigned to the tapped surface (in this embodiment, the front surface, that is, the surface facing the user) SF. (Step S194). That is, in the case of “up”, the buttons B1 and B2 in the active state are switched to the upper side. If it is “down”, the buttons B1 and B2 in the active state are switched to the lower side. In the case of “right”, the buttons B1 and B2 in the active state are switched to the right side. In the case of “left”, the buttons B1 and B2 in the active state are switched to the left side. In the case of “delete”, the stores corresponding to the buttons B1 and B2 in the active state are deleted from the candidate stores.

  If it is determined to be “determined”, it is divided into two operations. When the buttons B1 and B2 in the active state are the [store name] button B1, the identification information of the corresponding store, that is, the store name is input. Specifically, in the store input processing unit 155 of the control function unit 150 (FIG. 6) of the HMD 10, the buttons B1 and B2 in the active state in which the input of the store name “XX store” is accepted are the [Detail] button B2. If it is, detailed information about the corresponding store is displayed. The detailed information is based on the “detailed description” data (see FIG. 10) of the data set received from the customer server in step S140. In the present embodiment, detailed information is displayed for a predetermined time (for example, 5 seconds) in the interface area P2, and after 5 seconds, the interface area P2 returns to the original display. The processes in steps S170 to S190 and S194 correspond to the instruction execution unit 155c (FIG. 6).

  After executing step S194, the main processor 140 determines whether or not the buttons B1 and B2 in the active state are the [store name] button B1 and the instruction assigned to the tapped surface SF is “determine”. Is determined (step S196). Here, in the case of negative determination, that is, when the buttons B1 and B2 in the active state are [details] buttons B2, or when the command assigned to the tapped surface SF is a command other than “decision”, The process returns to step S170 to wait for the operator's next operation.

  On the other hand, if the determination in step S196 is affirmative, that is, if the [store name] button B1 is “determined”, the process proceeds to “end” and the store input processing routine is terminated. That is, when an affirmative determination is made, since the reception of the store name has been completed by the processing of step S194, this store input processing routine is terminated.

  The main processor 140 executes another process after the end of the store input process routine, thereby sending the store name accepted by the store input process routine to the store server 300 and displaying the guidance information sent from the store server 300 I do.

A-5. About effect of embodiment:
According to the HMD 100 of the first embodiment configured as described above, the user designates one store among a plurality of stores included in the field of view that can be visually recognized through the right and left light guide plates 26 and 28. By simply flicking and rotating the cubic switch SW and tapping the front surface SF, the designation can be performed. Therefore, according to the HMD 100, the operability for the user can be improved. In particular, according to the HMD 100, since the cubic switch SW is a regular hexahedron 3D image, processing can be performed by a simple and intuitive operation. Therefore, operability can be further improved.

B. Second embodiment:
The HMD according to the second embodiment of the present invention differs from the HMD 100 according to the first example in some functions realized by the control function unit 150 (FIG. 6). Specifically, the control function unit included in the HMD 100 is different from the store input processing unit 155 (FIG. 6) in the first embodiment in that a work support processing unit (not shown) is provided. The remaining configuration of the HMD in the second embodiment is the same as that of the first embodiment, and a description thereof will be omitted.

  FIG. 14 is an explanatory diagram illustrating a see-through display area P11 and an interface area P12 generated by the work support processing unit of the second embodiment. In the see-through display area P11, as in the see-through display area P1 (FIG. 11) of the first embodiment, a balloon A11 as an AR image is displayed so as to overlap the outside scene SC visually recognized through the see-through. The outside scene SC11 includes screws and plate materials for screwing. The balloon A11 includes an AR navigation screen for performing work instructions.

  In the interface area P12, a work standard document LT11 indicating a procedure for work is displayed. The worker reads the work standard document LT11 displayed in the interface area P12 and performs the screwing work according to the instruction on the AR navigation screen of the balloon A11. The work standard LT11 corresponds to the “image” in one embodiment of the present invention described in the “Summary of Invention” column.

  The interface area P12 includes the cubic switch SW11 as in the first embodiment. The cubic switch SW11 functions as a GUI for switching the display of the page of the work standard document LT11. On each surface SF of the cube (regular hexahedron) constituting the cubic switch SW11, “Page up”, “Page down”, “Work standard document enlargement”, “Work standard document reduction”, “AR navigation screen enlargement”, “AR navigation screen enlargement”, “ Each command of “AR navigation screen reduction” is assigned.

  “Page up” is an instruction to turn (send) one page of the work standard LT11. “Page dpwn” is an instruction to return one page of the work standard document LT11. “Enlarge work standard” is a command for enlarging and displaying the work standard LT11. “Work standard reduction” is a command for reducing and displaying the work standard LT11. “AR navigation screen enlargement” is a command for enlarging the AR navigation screen displayed in the balloon A11. “AR navigation screen reduction” is a command for reducing the AR navigation screen displayed in the balloon A11.

  Similarly to the cubic switch SW of the first embodiment, the cubic switch SW11 can rotate the cubic switch SW11 by placing (fitting) a fingertip FG on one surface of the cubic switch SW11 and flicking (sliding) it. . Further, by tapping the front surface of the cubic switch SW, it is possible to cause the HMD 100 to execute an instruction assigned to the front surface.

  According to the HMD of the second embodiment configured as described above, the page switch / page return / enlargement / reduction of the work standard document LT11 displayed in the interface area P12 is rotated by flicking the cubic switch SW11, Just tap the front side. Therefore, according to this HMD, user operability can be improved. In particular, according to the HMD, since the cubic switch SW11 is a regular hexahedral 3D image, the processing can be performed by a simple and intuitive operation. Therefore, operability can be further improved.

C. Variations:
The present invention is not limited to the first and second embodiments and their modifications, and can be carried out in various modes without departing from the gist thereof. For example, the following modifications are possible. Is also possible.

・ Modification 1:
In each embodiment and modification, the cubic switches SW and SW11 in which commands are assigned to each surface are regular hexahedral 3D images. On the other hand, as shown in FIG. 15, a switch SW21 in which instructions are assigned to each surface may be a regular tetrahedral 3D image, or a switch in which instructions are assigned to each surface as shown in FIG. SW31 may be a regular dodecahedron 3D image. That is, a 3D image of a polyhedron can be a 3D image having various numbers of equal-sized surfaces. Further, the sizes of the surfaces are not necessarily equal, and the surfaces can be different sizes. Further, a polyhedron may be represented by a plurality of 2D images instead of the 3D images.

Modification 2
In each embodiment and modification, it is configured such that an instruction assigned to the front surface can be executed by the HMD 100 by tapping the surface (front surface) facing the user of the cubic switches SW and SW11. On the other hand, as a modification, a configuration may be adopted in which an instruction assigned to the upper surface can be executed by the HMD 100 by tapping the upper surface of the cubic switches SW and SW11. The upper surface is a surface described as “determined” in the example of FIG. 11. In short, any surface having a predetermined orientation can be replaced with any surface. Furthermore, it is good also as a structure which can tap a some surface (for example, front surface and uppermost surface) instead of the structure which can tap only any one surface.

・ Modification 3:
In each embodiment and modification, it is configured such that an instruction assigned to the front surface can be executed by the HMD 100 by tapping the surface (front surface) facing the user of the cubic switches SW and SW11. On the other hand, as a modification, when switching the direction of the cubic switches SW and SW11, the HMD 100 may be configured to immediately execute the instruction assigned to the front surface without tapping. According to this configuration, operability can be further improved. As in the second modification, instead of the front side surface, a command assigned to a surface in another predetermined direction such as the upper side surface may be immediately executed.

-Modification 4:
In each embodiment and modification, one command is assigned to each face of the regular hexahedron constituting the cubic switches SW and SW11. On the other hand, as shown in FIG. 17, the cubic switch SW41 may be configured to assign a plurality of (for example, four) instructions to a predetermined face of a regular hexahedron. In the illustrated example, four instructions are assigned to only one face of the regular hexahedron. However, instead of this, a plurality of instructions may be assigned to all faces.

-Modification 5:
In each embodiment and modification, the polyhedron image is a switch, but it is not necessarily a polyhedron image, and may be a three-dimensional image. The “polyhedron” is a solid surrounded by a plurality of (four or more) planes, does not include a solid with a curved surface, and is limited to the case where the boundaries of all surfaces are straight lines. On the other hand, as a modification, a solid other than a polyhedron may be used as a switch. That is, a solid includes a curved surface, and a command is assigned to each surface or each area included in each surface. Such a solid may be used as a UI switch. Examples of solids other than polyhedrons include a cylinder, a cone, or a sphere.

Modification 6:
In each embodiment and modification, a balloon as an AR image is displayed in the see-through display areas P1 and P11. Alternatively, an AR image may not be displayed in the see-through display areas P1 and P11. In each embodiment and the modification, the see-through display areas P1 and P11 and the interface areas P2 and P12 are provided. However, instead of this, only the see-through display areas P1 and P11 are provided, and the see-through display areas P1 and P11 are provided. P11 may include a list display field LT or a work standard LT11, and cubic switches SW and SW11. That is, the field of view that can be visually recognized through the right and left light guide plates 26 and 28 may be configured by only one area or may be divided into a plurality of areas. Or a solid image other than the polyhedron) may be displayed.

Modification 7:
In the configuration in which the cubic switch is provided in the see-through display area of the modification example 6, the outside scene may be easily seen through the cubic switch. Specifically, by changing the luminance and / or size of the cubic switch based on the brightness of the outside scene, not only the cubic switch but also the outside scene through the cubic switch may be made easy to see. Further, the cubic switch may be a light color as a whole and an image in which only edges are coordinated.

-Modification 8:
In the first embodiment, the cubic switch SW is used to designate one store among a plurality of stores included in the field of view. In the second embodiment, the work standard document LT11 displayed in the interface area P12 is used. The cubic switch SW11 is used for page feed / page return / enlargement / reduction. On the other hand, a cubic switch may be used for designating one product among a plurality of products, and a book page forward / page return / used for various support such as learning support, sports support, etc. When enlarging / reducing, a cubic switch may be used. In short, any AR image may be used as long as a switch using a solid (polyhedron or a solid other than a polyhedron) is used to execute processing related to the AR image.

-Modification 9:
In each embodiment and modification, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. It may be.

Modification 10:
In the said embodiment, it illustrated about the structure of HMD. However, the configuration of the HMD can be arbitrarily determined without departing from the gist of the present invention, and for example, addition / deletion / conversion of components can be performed.

  In the above embodiment, the so-called transmission type HMD 100 in which the right light guide plate 26 and the left light guide plate 28 transmit external light has been described. However, the present invention can also be applied to a so-called non-transmissive HMD 100 that displays an image without transmitting through an outside scene, for example. Further, the non-transparent HMD 100 may capture an outside scene with a camera and display the captured image on a display unit. In these HMDs 100, in addition to AR (Augmented Reality) display that displays an image superimposed on the real space described in the above embodiment, MR (Mixed Reality) that displays a combination of the captured real space image and a virtual image. Display or VR (Virtual Reality) display for displaying a virtual space can also be performed.

  In the above embodiment, the functional units of the control device 10 and the image display unit 20 have been described, but these can be arbitrarily changed. For example, the following aspects may be adopted. A mode in which the storage function unit 122 and the control function unit 150 are mounted on the control device 10 and only the display function is mounted on the image display unit 20. A mode in which the storage function unit 122 and the control function unit 150 are mounted on both the control device 10 and the image display unit 20. The aspect which integrated the control apparatus 10 and the image display part 20. FIG. In this case, for example, the image display unit 20 includes all the components of the control device 10 and is configured as a glasses-type wearable computer. A mode in which a smartphone or a portable game device is used instead of the control device 10. The aspect which connected the control apparatus 10 and the image display part 20 by radio | wireless communication, and arranged the connection cable 40. FIG. In this case, for example, power supply to the control device 10 and the image display unit 20 may be performed wirelessly.

Modification 11:
In the said embodiment, it illustrated about the structure of the control apparatus. However, the configuration of the control device can be arbitrarily determined without departing from the gist of the present invention, and for example, addition / deletion / conversion of components can be performed.

  In the above embodiment, an example of the input unit provided in the control device 10 has been described. However, the control device 10 may be configured by omitting some of the illustrated input means, and may include other input means not described above. For example, the control device 10 may include an operation stick, a keyboard, a mouse, and the like. For example, the control device 10 may include an input unit that interprets a command associated with the movement of the user's body. The user's body movement or the like can be acquired by, for example, line-of-sight detection for detecting line-of-sight, gesture detection for detecting hand movement, foot switch for detecting foot movement, or the like. The line-of-sight detection can be realized by, for example, a camera that images the inside of the image display unit 20. Gesture detection can be realized, for example, by analyzing an image taken with the camera 61 over time.

  In the above embodiment, the control function unit 150 is operated by the main processor 140 executing the computer program in the storage function unit 122. However, the control function unit 150 can employ various configurations. For example, the computer program can be stored in the nonvolatile storage unit 121, the EEPROM 215, the memory 118, and other external storage devices (such as a USB memory inserted in various interfaces) instead of or together with the storage function unit 122. Or an external device such as a server connected via a network). Further, each function of the control function unit 150 may be realized using an ASIC (Application Specific Integrated Circuit) designed to realize the function.

Modification 12:
In the said embodiment, it illustrated about the structure of the image display part. However, the configuration of the image display unit can be arbitrarily determined without departing from the gist of the present invention, and for example, addition / deletion / conversion of components can be performed.

  FIG. 18 is a principal plan view showing the configuration of the optical system provided in the image display unit according to the modification. In the image display unit of the modified example, an OLED unit 221a corresponding to the user's right eye RE and an OLED unit 241a corresponding to the left eye LE are provided. The OLED unit 221a corresponding to the right eye RE includes an OLED panel 223a that develops white color and an OLED drive circuit 225 that drives the OLED panel 223a to emit light. A modulation element 227 (modulation device) is arranged between the OLED panel 223a and the right optical system 251. The modulation element 227 is configured by, for example, a transmissive liquid crystal panel, and generates image light L by modulating light emitted from the OLED panel 223a. The image light L that has been transmitted through the modulation element 227 and is modulated is guided to the right eye RE by the right light guide plate 26.

  The OLED unit 241a corresponding to the left eye LE includes an OLED panel 243a that emits white light and an OLED drive circuit 245 that drives the OLED panel 243a to emit light. A modulation element 247 (modulation device) is arranged between the OLED panel 243a and the left optical system 252. The modulation element 247 is constituted by, for example, a transmissive liquid crystal panel, and generates image light L by modulating light emitted from the OLED panel 243a. The image light L that has been modulated through the modulation element 247 is guided to the left eye LE by the left light guide plate 28. The modulation elements 227 and 247 are connected to a liquid crystal driver circuit (not shown). The liquid crystal driver circuit (modulation device driving unit) is mounted on a substrate disposed in the vicinity of the modulation elements 227 and 247, for example.

  According to the image display unit of the modified example, the right display unit 22 and the left display unit 24 respectively include OLED panels 223a and 243a as light source units and image light including a plurality of color lights by modulating light emitted from the light source units. Are provided as modulation elements 227 and 247 that output a video signal. Note that the modulation device that modulates the light emitted from the OLED panels 223a and 243a is not limited to a configuration in which a transmissive liquid crystal panel is employed. For example, a reflective liquid crystal panel may be used instead of the transmissive liquid crystal panel, a digital micromirror device may be used, or a laser retinal projection type HMD 100 may be used.

  In the above embodiment, the spectacle-type image display unit 20 has been described, but the aspect of the image display unit 20 can be arbitrarily changed. For example, the image display unit 20 may be mounted like a hat, or may be built in a body protector such as a helmet. Further, the image display unit 20 may be configured as a HUD (Head Up Display) mounted on a vehicle such as an automobile or an airplane, or other transportation means.

  In the above embodiment, a configuration in which virtual images are formed by the half mirrors 261 and 281 on a part of the right light guide plate 26 and the left light guide plate 28 as an optical system that guides image light to the user's eyes has been exemplified. However, this configuration can be arbitrarily changed. For example, a virtual image may be formed in a region that occupies the entire surface (or most) of the right light guide plate 26 and the left light guide plate 28. In this case, the image may be reduced by an operation for changing the display position of the image. The optical element of the present invention is not limited to the right light guide plate 26 and the left light guide plate 28 having the half mirrors 261 and 281, but is an optical component (for example, a diffraction grating, a prism, or the like) that makes image light incident on the user's eyes. Any form can be adopted as long as holography or the like is used.

Modification 13:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

    DESCRIPTION OF SYMBOLS 1 ... Information processing system, 10 ... Control apparatus, 12 ... Lighting part, 14 ... Touch pad, 16 ... Direction key, 17 ... Decision key, 18 ... Power switch, 19 ... Vibrator, 20 ... Image display part, 21 ... Right hold , 22 ... right display unit, 23 ... left holding unit, 24 ... left display unit, 26 ... right light guide plate, 27 ... front frame, 28 ... left light guide plate, 30 ... headset, 32 ... right earphone, 34 ... Left earphone, 40 ... connection cable, 46 ... connector, 61 ... camera, 63 ... microphone, 65 ... illuminance sensor, 67 ... LED indicator, 100 ... head-mounted display device (HMD), 110 ... operating unit, 111 ... 6 Axis sensor, 113 ... Magnetic sensor, 115 ... GPS receiver, 117 ... Wireless communication unit, 118 ... Memory, 120 ... Controller board, 121 ... Non-volatile Storage unit 122 ... Storage function unit 123 ... Setting data 124 ... Content data 130 ... Power supply unit 132 ... Battery 134 134Power control circuit 140 ... Main processor 145 ... Image processing unit 147 ... Display control unit 149: Imaging control unit, 150: Control function unit, 151 ... Input / output control unit, 153 ... Communication control unit, 155 ... Store input processing unit, 155a ... polyhedron image display unit, 155b ... polyhedron orientation switching unit, 155c ... command Execution unit, 180 ... audio codec, 182 ... audio interface, 184 ... external connector, 186 ... external memory interface, 188 ... USB connector, 192 ... sensor hub, 196 ... interface, 210 ... display unit board, 211 ... interface, 213 ... Receiving unit, 215 ... EEPROM, DESCRIPTION OF SYMBOLS 17 ... Temperature sensor 221, 221a ... OLED unit, 223, 223a ... OLED panel, 225 ... OLED drive circuit, 227 ... Modulation element, 230 ... Display unit board | substrate, 231 ... Interface, 233 ... Reception part, 235 ... 6 axis sensor 237 ... magnetic sensor, 239 ... temperature sensor, 241, 241a ... OLED unit, 243, 243a ... OLED panel, 245 ... OLED drive circuit, 247 ... modulation element, 251 ... right optical system, 252 ... left optical system, 261 ... Half mirror, 281 ... Half mirror, 300 ... Store server, 310 ... CPU, 312 ... Peripheral store information providing unit, 314 ... Store guide unit, 320 ... Storage unit, 322 ... Store database, 324 ... Map database, 330 ... ROM, 340 ... RAM, 360 ... Bus, B ... [Store name] button, B2 ... [Detail] button, BS ... Communication carrier, FG ... Fingertip, INT ... Internet, LT ... List display column, OL ... External light, P1 ... See-through display area, P11 ... See-through display area, P12 ... Interface area, SW ... Cubic switch, SW11 ... Cubic switch, SW21 ... Cubic switch, SW31 ... Cubic switch, SW41 ... Cubic switch

Claims (10)

A head-mounted display device having a display unit,
A display control unit for displaying an image on the display unit;
A processing unit that executes processing related to the displayed image;
With
The processor is
A polyhedron image display unit for displaying a polyhedron image in which an instruction is assigned to each surface of the polyhedron on the display unit;
In response to an operation on the polyhedron image, a polyhedron direction switching unit that switches a displayed direction of the polyhedron;
A command execution unit that executes the command assigned to a plane in a predetermined direction in the polyhedral image;
A head-mounted display device comprising: The head-mounted display device according to claim 1,
The display unit is configured to visually recognize an outside scene,
The processing unit is a head-mounted display device that causes the display unit to display, as the image, a related image related to the visually recognized outside scene. The head-mounted display device according to claim 2,
The related image includes identification information for identifying a plurality of stores included in the visually recognized outside scene,
The head-mounted display device, wherein the processing executed by the processing unit is store input processing for designating one store from the plurality of stores. The head-mounted display device according to claim 2,
The related image includes a support image for supporting a work related to the visually recognized outside scene,
The head-mounted display device, wherein the processing executed by the processing unit is processing for sequentially switching the support images. The head-mounted display device according to any one of claims 1 to 4,
The polyhedron image is a regular hexahedron 3D image,
The head-mounted display device, wherein the predetermined orientation surface is a surface facing a user. The head-mounted display device according to any one of claims 1 to 5,
The head-mounted display device, wherein the operation on the polyhedral image is an operation of flicking with a fingertip. A head-mounted display device having a display unit,
A display control unit for displaying an image on the display unit;
A processing unit that executes processing related to the displayed image;
With
The processor is
A stereoscopic image display unit for displaying a stereoscopic image in which a command is assigned to each surface of the stereoscopic unit on the display unit;
A three-dimensional direction switching unit that receives an operation on the three-dimensional image and switches a direction in which the three-dimensional image is displayed;
A command execution unit that executes the command assigned to a surface in a predetermined direction in the stereoscopic image;
A head-mounted display device comprising: A method for controlling a head-mounted display device including a display unit,
Displaying an image on the display unit;
Executing a process related to the displayed image;
With
The step of executing the process includes:
Displaying a polyhedral image in which an instruction is assigned to each surface of the polyhedron on the display unit;
Receiving the operation on the polyhedron image, and switching the displayed direction of the polyhedron;
Executing the command assigned to a face in a predetermined orientation in the polyhedral image;
A method for controlling a head-mounted display device comprising: A computer program for controlling a head-mounted display device having a display unit,
A function of displaying an image on the display unit;
A function of executing processing related to the displayed image;
To the computer,
The function of executing the process is as follows:
A function of causing the display unit to display a polyhedral image in which an instruction is assigned to each surface of the polyhedron;
A function of switching the display direction of the polyhedron in response to an operation on the polyhedron image;
A function for executing the command assigned to a plane in a predetermined direction in the polyhedral image;
A computer program comprising A computer program for controlling a head-mounted display device having a display unit,
A function of displaying an image on the display unit;
A function of executing processing related to the displayed image;
To the computer,
The function of executing the process is as follows:
A function of displaying a stereoscopic image in which a command is assigned to each surface of the stereoscopic image on the display unit;
A function of switching the direction in which the stereoscopic image is displayed in response to an operation on the stereoscopic image;
A function of executing the command assigned to a plane in a predetermined direction in the stereoscopic image;
A computer program comprising

Images