JP2018200415A - Head-mounted display device, program, and method for controlling head-mounted display device - Google Patents

Abstract

To improve operability to improve usability of a head-mounted display device.SOLUTION: An HMD 100 comprises: an image display unit 20 that displays a display image in a display area so that a user can visually recognize the display image together with exterior scenery; a display control unit 153 that controls the image display unit 20 to display a pointer 301 indicating an indication position in a display area 310; and a six-axis sensor 235 and detection control unit 151 that detect operations. The display control unit 153 switches between a first operation mode for moving the position of the pointer 301 corresponding to the exterior scenery according to the operations detected by the six-axis sensor 235 and detection control unit 151, and a second operation mode for moving the position of the pointer 301 in a mode different from the movement corresponding to the exterior scenery according to the operations detected by the six-axis sensor 235 and detection control unit 151.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a head-mounted display device, a program, and a control method for the head-mounted display device.

2. Description of the Related Art Conventionally, a pointing device that moves a marker to select a displayed menu or designate a displayed object is known. As a pointing device, a movement of a user's head is detected, and a marker is moved according to the detected movement of the head (for example, see Patent Document 1).
Patent Document 1 detects changes in the position and orientation of the head based on ultrasonic waves, electromagnetic waves, and the like transmitted from a plurality of transmitters mounted on the head, and detects the detected position and orientation of the head. A head-driven pointing device is disclosed that indicates a specific position on a display in response to a change.

JP-A-6-187092

By the way, when operating the marker by the movement of the head in the head-mounted display device, the user must keep the head tilted while moving the marker, and must maintain an unreasonable posture was there. Also, when operating the operation unit such as a trackpad to move the marker to the target position, if the operation unit is not fixed, it is not easy to move the marker as the user imagined, and operability There is a problem of lacking. For this reason, improvement in the operability of the head-mounted display device has been desired.
An object of the present invention is to improve operability and improve the usability of a head-mounted display device.

In order to solve the above problems, the present invention is a head-mounted display device mounted on a user's head, the display unit displaying a display image in a display area so as to be visible together with an outside scene, and an indication position A display control unit that causes the display unit to display the marker in the display area, and a detection unit that detects an operation, and the display control unit detects the position of the marker by the operation detected by the detection unit. And a second operation mode in which the position of the marker is moved in a manner different from the movement corresponding to the outside scene by the operation detected by the detection unit. Switch.
According to the present invention, it is possible to switch and execute the first operation mode in which the position of the marker is moved corresponding to the outside scene and the second operation mode in which the marker is moved in a manner different from the movement corresponding to the outside scene. For this reason, the marker can be moved in a plurality of modes, the operability of the marker can be improved, and the usability of the head-mounted display device can be improved.

According to the present invention, the detection unit detects a first operation and a second operation as the operation, and the display control unit determines the position of the marker according to the first operation detected by the detection unit. After the movement of the marker according to the first operation, the position of the marker is different from the movement of the marker according to the first operation according to the second operation detected by the detection unit. Move in a manner.
According to the present invention, the position of the marker can be moved by a plurality of operations of the first operation and the second operation, and the marker can be moved in a different manner between the first operation and the second operation. For this reason, the operability of the marker can be improved and the usability of the head-mounted display device can be improved.

Further, according to the present invention, the display control unit moves the position of the marker differently from the first operation according to the second operation, starting from the position of the marker moved by the first operation. Let
According to the present invention, the position of the pointer can be moved by the second operation, starting from the position of the marker moved by the first operation.

In the present invention, the detection unit detects the user's movement as at least one of the first operation and the second operation.
According to the present invention, since the user's motion is detected as at least one of the first operation and the second operation, the position of the marker can be moved by the motion.

In the present invention, the detection unit detects at least one of the inclination of the head relative to the body axis of the user and the rotation of the head around the body axis as the user's motion, The display control unit, based on at least one of the tilt of the head detected by the detection unit and the rotation around the body axis, the marker in the vertical direction or the horizontal direction of the display region The head-mounted display device according to claim 4, wherein a movement amount is determined and the position of the marker is moved based on the determined movement amount.
According to the present invention, the amount of movement of the marker in one of the vertical direction and the horizontal direction of the display area is determined by the tilt of the user's head or the rotation of the head around the body axis. Therefore, the marker can be moved by tilting the user's head or rotating the head around the body axis.

In the present invention, the detection unit detects an operation received by an operation unit included in the head-mounted display device as the first operation, and detects a movement of the head as the second operation.
According to the present invention, since the operation of the operation unit is detected as the first operation and the movement of the head is detected as the second operation, the position of the marker can be moved by the operation of the operation unit and the movement of the head. .

In addition, the present invention includes an imaging unit that images the outside scene, and the display control unit is an object that is visually recognized by overlapping the marker based on a captured image of the imaging unit during execution of the first operation mode. An object is specified, and the display image corresponding to the position of the marker is specified from the plurality of display images displayed in the display area during execution of the second operation mode.
According to the present invention, during execution of the first operation mode, a target object that is visually recognized overlapping with the marker is specified, and during execution of the second operation mode, a display image corresponding to the position of the marker is specified. For this reason, the target to be identified can be changed to the target object or the display image by changing the operation mode.

In addition, according to the present invention, the display control unit may detect the first operation or the second operation when the detection unit detects the first operation or the second operation during the execution of the first operation mode. When the marker is moved based on the outside scene visually recognized by the user according to an operation, and the first operation or the second operation is detected by the detection unit during the execution of the second operation mode. In response to the first operation or the second operation, the marker is moved with reference to the display image visually recognized by the user.
According to the present invention, the marker is moved with reference to the outside scene visually recognized by the user according to the first operation or the second operation detected during the execution of the first operation mode, and detected during the execution of the second operation mode. The marker is moved with reference to a display image that the user visually recognizes in accordance with the first operation or the second operation. Therefore, by switching the operation mode, the marker can be easily aligned with the target object or the display image.

Further, in the present invention, when the display control unit moves the marker in response to the second operation, the display control unit sets an amount of movement to move the marker in response to the second operation according to the first operation. By making it smaller than the movement amount when moving the marker, the movement of the marker according to the first operation and the movement of the marker according to the second operation are moved in different modes.
According to the present invention, the amount of movement of the pointer moved according to the second operation is smaller than the amount of movement of the pointer moved according to the first operation. For this reason, after the pointer is moved by the first operation having a larger moving amount than the second operation, the pointer is moved by the second operation having the smaller moving amount than the first operation, thereby moving the pointer to the target object or the display image. The number of operations for moving to the position can be reduced.

The present invention may further include a distance detection unit that detects a distance to a target object that the user sees through the display unit and the display control unit detects the target object detected by the distance detection unit. Depending on the distance, the marker is moved in a different manner.
According to the present invention, the pointer can be moved in different manners depending on the distance to the target object. Therefore, the operability of the pointer operation can be improved, and the usability of the head-mounted display device can be improved.

Further, in the present invention, the display control unit sets the operation detected by the detection unit when the distance to the target object detected by the distance detection unit is larger than a threshold as the first operation, When the distance to the target object detected by the distance detection unit is equal to or less than a threshold value, the operation detected by the detection unit is the second operation, and the amount of movement by which the marker is moved by the first operation, The amount of movement is smaller than the amount of movement of the marker by the second operation.
According to the present invention, when the distance to the target object is larger than the threshold value, the first operation whose movement amount is larger than the second operation is detected, and when the distance to the target object is less than the threshold value, The second operation whose movement amount is smaller than the first operation is detected. For this reason, the operation used for the movement of the marker can be changed according to the distance to the target object, and the amount of movement of the marker can be changed. Therefore, the operability of the marker operation can be improved and the usability of the head-mounted display device can be improved.

In order to solve the above-described problems, the present invention is a head-mounted display device that is worn on a user's head, and displays a display image in a display region so as to transmit the outside scene and be visible together with the outside scene. A display unit; an imaging unit that images the outside scene; and a display control unit that causes the display unit to display a marker indicating an indication position in the display area. The display control unit includes a captured image of the imaging unit. And a second operation for specifying the display image corresponding to the position of the marker from the plurality of display images displayed in the display area. Switch between modes.
According to the present invention, in the first operation mode, a target object that is visually recognized while overlapping with the marker is specified, and in the second operation mode, a display image corresponding to the position of the marker is specified. For this reason, the identifiable target can be switched by changing the operation mode to the first operation mode or the second operation mode.

The present invention further includes a detection unit that detects an operation, and the display control unit responds to the detected operation when an operation is detected by the detection unit during execution of the first operation mode. The marker is moved with reference to the outside scene visually recognized by the user, and when the operation is detected by the detection unit during execution of the second operation mode, the marker is detected according to the detected operation. Are moved with reference to the display image visually recognized by the user.
According to the present invention, the marker is moved with reference to the outside scene visually recognized by the user according to the operation detected during the execution of the first operation mode, and according to the operation detected during the execution of the second operation mode. The marker is moved with reference to a display image visually recognized by the user. For this reason, by changing the operation mode to the first operation mode or the second operation mode, the reference for moving the marker can be changed, and the marker can be easily aligned with the target object or the display image.

Further, according to the present invention, the detection unit detects an operation received by an operation unit included in the head-mounted display device as a first operation, detects a movement of the user's head as a second operation, and displays the display. The control unit moves the position of the marker according to the first operation detected by the detection unit, and the second operation detected by the detection unit after the position of the marker is moved according to the first operation. The position of the marker is moved according to the movement, and the movement amount when moving the marker according to the second operation is made smaller than the movement amount when moving the marker according to the first operation.
According to the present invention, the amount of movement of the marker moved according to the second operation is smaller than the amount of movement of the marker moved according to the first operation. Therefore, after the marker is moved by the first operation having a larger movement amount than the second operation, the marker is moved by the second operation having the smaller movement amount than the first operation, whereby the marker is moved to the target object or the display image. The number of operations for moving to a position can be reduced.

In the present invention, the display control unit displays a display indicating a movement direction of the marker with reference to the outside scene during the execution of the first operation mode, and the execution of the second operation mode A display indicating the movement direction of the marker with reference to the display image is displayed.
According to the present invention, during the execution of the first operation mode, the movement direction of the marker based on the outside scene is displayed, and during the execution of the second operation mode, the movement direction of the marker based on the display image is displayed. Is done. Therefore, the user can recognize the moving direction of the marker in the first operation mode and the second operation mode.

Further, in the present invention, the display control unit manages the display position of the marker by coordinates in the display area, and when the operation detected by the detection unit is set to be invalid, the display position of the marker is set. And managed in association with the coordinates of the three-dimensional space corresponding to the real space.
According to the present invention, when the display position of the marker is managed by the coordinates in the display area and the operation detected by the detection unit is set to be invalid, the display position of the marker is set in the three-dimensional space corresponding to the real space. Manage by associating with coordinates. Therefore, when the second operation is invalidated, the marker display position is managed in association with the coordinates in the three-dimensional space corresponding to the real space, so even if the movement of the head is detected, The display position is not changed.

In addition, the present invention includes an operation unit that receives an operation, and the display control unit detects a moving object that moves based on a captured image of the imaging unit, and the position of the display region corresponding to the detected moving object To display the marker.
According to the present invention, it is possible to detect a moving object from a captured image of the imaging unit and display a marker at a position of a display area corresponding to the moving object.

In addition, the present invention includes a mark detection unit that detects an identification mark arranged on the target object, and the display control unit is a position of the target object on which the identification mark detected by the mark detection unit is arranged. The marker is displayed, and the position of the marker is moved according to the operation detected by the detection unit.
According to the present invention, the marker can be displayed at the position of the target object where the identification mark is arranged, and the position of the displayed marker can be moved by the operation. For this reason, marker alignment can be performed easily.

The present invention further includes a receiving unit that receives a beacon signal transmitted from a beacon transmitter disposed on the target object, and the display control unit is configured to receive the beacon transmitter based on the beacon signal received by the receiving unit. And the additional information transmitted from the transmitter is received and displayed in the display area.
According to the present invention, it is possible to connect to a beacon transmitter based on a beacon signal transmitted from a beacon transmitter, and acquire and display additional information from the beacon transmitter.

In order to solve the above problems, the present invention provides a display unit that displays a display image in a display area so that it can be seen through the outside scene and is visible together with the outside scene, and a marker that indicates an indicated position is displayed in the display area by the display unit. A program that is executed by a computer that is mounted on a head-mounted display device that is mounted on a user's head, the display control unit being operated, and a detection unit that detects an operation. The procedure of moving the position of the marker corresponding to the outside scene by the operation detected by the part and the movement of the marker corresponding to the outside scene by the operation detected by the detecting unit are different. And a procedure for moving.
According to the present invention, the procedure for moving the position of the marker corresponding to the outside scene by the operation detected by the detection unit is different from the movement corresponding to the outside scene by the operation detected by the detection unit. The procedure to move can be switched. Therefore, the position of the marker can be moved in a plurality of modes, the operability of the marker can be improved, and the usability of the head-mounted display device can be improved.

In order to solve the above problems, the present invention provides a display unit that displays a display image in a display area so that it can be seen through the outside scene and is visible together with the outside scene, and a marker that indicates an indicated position is displayed in the display area by the display unit. A display control unit for detecting an operation, and a detection unit for detecting an operation, and a control method for a head-mounted display device mounted on a user's head, wherein the detection unit detects the operation, And a step of moving the position of the marker corresponding to the outside scene, and a step of moving the position of the marker differently from the movement corresponding to the outside scene by the operation detected by the detection unit.
According to the present invention, the step of moving the marker position corresponding to the outside scene by the operation detected by the detection unit and the movement of the marker position corresponding to the outside scene by the operation detected by the detection unit are different. The moving step can be switched. Therefore, the position of the marker can be moved in a plurality of modes, the operability of the marker can be improved, and the usability of the head-mounted display device can be improved.

FIG. The principal part top view which shows the structure of the optical system of HMD. The perspective view which shows the structure of an image display part. The block diagram of HMD. The functional block diagram of a control apparatus. Explanatory drawing for demonstrating rotation around the Z-axis of a user's head. Explanatory drawing for demonstrating rotation around the Y-axis of a user's head. Explanatory drawing for demonstrating rotation around the X-axis of a user's head. The figure which shows the visual field of the user who mounted | wore the image display part. Explanatory drawing for demonstrating the visual field of the user with which the image display part was mounted | worn. Explanatory drawing for demonstrating the visual field of the user with which the image display part was mounted | worn. Explanatory drawing for demonstrating the visual field of the user with which the image display part was mounted | worn. Explanatory drawing for demonstrating the visual field of the user with which the image display part was mounted | worn. Explanatory drawing for demonstrating the visual field of the user with which the image display part was mounted | worn. Explanatory drawing for demonstrating the visual field of the user with which the image display part was mounted | worn. Explanatory drawing for demonstrating the visual field of the user with which the image display part was mounted | worn. Explanatory drawing for demonstrating the visual field of the user with which the image display part was mounted | worn. Explanatory drawing for demonstrating the visual field of the user with which the image display part was mounted | worn. Explanatory drawing for demonstrating the visual field of the user with which the image display part was mounted | worn. Explanatory drawing for demonstrating the relationship between the distance to a target, and the movement distance of a pointer. The flowchart which shows operation | movement of HMD. Explanatory drawing for demonstrating arrangement | positioning of the detection apparatus with which the user's shoe sole or the inside of shoes was mounted | worn. Explanatory drawing for demonstrating arrangement | positioning of the detection apparatus with which the user's shoe sole or the inside of shoes was mounted | worn. Explanatory drawing for demonstrating arrangement | positioning of the detection apparatus with which the user's shoe sole or the inside of shoes was mounted | worn.

[First Embodiment]
FIG. 1 is an external view showing an external configuration of an HMD (Head Mounted Display) 100 to which the present invention is applied.
HMD100 is a display apparatus provided with the image display part 20 (display part) which makes a user visually recognize a virtual image in the state with which the user (user) was mounted | worn, and the control apparatus 10 which controls the image display part 20. It is.

  The control device 10 includes a flat box-shaped case 10A (also referred to as a housing or a main body) as shown in FIG. Case 10 </ b> A includes buttons 11, LED indicator 12, track pad 14, up / down key 15, changeover switch 16, and power switch 18. The button 11, the track pad 14, the up / down key 15, the changeover switch 16 and the power switch 18 are operated parts operated by the user. In addition, the LED indicator 12 functions as a sub display unit that indicates an operation state of the HMD 100, for example. The user can operate the HMD 100 by operating the operated part. The control device 10 functions as a controller of the HMD 100.

The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 in a main body having a right holding unit 21, a left holding unit 23, and a front frame 27.
Each of the right holding unit 21 and the left holding unit 23 extends rearward from both ends of the front frame 27, and holds the image display unit 20 on the user's head like a temple of glasses. Here, of both ends of the front frame 27, an end located on the right side of the user in the mounted state of the image display unit 20 is defined as an end ER, and an end located on the left side of the user is defined as an end EL. To do. The right holding unit 21 extends from the end ER of the front frame 27 to a position corresponding to the right side of the user when the image display unit 20 is mounted. The left holding unit 23 is provided to extend from the end EL to a position corresponding to the left side of the user when the image display unit 20 is mounted.

  The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is positioned in front of the right eye of the user when the image display unit 20 is mounted, and causes the right eye to visually recognize the image. The left light guide plate 28 is positioned in front of the left eye of the user when the image display unit 20 is mounted, and causes the left eye to visually recognize the image.

  The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other, and this connection position is in a wearing state in which the user wears the image display unit 20. Corresponding to the eyebrows. The front frame 27 may be provided with a nose pad portion that comes into contact with the user's nose when the image display unit 20 is mounted at a connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the user's head by the nose pad, the right holding unit 21 and the left holding unit 23. Further, a belt (not shown) that contacts the back of the user when the image display unit 20 is mounted may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 is used by the belt. Can be held on the person's head.

  The right display unit 22 realizes image display by the right light guide plate 26. The right display unit 22 is provided in the right holding unit 21 and is located in the vicinity of the right side of the user in the mounted state. The left display unit 24 realizes image display by the left light guide plate 28. The left display unit 24 is provided in the left holding part 23 and is located in the vicinity of the left side of the user in the wearing state.

The right light guide plate 26 and the left light guide plate 28 of the present embodiment are optical units formed of a light-transmitting resin or the like, and are, for example, prisms, and output image light output from the right display unit 22 and the left display unit 24. Lead to the eyes of the user.
A light control plate (not shown) may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The light control plate is an optical element on a thin plate having different transmittance depending on the wavelength region of light, and functions as a so-called wavelength filter. For example, the light control plate is disposed so as to cover the front side of the front frame 27 which is the side opposite to the eye side of the user. By appropriately selecting the optical characteristics of the light control plate, the transmittance of light in an arbitrary wavelength region such as visible light, infrared light, and ultraviolet light can be adjusted. The amount of external light incident on the light plate 28 and transmitted through the right light guide plate 26 and the left light guide plate 28 can be adjusted.

  The image display unit 20 includes an optical system that guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, respectively (see FIG. 2). The image light guided to the right light guide plate 26 and the left light guide plate 28 enters the right eye and the left eye of the user and makes the user visually recognize a virtual image. Thereby, the image display unit 20 displays an image.

  When the external light enters the user's eyes through the right light guide plate 26 and the left light guide plate 28 from the front of the user, the image light and the external light constituting the virtual image enter the user's eyes. In other words, the visibility of the virtual image is affected by the intensity of external light. For this reason, for example, by attaching a light control plate to the front frame 27 and appropriately selecting or adjusting the optical characteristics of the light control plate, the ease of visual recognition of the virtual image can be adjusted. In a typical example, it is possible to use a light control plate having a light transmittance that allows a user wearing the HMD 100 to visually recognize at least the outside scenery. In addition, when the dimming plate is used, it is possible to protect the right light guide plate 26 and the left light guide plate 28 and to suppress the damage to the right light guide plate 26 and the left light guide plate 28, the adhesion of dirt, and the like. The light control plate may be detachable with respect to the front frame 27 or each of the right light guide plate 26 and the left light guide plate 28, or may be mounted by replacing a plurality of types of light control plates. May be omitted.

The camera 61 is disposed on the front frame 27 of the image display unit 20. The configuration and arrangement of the camera 61 are determined so as to capture an outside scene direction visually recognized by the user while wearing the image display unit 20. For example, the camera 61 is provided on the front surface of the front frame 27 at a position that does not block external light that passes through the right light guide plate 26 and the left light guide plate 28. In the example of FIG. 1, the camera 61 is disposed on the end ER side of the front frame 27, but the camera 61 may be disposed on the end EL side, and the right light guide plate 26 and the left light guide plate 28. You may arrange | position in the connection part.
The camera 61 corresponds to an “imaging unit” of the present invention.

The camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like. The camera 61 of this embodiment is a monocular camera, but may be a stereo camera. The camera 61 images at least a part of the outside scene (real space) in the front side direction of the HMD 100, in other words, the user's viewing direction when the HMD 100 is worn. In another expression, the camera 61 captures a range or direction that overlaps the user's field of view, and captures a direction in which the user gazes. The direction and width of the angle of view of the camera 61 can be set as appropriate. In the present embodiment, as will be described later, the angle of view of the camera 61 includes the outside world visually recognized by the user through the right light guide plate 26 and the left light guide plate 28. More preferably, the angle of view of the camera 61 is set so that the entire field of view of the user who can see through the right light guide plate 26 and the left light guide plate 28 can be imaged.
The camera 61 performs imaging according to the control of the imaging control unit 147 included in the control unit 150 (FIG. 5). The camera 61 outputs captured image data to the control unit 150 via an interface 211 described later.

  The HMD 100 may include a distance sensor (not shown) that detects a distance to a measurement object positioned in a preset measurement direction. The distance sensor can be disposed, for example, at a connection portion between the right light guide plate 26 and the left light guide plate 28 in the front frame 27. In this case, when the image display unit 20 is mounted, the position of the distance sensor is substantially in the middle between the user's eyes in the horizontal direction and above the user's eyes in the vertical direction. The measurement direction of the distance sensor can be, for example, the front side direction of the front frame 27, in other words, the direction overlapping the imaging direction of the camera 61. The distance sensor can be configured to include, for example, a light source such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured. The distance sensor may perform a triangulation process or a distance measurement process based on a time difference in accordance with the control of the control unit 150. The distance sensor may be configured to include a sound source that emits ultrasonic waves and a detection unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance sensor may perform the distance measurement process based on the time difference until the reflection of the ultrasonic wave according to the control of the control unit 150.

FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. FIG. 2 shows the user's left eye LE and right eye RE for explanation.
As shown in FIG. 2, the right display unit 22 and the left display unit 24 are configured symmetrically. The right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 that emits image light and a lens group that guides the image light L emitted from the OLED unit 221 as a configuration that allows the user's right eye RE to visually recognize an image. Right optical system 251. The image light L is guided to the right light guide plate 26 by the right optical system 251.

The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-luminous display panel configured by arranging, in a matrix, light emitting elements that emit light of R (red), G (green), and B (blue) by emitting light by organic electroluminescence. It is. The OLED panel 223 includes a plurality of pixels, each of which includes one R, G, and B element, and forms an image with pixels arranged in a matrix. The OLED drive circuit 225 selects a light emitting element included in the OLED panel 223 and energizes the light emitting element under the control of the control unit 150 (FIG. 5), and causes the light emitting element of the OLED panel 223 to emit light. The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back surface of the light emitting surface by bonding or the like. The OLED drive circuit 225 may be formed of a semiconductor device that drives the OLED panel 223, for example, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 223. A temperature sensor 217 is mounted on this substrate.
Note that the OLED panel 223 may have a configuration in which light emitting elements that emit white light are arranged in a matrix, and color filters corresponding to R, G, and B colors are stacked. Further, an OLED panel 223 having a WRGB configuration provided with a light emitting element that emits W (white) light in addition to the light emitting elements that respectively emit R, G, and B color light may be used.

  The right optical system 251 includes a collimator lens that converts the image light L emitted from the OLED panel 223 into a parallel light flux. The image light L converted into a parallel light beam by the collimator lens is incident on the right light guide plate 26. A plurality of reflecting surfaces that reflect the image light L are formed in an optical path that guides light inside the right light guide plate 26. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 (reflection surface) located in front of the right eye RE is formed on the right light guide plate 26. The image light L is reflected by the half mirror 261 and emitted from the right light guide plate 26 toward the right eye RE. The image light L forms an image on the retina of the right eye RE, and allows the user to visually recognize the image.

  In addition, the left display unit 24 has a left optical system including an OLED unit 241 that emits image light and a lens group that guides the image light L emitted from the OLED unit 241 as a configuration that allows the user's left eye LE to visually recognize an image. 252. The image light L is guided to the left light guide plate 28 by the left optical system 252.

  The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. The OLED panel 243 is a self-luminous display panel configured similarly to the OLED panel 223. The OLED driving circuit 245 causes the light emitting elements of the OLED panel 243 to emit light by selecting a light emitting element included in the OLED panel 243 and energizing the light emitting element according to the control of the control unit 150 (FIG. 5). The OLED drive circuit 245 is fixed to the back surface of the OLED panel 243, that is, the back surface of the light emitting surface by bonding or the like. The OLED drive circuit 245 may be configured by a semiconductor device that drives the OLED panel 243, for example, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 243. A temperature sensor 239 is mounted on this substrate.

  The left optical system 252 includes a collimating lens that converts the image light L emitted from the OLED panel 243 into a light beam in a parallel state. The image light L converted into a parallel light beam by the collimator lens enters the left light guide plate 28. The left light guide plate 28 is an optical element formed with a plurality of reflecting surfaces that reflect the image light L, and is, for example, a prism. The image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 28. The left light guide plate 28 is formed with a half mirror 281 (reflection surface) located in front of the left eye LE. The image light L is reflected by the half mirror 281 and emitted from the left light guide plate 28 toward the left eye LE. The image light L forms an image on the retina of the left eye LE, and allows the user to visually recognize the image.

According to this configuration, the HMD 100 functions as a see-through display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 are incident on the user's right eye RE. Further, the image light L reflected by the half mirror 281 and the external light OL transmitted through the half mirror 281 are incident on the left eye LE. In this way, the HMD 100 superimposes the image light L of the internally processed image and the external light OL so as to enter the user's eyes, and for the user, the outside light is transmitted through the right light guide plate 26 and the left light guide plate 28. The image by the image light L is visually recognized over the outside scene.
The half mirrors 261 and 281 are image extraction units that extract image by reflecting image light output from the right display unit 22 and the left display unit 24, respectively, and can be called display units.

  The left optical system 252 and the left light guide plate 28 are collectively referred to as a “left light guide”, and the right optical system 251 and the right light guide plate 26 are collectively referred to as a “right light guide”. The configuration of the right light guide and the left light guide is not limited to the above example, and any method can be used as long as a virtual image is formed in front of the user's eyes using image light. It may be used, or a semi-transmissive reflective film may be used.

Returning to FIG. 1, the control device 10 and the image display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to a connector (not shown) provided at the lower part of the case 10 </ b> A, and is connected to various circuits provided in the image display unit 20 from the tip of the left holding unit 23. The connection cable 40 includes a metal cable that transmits digital data or an optical fiber cable, and may include a metal cable that transmits an analog signal. A connector 46 is provided in the middle of the connection cable 40. The connector 46 is a jack for connecting a stereo mini plug, and the connector 46 and the control device 10 are connected by a line for transmitting an analog audio signal, for example. In the configuration example shown in FIG. 1, a headset 30 having a right earphone 32 and a left earphone 34 and a microphone 63 constituting a stereo headphone is connected to a connector 46.
The control device 10 and the image display unit 20 may be wirelessly connected. For example, the control device 10 and the image display unit 20 mutually transmit and receive control signals and data by wireless communication complying with standards such as Bluetooth (registered trademark) and wireless LAN (including Wi-Fi (registered trademark)). It is good also as a structure.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collection unit of the microphone 63 faces the user's line of sight, collects sound, and outputs a sound signal to the sound interface 182 (FIG. 4). To do. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.

  The control device 10 includes a button 11, an LED indicator 12, a track pad 14, an up / down key 15, a changeover switch 16, and a power switch 18 as operated parts operated by a user. These operated parts are arranged on the surface of the case 10A.

The button 11 includes keys and switches for operating the control device 10, and these keys and switches are displaced by a pressing operation. For example, the button 11 includes a menu key, a home key, and a “return” key for performing operations related to the operating system 143 (see FIG. 5) executed by the control device 10.
The LED indicator 12 lights up or blinks in accordance with the operating state of the HMD 100. The up / down key 15 is used to input an instruction to increase / decrease the volume output from the right earphone 32 and the left earphone 34 and to input an instruction to increase / decrease the display brightness of the image display unit 20. The changeover switch 16 is a switch for changing an input corresponding to the operation of the up / down key 15. The power switch 18 is a switch that switches on / off the power of the HMD 100, and is, for example, a slide switch.

  The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. The detection method on the operation surface is not limited, and an electrostatic method, a pressure detection method, an optical method, or other methods can be adopted. Contact (touch operation) to the track pad 14 is detected by a touch sensor (not shown). An LED display unit 17 is installed on the track pad 14. The LED display unit 17 includes a plurality of LEDs, and light of each LED is transmitted through the track pad 14 to display an operation icon or the like. This icon or the like functions as a software button. The track pad 14 corresponds to the “detection unit” of the present invention.

FIG. 3 is a perspective view showing a configuration of the image display unit 20 and shows a main configuration of the image display unit 20 as viewed from the user's head side. FIG. 3 shows the side of the image display unit 20 that contacts the user's head, in other words, the side that can be seen by the user's right eye RE and left eye LE. In other words, the back sides of the right light guide plate 26 and the left light guide plate 28 are visible.
In FIG. 3, the half mirror 261 that irradiates the user's right eye RE with the image light and the half mirror 281 that irradiates the left eye LE with the image light appear as substantially rectangular areas. Further, the entire right light guide plate 26 and left light guide plate 28 including the half mirrors 261 and 281 transmit external light as described above. For this reason, the user sees the outside scene through the entire right light guide plate 26 and the left light guide plate 28, and sees a rectangular display image at the positions of the half mirrors 261 and 281.

  The camera 61 is disposed at the right end of the image display unit 20 and images the direction in which both eyes of the user face, that is, the front for the user. The optical axis of the camera 61 is a direction including the line-of-sight directions of the right eye RE and the left eye LE. The outside scene that the user can visually recognize in a state where the HMD 100 is worn is not always at infinity. For example, when the user gazes at an object located in front of the user with both eyes, the distance from the user to the object is often about 30 cm to 10 m, and is about 1 m to 4 m. is more than. Therefore, for the HMD 100, an upper limit and a lower limit of the distance from the user to the object during normal use may be determined. This standard may be obtained by investigation or experiment, or may be set by the user. The optical axis and the angle of view of the camera 61 are included in the angle of view when the distance to the object during normal use corresponds to the set upper limit guide and the lower limit guide. It is preferable to set as follows.

  In general, the viewing angle of a human is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Among them, the effective field of view with excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. Furthermore, the stable gaze field in which the gaze point that the human gazes at appears to be quickly and stably is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. When the gazing point is an object positioned in front of the user, the effective visual field is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction around the line of sight of each of the right eye RE and the left eye LE. It is. Further, the stable viewing field is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction, and the viewing angle is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Furthermore, the actual visual field that the user sees through the right light guide plate 26 and the left light guide plate 28 can be referred to as a real field of view (FOV: Field Of View). In the configuration of the present embodiment shown in FIGS. 1 and 2, the actual visual field corresponds to an actual visual field that is transmitted through the right light guide plate 26 and the left light guide plate 28 and visually recognized by the user. The real field of view is narrower than the viewing angle and stable field of view, but wider than the effective field of view.

  The field angle of the camera 61 is preferably capable of capturing an image in a wider range than the user's field of view, and specifically, the field angle is preferably at least wider than the user's effective field of view. Further, it is more preferable that the angle of view is wider than the actual field of view of the user. More preferably, the angle of view is wider than the stable viewing field of the user, and most preferably, the angle of view is wider than the viewing angle of both eyes of the user.

  The camera 61 may include a so-called wide-angle lens as an imaging lens so that a wide angle of view can be captured. The wide-angle lens may include a lens called a super-wide-angle lens or a quasi-wide-angle lens, may be a single focus lens or a zoom lens, and the camera 61 includes a lens group including a plurality of lenses. There may be.

FIG. 4 is a block diagram illustrating a configuration of each unit included in the HMD 100.
The control device 10 includes a main processor 140 that executes a program and controls the HMD 100. A non-volatile storage unit 121 and a memory 123 are connected to the main processor 140. The main processor 140 is connected to the track pad 14 and the operation unit 110 as input devices. In addition, a 6-axis sensor 111 and a magnetic sensor 113 are connected to the main processor 140 as sensors. The main processor 140 is connected to a GPS receiver 115, a communication unit 117, a beacon receiver 119, an audio codec 180, an external connector 184, an external memory interface 186, a USB connector 188, a sensor hub 192, and an FPGA 194. These function as an interface with the outside. The beacon receiving unit 119 corresponds to the “receiving unit” of the present invention.

  The main processor 140 is mounted on the controller board 120 built in the control device 10. In addition to the main processor 140, a memory 123, a beacon receiving unit 119, a nonvolatile storage unit 121, and the like may be mounted on the controller board 120. In the present embodiment, a six-axis sensor 111, a magnetic sensor 113, a GPS receiver 115, a communication unit 117, a memory 123, a nonvolatile storage unit 121, an audio codec 180, and the like are mounted on the controller board 120. Further, the external connector 184, the external memory interface 186, the USB connector 188, the sensor hub 192, the FPGA 194, and the interface 196 may be mounted on the controller board 120.

  The memory 123 constitutes a work area that temporarily stores a program to be executed and data to be processed when the main processor 140 executes the program. The non-volatile storage unit 121 is configured by a flash memory or an eMMC (embedded Multi Media Card). The nonvolatile storage unit 121 stores a program executed by the main processor 140 and various data processed by the main processor 140 executing the program.

The main processor 140 detects a touch operation on the operation surface of the track pad 14 based on an operation signal input from the track pad 14 and acquires an operation position.
The operation unit 110 includes a button 11 and an LED display unit 17. The operation unit 110 outputs an operation signal corresponding to the operated operation element to the main processor 140 when an operation element such as a button or a switch included in the button 11 is operated.

  The LED display unit 17 controls the turning on and off of the LED indicator 12 according to the control of the main processor 140. Further, the LED display unit 17 may have a configuration including an LED (not shown) arranged immediately below the track pad 14 (FIG. 1) and a drive circuit for lighting the LED. In this case, the LED display unit 17 turns on, blinks, and turns off the LED according to the control of the main processor 140.

The six-axis sensor 111 is a motion sensor (inertial sensor) including a three-axis acceleration sensor and a three-axis gyro (angular velocity) sensor, and detects a user's operation as an operation. The 6-axis sensor 111 may employ an IMU (Internal Measurement Unit) in which the above sensors are modularized.
The magnetic sensor 113 is, for example, a triaxial geomagnetic sensor.
The 6-axis sensor 111 and the magnetic sensor 113 output the detection value to the main processor 140 according to a sampling period specified in advance. The 6-axis sensor 111 and the magnetic sensor 113 output detection values to the main processor 140 at a timing designated by the main processor 140 in response to a request from the main processor 140.

  The GPS receiving unit 115 includes a GPS antenna (not shown) and receives a GPS signal transmitted from a GPS satellite. The GPS receiving unit 115 outputs the received GPS signal to the main processor 140. Further, the GPS receiving unit 115 measures the signal strength of the received GPS signal and outputs it to the main processor 140. Information such as received signal strength (RSSI: Received Signal Strength Indication), electric field strength, magnetic field strength, signal to noise ratio (SNR) can be used as the signal strength.

  The communication unit 117 performs wireless communication with an external device. The communication unit 117 includes an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like, or a device in which these are integrated. The communication unit 117 performs wireless communication complying with standards such as Bluetooth and wireless LAN (including Wi-Fi), for example.

  The beacon receiving unit 119 receives a beacon signal transmitted from a beacon signal transmission device (not shown). The beacon receiving unit 119 detects a beacon ID included in the received beacon signal. The beacon ID is identification information of the transmission device. Further, the beacon receiving unit 119 detects the received signal strength indication (RSSI) of the received beacon signal, and estimates the distance from the transmission device based on the detected radio field strength. As the beacon signal, for example, a signal conforming to a short-range wireless communication standard such as NFC (Near Field Communication), Bluetooth, Bluetooth Low Energy, Bluetooth Smart, or iBeacon (registered trademark) is used. The beacon receiving unit 119 corresponds to the “distance detection unit” of the present invention.

  The audio interface 182 is an interface for inputting and outputting audio signals. In the present embodiment, the audio interface 182 includes a connector 46 (FIG. 1) provided on the connection cable 40. The connector 46 is connected to the headset 30. The audio signal output from the audio interface 182 is input to the right earphone 32 and the left earphone 34, whereby the right earphone 32 and the left earphone 34 output audio. Further, the microphone 63 provided in the headset 30 collects sound and outputs a sound signal to the sound interface 182. An audio signal input from the microphone 63 to the audio interface 182 is input to the external connector 184.

  The audio codec 180 is connected to the audio interface 182 and encodes and decodes an audio signal input / output via the audio interface 182. The audio codec 180 may include an A / D converter that converts analog audio signals to digital audio data, and a D / A converter that performs the reverse conversion. For example, the HMD 100 according to the present embodiment outputs sound by the right earphone 32 and the left earphone 34 and collects sound by the microphone 63. The audio codec 180 converts the digital audio data output from the main processor 140 into an analog audio signal and outputs the analog audio signal via the audio interface 182. The audio codec 180 converts an analog audio signal input to the audio interface 182 into digital audio data and outputs the digital audio data to the main processor 140.

  The external connector 184 is a connector that connects an external device that communicates with the main processor 140. The external connector 184 is an interface for connecting an external device when, for example, an external device is connected to the main processor 140 to debug a program executed by the main processor 140 or collect a log of the operation of the HMD 100. It is.

  The external memory interface 186 is an interface to which a portable memory device can be connected. For example, the external memory interface 186 includes a memory card slot in which a card type recording medium is mounted and data can be read and an interface circuit. In this case, the size, shape and standard of the card type recording medium are not limited and can be changed as appropriate.

  The USB (Universal Serial Bus) connector 188 includes a connector conforming to the USB standard and an interface circuit. The USB connector 188 can connect a USB memory device, a smartphone, a computer, or the like. The size and shape of the USB connector 188 and the compatible USB standard version can be appropriately selected and changed.

  The sensor hub 192 and the FPGA 194 are connected to the image display unit 20 via an interface (I / F) 196. The sensor hub 192 acquires detection values of various sensors included in the image display unit 20 and outputs them to the main processor 140. The FPGA 194 executes processing of data transmitted / received between the main processor 140 and each unit of the image display unit 20 and transmission via the interface 196.

  The HMD 100 includes a vibrator 19. The vibrator 19 includes a motor and an eccentric rotor (both not shown), and may have other necessary configurations. The vibrator 19 generates vibrations by rotating the motor according to the control of the main processor 140. For example, the HMD 100 generates vibration by the vibrator 19 with a predetermined vibration pattern when an operation on the operation unit 110 is detected, when the power of the HMD 100 is turned on / off, or in other cases.

  The right display unit 22 and the left display unit 24 of the image display unit 20 are each connected to the control device 10. As shown in FIG. 1, in the HMD 100, a connection cable 40 is connected to the left holding unit 23, and wirings connected to the connection cable 40 are laid inside the image display unit 20, and each of the right display unit 22 and the left display unit 24. Is connected to the control device 10.

The right display unit 22 includes a display unit substrate 210. On the display unit substrate 210, an interface (I / F) 211 connected to the interface 196, a receiving unit (Rx) 213 that receives data input from the control device 10 via the interface 211, and an EEPROM 215 are mounted. .
The interface 211 connects the receiving unit 213, the EEPROM 215, the temperature sensor 217, the camera 61, the illuminance sensor 65, and the LED indicator 67 to the control device 10.

  An EEPROM (Electrically Erasable Programmable Read-Only Memory) 215 stores various data so that the main processor 140 can read the data. The EEPROM 215 stores, for example, data relating to light emission characteristics and display characteristics of the OLED units 221 and 241 provided in the image display unit 20, data relating to characteristics of sensors provided in the right display unit 22 or the left display unit 24, and the like. Specifically, parameters relating to gamma correction of the OLED units 221, 241 and data for compensating the detection values of the temperature sensors 217, 239 are stored. These data are generated by the factory inspection of the HMD 100 and written in the EEPROM 215. After the shipment, the main processor 140 can perform processing using the data in the EEPROM 215.

The camera 61 executes imaging according to a signal input via the interface 211 and outputs captured image data or a signal indicating the imaging result to the control device 10.
As shown in FIG. 1, the illuminance sensor 65 is provided at an end ER of the front frame 27 and is arranged to receive external light from the front of the user wearing the image display unit 20. The illuminance sensor 65 outputs a detection value corresponding to the amount of received light (received light intensity).
As shown in FIG. 1, the LED indicator 67 is disposed near the camera 61 at the end ER of the front frame 27. The LED indicator 67 is lit during execution of imaging by the camera 61 to notify that imaging is in progress.

  The temperature sensor 217 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 217 is mounted on the back side of the OLED panel 223 (FIG. 2). For example, the temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223.

  The receiving unit 213 receives data transmitted from the main processor 140 via the interface 211. When the image data to be displayed (referred to as image data) is displayed by the OLED unit 221, the reception unit 213 outputs the received image data to the OLED drive circuit 225 (FIG. 2).

The left display unit 24 includes a display unit substrate 210. On the display unit substrate 210, an interface (I / F) 231 connected to the interface 196 and a receiving unit (Rx) 233 that receives data input from the control device 10 via the interface 231 are mounted. In addition, a six-axis sensor 235 and a magnetic sensor 237 are mounted on the display unit substrate 210.
The interface 231 connects the receiving unit 233, the six-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 to the control device 10.

The 6-axis sensor 235 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 235 may employ an IMU in which the above sensors are modularized. The 6-axis sensor 235 corresponds to the “detection unit” of the present invention.
The magnetic sensor 237 is, for example, a triaxial geomagnetic sensor.

The temperature sensor 239 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 239 is mounted on the back side of the OLED panel 243 (FIG. 2). The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243.
Further, the temperature sensor 239 may be incorporated in the OLED panel 243 or the OLED drive circuit 245. The substrate may be a semiconductor substrate. Specifically, when the OLED panel 243 is mounted as an Si-OLED as an integrated circuit on an integrated semiconductor chip together with the OLED drive circuit 245 and the like, the temperature sensor 239 may be mounted on the semiconductor chip.

The camera 61, the illuminance sensor 65, the temperature sensor 217, and the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 included in the left display unit 24 are connected to the sensor hub 192. The sensor hub 192 sets and initializes the sampling period of each sensor according to the control of the main processor 140. The sensor hub 192 executes energization to each sensor, transmission of control data, acquisition of a detection value, and the like in accordance with the sampling period of each sensor. The sensor hub 192 outputs detection values of the sensors included in the right display unit 22 and the left display unit 24 to the main processor 140 at a preset timing. The sensor hub 192 may have a function of temporarily holding the detection value of each sensor in accordance with the output timing to the main processor 140. In addition, the sensor hub 192 may have a function of converting the data of the output value of each sensor or the data format into data of a unified data format and outputting the data to the main processor 140.
The sensor hub 192 starts and stops energization of the LED indicator 67 according to the control of the main processor 140, and lights or blinks the LED indicator 67 in accordance with the timing when the camera 61 starts and ends imaging.

  The control device 10 includes a power supply unit 130 and operates with power supplied from the power supply unit 130. The power supply unit 130 includes a rechargeable battery 132 and a power supply control circuit 134 that detects the remaining capacity of the battery 132 and controls charging of the battery 132. The power control circuit 134 is connected to the main processor 140 and outputs a detected value of the remaining capacity of the battery 132 or a detected value of the voltage to the main processor 140. Further, the power may be supplied from the control device 10 to the image display unit 20 based on the power supplied from the power supply unit 130. The power supply state from the power supply unit 130 to each unit of the control device 10 and the image display unit 20 may be configured to be controllable by the main processor 140.

  FIG. 5 is a functional block diagram of the storage unit 125 and the control unit 150 that constitute the control system of the control device 10. The storage unit 125 illustrated in FIG. 5 is a logical storage unit configured by the nonvolatile storage unit 121 (FIG. 4), and may include an EEPROM 215. The control unit 150 and various functional units included in the control unit 150 are formed by cooperation of software and hardware by the main processor 140 executing a program. Each functional unit constituting the control unit 150 and the control unit 150 includes, for example, a main processor 140, a nonvolatile storage unit 121, and a memory 123.

The control unit 150 executes various processes using data stored in the storage unit 125 and controls the HMD 100.
The storage unit 125 stores various data processed by the control unit 150. The storage unit 125 stores setting data 127 and content data 128.
The setting data 127 includes various setting values that set the operation of the HMD 100. Further, when the control unit 150 uses the parameters, determinant, arithmetic expression, LUT (Look Up Table), or the like when controlling the HMD 100, these may be included in the setting data 127.

  The content data 128 is content data displayed by the image display unit 20 under the control of the control unit 150, and includes, for example, still image data and moving image data. Further, the content data 128 may include audio data. Further, the content data 128 may include data of a plurality of images. In this case, the plurality of images are not limited to images displayed on the image display unit 20 at the same time.

  The content data 128 is an interactive content in which when the content is displayed on the image display unit 20, the control device 10 receives a user operation and the control unit 150 executes a process according to the received operation. It may be. In this case, the content data 128 may include image data of a menu screen that is displayed when an operation is accepted, data that defines processing corresponding to items included in the menu screen, and the like.

The control unit 150 has functions of an operating system (OS) 143, an image processing unit 145, an imaging control unit 147, an operation control unit 149, a detection control unit 151, and a display control unit 153. The operation control unit 149 and the detection control unit 151 correspond to the “detection unit” of the present invention.
The functions of the operating system 143 are functions of a control program stored in the storage unit 125, and the other units are functions of application programs executed on the operating system 143.

  The image processing unit 145 generates a signal to be transmitted to the right display unit 22 and the left display unit 24 based on the image data displayed by the image display unit 20. The signal generated by the image processing unit 145 may be a vertical synchronization signal, a horizontal synchronization signal, a clock signal, an analog image signal, or the like.

Further, the image processing unit 145 may perform resolution conversion processing for converting the resolution of the image data to a resolution suitable for the right display unit 22 and the left display unit 24 as necessary. The image processing unit 145 also performs image adjustment processing for adjusting the brightness and saturation of image data, 2D image data from 3D image data, 2D / 3D conversion processing for generating 3D image data from 2D image data, and the like. May be executed. When these image processes are executed, the image processing unit 145 generates a signal for displaying an image based on the processed image data and transmits the signal to the image display unit 20.
The image processing unit 145 may be configured by hardware different from the main processor 140 (for example, a digital signal processor (DSP)), in addition to the configuration realized by the main processor 140 executing a program.

  The imaging control unit 147 controls the camera 61 to execute imaging, generates captured image data, and temporarily stores it in the storage unit 125. When the camera 61 is configured as a camera unit including a circuit that generates captured image data, the imaging control unit 147 acquires captured image data from the camera 61 and temporarily stores the captured image data in the storage unit 125.

  The operation control unit 149 detects an operation on the track pad 14 and the operation unit 110 and outputs operation data corresponding to the operation. For example, when a button or the like of the operation unit 110 is operated, the operation control unit 149 generates operation data indicating the operation content and outputs the operation data to the display control unit 153. The display control unit 153 changes the display state according to the operation data input from the operation control unit 149.

  Further, the operation control unit 149 acquires the coordinates of the operation position on the track pad 14 when the operation on the track pad 14 is detected. The operation control unit 149 generates a locus of the operation position. The locus of the operation position is a locus of the operation position while the touch operation of the track pad 14 is not released, that is, while the state where the user's hand or finger operation body is in contact with the track pad 14 continues. The operation control unit 149 outputs operation data indicating the locus of the detected operation position to the display control unit 153. The operation control unit 149 corresponds to the “detection unit” of the present invention.

  The detection control unit 151 detects an operation based on the detection value of the 6-axis sensor 235. Specifically, the detection control unit 151 detects the movement of the user's head as an operation. The 6-axis sensor 235 performs sampling (measurement) every predetermined cycle (for example, 50 msec), and outputs an analog voltage value indicating the measured acceleration and angular velocity values.

The 6-axis sensor 235 measures the acceleration in the X-axis direction, the acceleration in the Y-axis direction, and the acceleration in the Z-axis direction shown in FIG. Further, the 6-axis sensor 235 determines the angular velocity of rotation about the X axis, the angular velocity of rotation about the Y axis, and the angular velocity of rotation about the Z axis shown in FIG. 1 at the measurement reference point of the built-in measurement mechanism (not shown). taking measurement. The X-axis, Y-axis, and Z-axis are three directions orthogonal to each other as shown in FIG. 1, the Z-axis direction corresponds to the vertical direction, and the X-axis direction is the left-right direction of the user's head. Correspondingly, the Y-axis direction corresponds to the front-rear direction of the user's head. The Z-axis direction corresponding to the vertical direction corresponds to the direction of the user's body axis.
The analog voltage value output from the 6-axis sensor 235 is converted into a digital voltage value by an A / D converter (not shown) and input to the control unit 150.

6 is a diagram showing rotation around the Z axis, which is the body axis of the user's head, FIG. 7 is a diagram showing rotation around the Y axis of the user's head, and FIG. It is a figure which shows rotation around the X-axis of a user's head.
The detection control unit 151 detects a roll angle that is a rotation around the Z axis (body axis) based on a detection value of the 6-axis sensor 235, that is, a movement of the user rotating the head (face) to the left or right. To do. In addition, the detection control unit 151 tilts the head in the left direction (left shoulder direction) or the right direction (right shoulder direction) by the yaw angle that is rotation about the Y axis based on the detection value of the 6-axis sensor 235. The movement (the tilt of the head relative to the body axis) is detected. In addition, the detection control unit 151 detects a pitch angle that is rotation about the X axis based on a detection value of the 6-axis sensor 235, that is, a movement in which the user tilts the head in the forward direction (gaze direction) or the backward direction (relative to the body axis). Head tilt). Hereinafter, the movement of the user turning the head leftward or rightward and the movement of tilting the head leftward, rightward, upward, or downward are collectively referred to as “head movement”.
Based on the detection value of the 6-axis sensor 235, the detection control unit 151 generates motion data indicating the direction in which the head motion is detected and the angle of the moved head, and outputs the motion data to the display control unit 153.

  The display control unit 153 generates a control signal for controlling the right display unit 22 and the left display unit 24 and outputs the control signal to the right display unit 22 and the left display unit 24. Specifically, the display control unit 153 controls the timing at which the OLED drive circuits 225 and 245 draw an image on the OLED panels 223 and 243 based on the signal output from the image processing unit 145, and the brightness of the OLED panels 223 and 243. A control signal for controlling is generated.

Further, the display control unit 153 moves the position of the pointer 301 (FIG. 9) displayed on the display area 310 according to the operation data input from the operation control unit 149. A pointer 301 indicates a designated position in the display area 310. Further, instead of the pointer 301, a cursor, a mark, a character, a symbol, a sign, an arrow, or a game character can be used as a marker. The pointer 301 corresponds to the “marker” of the present invention.
The display control unit 153 receives data indicating the movement distance on the operation surface and the movement direction as operation data from the operation control unit 149. The display control unit 153 generates a control signal indicating a moving direction and a moving distance for moving the pointer 301 based on the input operation data, and outputs the generated control signal to the right display unit 22 and the left display unit 24. To do.

  Further, the display control unit 153 generates a control signal for moving the position of the pointer 301 displayed in the display area 310 based on the motion data input from the detection control unit 151. The display control unit 153 receives data indicating the direction in which the movement of the head is detected and the angle of the head as movement data from the detection control unit 151. The display control unit 153 determines a moving direction and a moving distance for moving the pointer 301 in the display area 310 based on the input motion data. The display control unit 153 generates a control signal indicating the determined moving direction and moving distance, and outputs the control signal to the right display unit 22 and the left display unit 24.

In addition, the display control unit 153 performs calibration and generates a parameter that associates the coordinates of the display area 310 with the captured image of the camera 61.
The display control unit 153 recognizes and tracks the reference real object corresponding to the calibration image from the captured image data of the camera 61 while displaying the calibration image on the image display unit 20. In this state, the user moves the position of the calibration image on the image display unit 20 via the user interface such as the track pad 14. Then, at the timing when the user perceives that the calibration image and the reference real object overlap (at least one of the position, size, and orientation substantially coincides), the display control unit 153 is connected to the display control unit 153 via the user interface. Notice. In accordance with the acquired notification, the display control unit 153 sets a parameter that associates the detection result of the reference real object (position on the captured image data) at the timing with the display position of the calibration image in the display area 310. Generate. The display control unit 153 stores the generated parameter in the storage unit 125.
This parameter is used when the target object 305 selected by the pointer 301 is specified. The control unit 150 acquires captured image data of the camera 61, converts the coordinates of the display area 310 on which the pointer 301 is displayed, into coordinates on the captured image data, and identifies the target object 305.

  The HMD 100 may include an interface (not shown) for connecting various external devices that are content supply sources. For example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface may be used, or a wireless communication interface may be used. The external device in this case is an image supply device that supplies an image to the HMD 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used. In this case, the HMD 100 can output images and sounds based on the content data 128 input from these external devices.

FIG. 9 is a diagram showing the field of view 300 of the user wearing the image display unit 20.
The user includes a person existing in the real world (hereinafter referred to as an outside scene), a building, a road, and other objects (hereinafter collectively referred to as a target object 305) and an image displayed by the image display unit 20 ( (Hereinafter referred to as a display image) through the image display unit 20. The display image includes, for example, a pointer 301 and a still image and a moving image that are displayed by reproducing the content data 128. A display area 310 indicates an area in which the image display unit 20 can display a display image.

  The user performs a first operation and a second operation on the HMD 100 to move the pointer 301 to a target position. For example, when the user intends to purchase a beverage sold by the vending machine 305A as the target object 305, the target position is a position overlapping the product button of the product of the vending machine 305A. The target position is hereinafter referred to as a target.

  The HMD 100 according to the present embodiment moves the pointer 301 in a different manner depending on whether the position of the pointer 301 is moved according to the first operation or when the position of the pointer 301 is moved according to the second operation. For example, when the pointer 301 is moved according to the first operation and when the pointer 301 is moved according to the second operation, the amount of movement of the pointer 301 (for example, movement distance, movement speed, acceleration, etc.) is changed. change.

It is assumed that the distance for moving the pointer 301 by one operation is set to be small or the moving speed is set to be slow. In this case, if the distance between the position of the pointer 301 and the target is long, it takes time to move the pointer 301 to the vicinity of the target. Further, it is assumed that the distance for moving the pointer 301 by one operation is set to be large or the moving speed is set to be fast. In this case, fine adjustment to align the pointer 301 with the target position may be difficult. The target may be a display image displayed by the image display unit 20 or the target object 305 visually recognized as an outside scene.
For this reason, the HMD 100 of the present embodiment moves the pointer 301 to the target by two operations of the first operation and the second operation in which at least one of the moving distance, moving speed, and acceleration of the pointer 301 is different.

The first operation is performed before the second operation. The first operation is an operation for moving the pointer 301 to the vicinity of the target in a short time. The second operation is an operation for finely adjusting the position of the pointer 301 and moving the pointer 301 to the target position.
For example, it is assumed that the first operation and the second operation are pressing operations of switches and buttons. In this case, when the pressing operation by the first operation is accepted once, the movement distance, the moving speed, or the acceleration for moving the pointer 301 is moved. When the pressing operation by the second operation is accepted once, the pointer 301 is moved by the movement. Set larger than distance, moving speed or acceleration. The movement distance, the movement speed, or the acceleration corresponds to the “movement amount” of the present invention.

  Further, it is assumed that the first operation and the second operation are contact operations of the user's finger with respect to the operation surface, and are operations for moving the pointer 301 according to the detected contact distance on the operation surface. In this case, the movement distance, movement speed, or acceleration of the pointer 301 corresponding to the contact distance as the first operation is larger or faster than the movement distance, movement speed, or acceleration of the pointer 301 corresponding to the contact distance as the second operation. Set.

  In addition, the first operation and the second operation are a touch operation on the operation surface or an operation of moving the pointer 301 by detecting the movement of the user's head, and detecting the contact operation or the movement of the head. It is assumed that the operation is to move the pointer 301 by a distance corresponding to time. In this case, the moving distance, moving speed, or acceleration for moving the pointer 301 corresponding to the detection time of the first operation is obtained from the moving distance, moving speed, or acceleration for moving the pointer 301 corresponding to the detection time of the second operation. Also set larger or faster.

  In the present embodiment, an operation for moving the pointer 301 in accordance with the operation of the track pad 14 is a first operation. When the pointer 301 is moved to the target position by operating the track pad 14, it is not easy to move the pointer 301 as the user imagines if the track pad 14 is not fixed. For this reason, in this embodiment, the operation of the track pad 14 is used as an operation of moving the pointer 301 in the vicinity of the target.

  In the present embodiment, the second operation is an operation in which the movement of the user's head is detected as an operation and the pointer 301 is moved in accordance with the detected movement. The second operation is an operation for moving the position of the pointer 301 with the position of the pointer 301 moved by the first operation as a starting point. The operation of the pointer 301 performed by the user moving the head corresponds to the “user operation” of the present invention. For example, when the user performs an action of tilting the head as a movement of the head, the user must continue to tilt the head until the operation is completed, and must maintain an unreasonable posture was there. However, in the present embodiment, since the pointer 301 has already been moved in the vicinity of the target by the first operation performed before the second operation, the time for performing the second operation can be shortened.

9 corresponds to the horizontal direction of the display area 310, the y axis corresponds to the depth direction of the display area 310, and the z axis corresponds to the vertical direction of the display area 310.
When the rotation about the X axis of the user's head shown in FIG. 1, that is, the movement of tilting the head back and forth with respect to the body axis, is detected by the control unit 150, for example, the pointer 301 is shown in FIG. It is moved in the z-axis direction, that is, in the vertical direction of the display area 310.
Further, when the rotation of the user's head around the Y axis shown in FIG. 1, that is, the movement of tilting the head to the left or right with respect to the body axis is detected, the control unit 150 sets the pointer 301 to, for example, FIG. Are moved in the y-axis direction shown in FIG.
Further, when the rotation of the user's head around the Z axis shown in FIG. 1, that is, the movement of rotating the head with respect to the body axis is detected, the control unit 150, for example, sets the pointer 301 in FIG. It is moved in the x-axis direction shown, that is, in the horizontal direction of the display area 310.

  The association between the direction in which the user tilts the head and the movement direction in which the pointer 301 is moved is not limited to that described above, and can be arbitrarily set. For example, when the rotation of the user's head around the Y axis shown in FIG. 7, that is, the movement of tilting the head left and right with respect to the body axis is detected, the control unit 150 sets the pointer 301 in FIG. You may make it move to the x-axis direction shown, ie, the horizontal direction of the display area 310. FIG. Further, the control unit 150 sets the pointer 301 in FIG. 9 when the rotation of the user's head shown in FIG. 8 around the X axis, that is, the movement of tilting the head back and forth with respect to the body axis is detected. You may make it move to the y-axis direction shown, ie, the depth direction of the display area 310. FIG.

In the present embodiment, the timing for switching the operation for moving the pointer 301 from the first operation to the second operation is detected whether or not the user's hand or finger (hereinafter referred to as a finger) has moved away from the track pad 14. Do that. The control unit 150 determines that the user intends to operate the track pad 14 while detecting contact with the operation surface of the track pad 14, and moves the pointer 301 by operating the track pad 14. In addition, when the contact with the operation surface cannot be detected and the finger is separated from the operation surface, the control unit 150 determines that the willingness to operate the track pad 14 is lost, and performs an operation for moving the pointer 301 as the second operation. Switch to.
Further, switching from the first operation to the second operation is not limited to whether or not contact with the operation surface is detected. For example, when the preset button 11, the up / down key 15, the changeover switch 16 is operated, or when an operation of tapping the track pad 14 for a preset number of times is accepted, the first operation is switched to the second operation. May be. In addition, the first operation may be switched to the second operation by sound input from the microphone 63. For example, when a finger snap sound or a word uttered by the user is input from the microphone 63 and the input voice data matches the preset voice data, the first operation may be switched to the second operation. .

In the present embodiment, the operation of the pointer 301 by the track pad 14 is the first operation, and the operation of the pointer 301 based on the movement of the user's head is the second operation. However, the first operation and the second operation are these operations. It is not limited to the operation of. For example, the operation of detecting the movement of the user's head and moving the pointer 301 may be the first operation, and the operation of the pointer 301 by the track pad 14 may be the second operation. Moreover, you may use a hand sign and a gesture for any one of 1st operation and 2nd operation. The hand sign and the gesture correspond to the “user action” of the present invention. When the first or second operation is performed by hand sign or gesture, a dictionary used for specifying the hand sign or gesture is stored in the storage unit 125 in advance. For example, data indicating the position and shape of the finger is registered in the dictionary for each hand sign. In addition, if the gesture is to perform preset hand shapes (forms) in the preset order, data indicating the finger position and shape in each shape (form) is registered in the dictionary for each gesture. The The control unit 150 detects a region where the user's hand or finger is imaged (hereinafter referred to as a finger region) from the captured image of the camera 61, detects the shape or position of the finger from the detected finger region, and stores it in the dictionary. Recognize hand signs and gestures by matching with registered data.
In addition, the control unit 150 detects the movement of the fingernail of the user's hand, the ring fitted on the finger of the user, the bracelet fitted on the arm, the watch, or the band as one of the first operation and the second operation. May be. The control unit 150 detects movements of the user's fingernails, a ring fitted on the finger of the user, a bracelet fitted on the arm, a watch, and a band from the captured image of the camera 61, and according to the detected movement. The pointer 301 is moved.

In addition, the control unit 150 includes a first operation mode and a second operation mode as operation modes, and switches between these operation modes for execution.
The first operation mode is a mode in which the pointer 301 is moved corresponding to the outside scene that can be seen through the image display unit 20. More specifically, in the first operation mode, the control unit 150 moves the pointer 301 with reference to the target object 305 existing in the outside scene seen through the image display unit 20. For example, when the user tilts the head by the second operation, the display area 310 tilts corresponding to the tilt of the head, but the target object 305 existing in the outside scene does not tilt. When the operation mode is the first operation mode, when an operation for moving the pointer 301 to the left or right is input, the control unit 150 moves the pointer 301 in a horizontal direction relative to the ground (hereinafter simply referred to as a horizontal direction). To the left or right). In addition, when the operation mode is the first operation mode, when an operation for moving the pointer 301 up or down is input, the control unit 150 moves the pointer 301 in the vertical direction upward or downward.

In the second operation mode, the pointer 301 is moved in a manner different from the movement corresponding to the outside scene.
More specifically, the control unit 150 moves the pointer 301 with reference to the display image displayed in the display area 310 in the second operation mode. When the user tilts the head by the second operation, the display area 310 also tilts in accordance with the tilt of the head. Therefore, when an operation for moving the pointer 301 to the left or right is input when the operation mode is the second operation mode, the control unit 150 moves the pointer 301 in the horizontal direction of the tilted display area 310. Move in a direction parallel to the axis. In addition, when the operation mode is the second operation mode and the operation for moving the pointer 301 up or down is input, the control unit 150 moves the pointer 301 to the z axis that is the vertical direction of the tilted display area 310. Move in a direction parallel to.

In addition, when the operation mode is the first operation mode, the control unit 150 controls the movement of the pointer 301 so that the pointer 301 can be easily aligned with the target object 305. In addition, when the operation mode is the second operation mode, the control unit 150 controls the movement of the pointer 301 so that the pointer 301 can be easily aligned with the display image.
For example, when the operation mode is the first operation mode, the control unit 150 detects the target object 305 from the captured image data of the camera 61 and identifies the target object 305 that overlaps the display position of the pointer 301. For example, when the target object 305 and the display image displayed superimposed on the target object 305 overlap at the display position of the pointer 301, the control unit 150 identifies the target object 305 that overlaps the display position of the pointer 301. .
In addition, when the operation mode is the second operation mode, the control unit 150 displays a plurality of display images displayed in the display area 310 or a display image overlapping the position of the pointer 301 from the target object 305 visually recognized in the display area 310. Is identified. For example, when the target object 305 and the display image displayed superimposed on the target object 305 overlap at the display position of the pointer 301, the control unit 150 specifies the display image that overlaps the display position of the pointer 301.

10 and 11 are explanatory diagrams for explaining the field of view of the user wearing the image display unit 20. In particular, FIGS. 10 and 11 show a state in which the vending machine 305A can be seen as the target object 305 included in the outside scene.
When the user operates the pointer 301 by the second operation, the user may perform an operation of tilting the head to the left shoulder side or the right shoulder side (see FIG. 7). When the user rotates the head around the Y axis and tilts the head toward the left shoulder, the display area 310 that the user visually recognizes also tilts according to the tilt of the head and is displayed in the display area 310. The display image is also tilted. In this case, the display area 310 of the image display unit 20 is also tilted according to the tilt of the head, but the target object 305 seen through the image display unit 20 is not tilted.

  For example, when the user tilts his / her head toward the left shoulder, the display area 310 tilts so that the left side is lowered and the right side is viewed from the user. At this time, when the operation mode is the first operation mode, the control unit 150 does not move the pointer 301 in the horizontal direction of the tilted display area 310 but moves the pointer 301 in the horizontal direction. The control unit 150 detects the tilt of the head based on the detection value of the 6-axis sensor 235, and corrects the moving direction of the pointer 301 based on the detected tilt of the head. FIG. 10 shows a state in which the pointer 301 is moved in the horizontal direction with respect to the vending machine 305 </ b> A as the target object 305.

In addition, after returning the tilt of the tilted head, the user moves the head 301 back and forth with respect to the body axis, thereby moving the pointer 301 in the vertical direction of the display area 310 upward. Can be moved to.
In addition, when the user wants to move the pointer 301 in the z-axis direction of the display area 310 in a state where the user tilts his / her head toward the left shoulder side or the right shoulder side and is viewed with the display area 310 tilted, The operation by the track pad 14 may be performed simultaneously with the operation by the unit. When the operation of tilting the head to the left shoulder side or the right shoulder side is detected, for example, when the operation of moving the pointer 301 in the vertical direction by the operation of the track pad 14 is received, the control unit 150 performs both operations. In response, the pointer 301 is moved. That is, the control unit 150 moves the pointer 301 simultaneously in the horizontal direction and the vertical direction, and moves the pointer 301 obliquely upward as shown in FIG.

FIG. 12 is an explanatory diagram for explaining the field of view of the user wearing the image display unit 20. In particular, FIG. 12 shows a state where a display screen 302 of word processor software is displayed in the display area 310.
When the user's head is tilted, the image display unit 20 attached to the user's head is also tilted. For this reason, the display image displayed by the image display unit 20 is also tilted and displayed. When the operation mode is the second operation mode, the control unit 150 moves the display area 310 in the direction parallel to the x axis, which is the horizontal direction. FIG. 12 shows how the pointer 301 is moved in accordance with the inclination of the display screen 302 of the word processor software.

Further, a display indicating whether the operation mode of the HMD 100 is the first operation mode or the second operation mode may be displayed on the display area 310.
For example, when the operation mode of the HMD 100 is the first operation mode, “first operation mode” is displayed in the display area 310, and when the operation mode is the second operation mode, “second operation mode” is displayed in the display area 2310. “Mode” may be displayed.
Further, the color of the display image may be changed depending on whether the operation mode is the first operation mode or the second operation mode.

13 and 14 are explanatory diagrams for explaining the field of view of the user wearing the image display unit 20. In particular, FIG. 13 is a diagram illustrating the moving direction of the pointer 301 when the operation mode is the first operation mode. FIG. 14 is a diagram illustrating the moving direction of the pointer 301 when the operation mode is the second operation mode.
When the pointer 301 is in a stationary state, an arrow indicating a direction in which the pointer 301 can move may be displayed as the pointer 301. For example, when the operation mode of the HMD 100 is the first operation mode, the control unit 150 displays arrows indicating four directions, ie, a vertical upward direction, a vertical downward direction, a horizontal left direction, and a horizontal right direction, as shown in FIG. The displayed pointer 301 is displayed. Further, when the operation mode of the HMD 100 is the second operation mode, the control unit 150, as shown in FIG. 14, the arrow indicating the vertical upward and the vertical downward of the display area 310, the horizontal left of the display area 310, and A pointer 301 on which an arrow indicating a horizontal rightward direction is displayed is displayed.
Further, instead of displaying an arrow indicating the moving direction, auxiliary symbols and auxiliary lines indicating the moving direction may be displayed.

15 and 16 are explanatory diagrams for explaining the field of view of the user wearing the image display unit 20. In particular, FIGS. 15 and 16 are diagrams illustrating display images displayed when the pointer 301 is moved to the vicinity of the target by the first operation.
When the pointer 301 is operated by the first operation and the pointer 301 is moved close to the target, the target existing near the pointer 301 is enlarged and displayed as shown in FIG.

The control unit 150 acquires captured image data captured by the camera 61 and detects the target object 305 from the acquired captured image data. When the control unit 150 detects the target object 305 within a preset range from the display position of the pointer 301, the control unit 150 extracts an image of the target object 305 from the captured image data, and enlarges the extracted image of the target object 305. It is displayed on the display area 310.
In addition, when the target object 305 is detected within a preset range from the display position of the pointer 301, the control unit 150 displays a display for inquiring whether or not to enlarge the image of the target object 305 in the display area 310. It may be displayed. For example, when the instruction to enlarge the image of the target object 305 is received by the button 11 or the track pad 14, the control unit 150 enlarges the image of the target object 305 and displays it on the display area 310. An image of the target object 305 displayed in an enlarged manner is referred to as an enlarged image 303.

  The user finely adjusts the position of the pointer 301 by the second operation so that the pointer 301 overlaps the target while the enlarged image 303 is displayed. When the pointer 301 overlaps the target, the user performs a determination operation. When accepting the confirmation operation, control unit 150 determines that the target overlapping the display position of pointer 301 has been selected. For the confirmation operation, an operation other than the second operation for detecting the movement of the user's head and moving the pointer 301 is used. The confirmation operation may be, for example, an operation of hitting the operation surface of the track pad 14, or an operation of the button 11, the up / down key 15, the changeover switch 16, and the like.

Further, as shown in FIG. 16, an image 307 indicating the user may be displayed in the display area 310. As the user image 307, for example, a user image captured in advance, an icon indicating a user, an avatar, or the like can be used.
When the enlarged image 303 is displayed in the display area 310, the user inputs information indicating whether or not the target object 305 displayed as the enlarged image 303 is a target to the HMD 100 by performing a predetermined operation. For this predetermined operation, as an operation showing affirmation, for example, an OK sign performed by a finger of a hand, a movement of rotating the head with respect to the X axis and moving the head in the front-rear direction, that is, a “nodding” operation. included. In addition, for a predetermined operation, as an operation indicating negative, for example, a sign for crossing the toes of both hands in front of the user's face, or rotating the head with respect to the Z axis, Movements, i.e., "swing" movements.
The control unit 150 may display a detection result of a predetermined operation performed by the user in the vicinity of the user image 307. FIG. 16 shows an example in which a pop display 308 is displayed in association with the user image 307 when an operation indicating affirmative is detected as the predetermined operation.

  Further, the HMD 100 may determine that a confirmation operation has been received when a signal transmitted from an external device that is wirelessly connected is received. For example, a wearable terminal on which a user's arm or the like is worn can be used as the external device. When the pointer 301 overlaps the target, the user inputs an operation preset by the wearable terminal. When the wearable terminal receives a preset operation, the wearable terminal transmits a predetermined signal to the HMD 100. When receiving a predetermined signal from the wearable terminal, the control unit 150 of the HMD 100 determines that a confirmation operation has been received.

FIG. 17 is an explanatory diagram for explaining the field of view of the user wearing the image display unit 20. In particular, FIG. 17 is a diagram illustrating a state in which a refrigerated showcase 305B installed in a store can be visually recognized as the target object 305.
The refrigerated showcase 305B is provided with a QR code (registered trademark) 340 as an identification mark corresponding to each shelf of the refrigerated showcase 305B. The refrigerated showcase 305B is provided with a beacon transmitter 345 that transmits a beacon signal. The control unit 150 captures an image of the QR code 340 installed on the shelf on which the product is displayed by the camera 61. In the present embodiment, the case where the identification mark is the QR code 340 will be described. However, the identification mark may be a barcode, an AR marker, or the like.

  The control unit 150 detects the QR code 340 from the captured image data of the camera 61 and displays an image having a predetermined shape (hereinafter referred to as a target image) at a position corresponding to the QR code 340 of each shelf. The camera 61 corresponds to a “mark detection unit” of the present invention. The target image may be an image of any color and shape as long as the user can easily recognize the target image. When the target image is displayed corresponding to the shelf, the user puts the fingertip on the target image of the shelf on which the product to be purchased is displayed. The control unit 150 detects the user's fingertip from the captured image data of the camera 61 and moves the pointer 301 to the position of the target image corresponding to the detected fingertip.

  When the pointer 301 is displayed on the target image of the shelf on which the product to be purchased is displayed, the user moves the pointer 301 to the position of the product to be purchased by operating the track pad 14 or the head. Thereafter, when the control unit 150 receives a preset operation such as an operation of hitting the operation surface of the track pad 14 or a user's nodding operation, the control unit 150 communicates with a server device installed in the store. Then, the purchase processing of the product that is visually recognized by overlapping the pointer 301 is performed.

In the above description, the case where a plurality of QR codes 340 are imaged on the captured image data of the camera 61 has been described. Only the image may be taken. In this case, the control unit 150 moves the pointer 301 to the shelf of the captured QR code 340. Thereafter, the user moves the pointer 301 to the position of the product to be purchased by operating the track pad 14 or the head.
In addition, when a plurality of QR codes 340 are captured in the captured image data of the camera 61, the control unit 150 displays the captured image data in the display area 310 and allows the user to select one of the plurality of QR codes 340. Perform an operation to select. For example, when the captured image data is displayed in the display area 310, the user selects any one QR code 340 by operating the operation unit 110 or operating the head. Upon receiving an operation for selecting the QR code 340, the control unit 150 moves the pointer 301 to the position of the refrigerated showcase 305B to which the selected QR code 340 is attached. Thereafter, the user moves the pointer 301 to the position of the product to be purchased by operating the track pad 14 or the head.

  The control unit 150 also controls the beacon transmitter 345 and a server device (not shown) installed in the store based on information included in the beacon signal received from the beacon transmitter 345 installed on the shelf of the refrigerated showcase 305B. Connect to. The control unit 150 receives additional information transmitted from the connected beacon transmitter 345 and the server device. The control unit 150 displays the received additional information on the display area 310. The additional information includes, for example, discount information such that when a sandwich is purchased together with a drink, a fee discount service can be received.

FIG. 18 is an explanatory diagram for explaining the field of view of the user wearing the image display unit 20. In particular, FIG. 18 shows a state where a display screen 302 of word processor software and a software keyboard 304 are displayed in the user's view 300.
The user moves the pointer 301 on the key of the software keyboard 304 by operating the track pad 14 as the first operation. When the control unit 150 receives an operation on the track pad 14, the control unit 150 moves the display position of the pointer 301 on the display area 310 in accordance with the received operation.

  The software keyboard 304 has a first key screen 304A and a second key screen 304B. The second key screen 304B displays a plurality of keys associated with the keys displayed on the first key screen 304A. On the first key screen 304A, keys “A”, “F”, “K”, “P”, “U”, and “Z” are displayed. For example, when the key “A” on the first key screen 304A is selected, the control unit 150 displays “A”, “B”, “C”, “D”, and “E” as the second key screen 304B. Display each character. When the key “F” on the first key screen 304A is selected, the control unit 150 displays “F”, “G”, “H”, “I”, “J” as the second key screen 304B. Display each character.

  The control unit 150 displays the second key screen 304B associated with the key on which the pointer 301 is positioned when the pointer 301 is positioned on the key of the first key screen 304A for a predetermined time or longer. When the second key screen 304B is displayed in the display area 310, the user moves the head in the direction of the key to be input. For example, in the example shown in FIG. 18, the letter “B” is selected by moving the head upward, the letter “D” is selected by moving the head downward, and the head is moved to the left. The character “C” is selected by pointing toward “”, and the character “E” is selected by pointing the head in the right direction. Further, the character “A” is selected by tilting the head to the left shoulder side or the right shoulder side.

FIG. 19 is an explanatory diagram for explaining the field of view of the user wearing the image display unit 20. In particular, FIG. 19 is a diagram illustrating a field of view 300 of a driver who drives a vehicle (moving object) with the HMD 100 attached.
When the finger detected from the captured image data of the camera 61 indicates the same position for a certain time or longer, the control unit 150 displays the pointer 301 at a position corresponding to the fingertip of the finger. FIG. 19 shows a state where a pointer 301 is displayed at a position corresponding to a vehicle traveling ahead. The display position of the pointer 301 can be finely adjusted by movement of the user's head or voice input by the microphone 63. The control unit 150 inputs a user's voice through the microphone 63, analyzes the input voice, and analyzes an instruction from the user. For example, when the voice analysis result is “rightward”, the control unit 150 moves the display position of the pointer 301 to the right.
Further, the control unit 150 may detect a vehicle as a moving object from the captured image data of the camera 61 and display the pointer 301 at the position of the display area 310 corresponding to the detected vehicle.

  In addition, when the “tracking” instruction is received by voice input in a state where the pointer 301 is displayed at a position corresponding to the vehicle traveling in front, the control unit 150 causes the pointer 301 to follow the vehicle traveling in front.

Further, when a line-of-sight sensor (not shown) is mounted on the HMD 100 and the target object 305 exists in the line-of-sight direction detected by the line-of-sight sensor, the control unit 150 may display the pointer 301. Here, the line-of-sight sensor will be described.
A pair of line-of-sight sensors is provided at the center position of the right light guide plate 26 and the left light guide plate 28 so as to correspond to the right eye RE and the left eye LE of the user. The line-of-sight sensor includes, for example, a pair of cameras that capture images of the user's right eye RE and left eye LE, respectively. The line-of-sight sensor captures images according to the control of the control unit 150, and the control unit 150 detects reflected light and pupil images on the eyeball surfaces of the right eye RE and the left eye LE from the captured image data, and identifies the line-of-sight direction.

  Switching between the first operation mode and the second operation mode may be performed, for example, when an operation of the button 11, the up / down key 15, the changeover switch 16, or the like is received. The operation mode may be switched when an operation of hitting the track pad 14 a preset number of times is received.

Further, the first operation mode and the second operation mode may be switched depending on whether the target object 305 exists in the line-of-sight direction detected by the line-of-sight sensor mounted on the HMD 100 or the display image exists.
When the target object 305 exists in the line-of-sight direction detected by the line-of-sight sensor, the control unit 150 sets the operation mode of the HMD 100 to the first operation mode. In addition, when a display image exists in the line-of-sight direction detected by the line-of-sight sensor, the control unit 150 sets the operation mode of the HMD 100 to the second operation mode.

FIG. 20 is a diagram illustrating the relationship between the distance to the target and the movement distance of the pointer 301.
In the above description, the moving distance, moving speed, and acceleration of the pointer 301 are changed between the case where the pointer 301 is operated by the first operation and the case where the pointer 301 is operated by the second operation.
As other operations, the movement distance, the movement speed, and the acceleration, which are modes of movement of the pointer 301, may be changed according to the distance between the HMD 100 and the target. For example, when the target includes a beacon transmission unit 321 that transmits a beacon signal as illustrated in FIG. 20, the HMD 100 receives a beacon signal transmitted from the target by the beacon reception unit 119. The control unit 150 estimates the distance to the target based on the radio wave intensity of the received beacon signal. The control unit 150 changes the moving distance, moving speed, and acceleration of the pointer 301 that is moved by the first or second operation according to the estimated distance to the target.

In the present embodiment, as shown in FIG. 20, the movement distance of the pointer 301 when the distance between the HMD 100 and the target is a distance A is larger than the movement distance of the pointer 301 when the distance to the target is a distance B. Set smaller. Assume that distance A is greater than the threshold and distance B is less than the threshold.
For example, the control unit 150 compares the distance estimated based on the radio wave intensity of the beacon signal with a preset threshold value, and the distance between the HMD 100 and the target is greater than the distance set as the threshold value. Determine whether it is larger.
When the distance between the HMD 100 and the target is larger (separated) than the threshold value, the control unit 150 detects the operation of the track pad 14 as the first operation, and the pointer 301 of the pointer 301 is detected according to the detected first operation. Move position. In addition, when the distance from the target is larger (separated) from the target, the control unit 150 sets the moving distance of the pointer 301 moved by the first operation as the first distance.
In addition, when the distance from the target is equal to or less than the threshold value (close), the control unit 150 detects the movement of the head, which is the second operation, and moves the position of the pointer 301 according to the detected inclination. In addition, when the distance from the target is equal to or smaller than the threshold value, the control unit 150 sets the distance of the pointer 301 moved by the second operation as the second distance.
The second distance is set to a value larger than the first distance. That is, the moving distance of the pointer 301 is set to be larger as the distance between the HMD 100 and the target is shorter.

  When the HMD 100 and the target are separated from each other, the size of the target (the vending machine 305A or the product button) in the user's field of view 300 is small. If the moving distance of the pointer 301 by the first operation is set to be large, the pointer 301 may be greatly deviated from the target. For this reason, when the distance between the HMD 100 and the target is larger than the threshold value, the moving distance of the pointer 301 is set small. When the HMD 100 and the target are close to each other, the size of the target in the user's field of view 300 is large. For this reason, even if the moving distance of the pointer 301 is set to be large, the pointer 301 is hardly deviated from the target. For this reason, when the distance between the HMD 100 and the target is closer than the threshold value, the moving distance of the pointer 301 is set large.

Further, the moving distance of the pointer 301 may be continuously changed according to the distance between the target and the HMD 100. The control unit 150 estimates the distance to the target based on the radio wave intensity of the beacon signal, and changes the moving distance for moving the pointer 301 by the first or second operation according to the estimated distance. The control unit 150 sets the distance to move the pointer 301 by the first or second operation as the distance from the target is shorter.
In contrast to the above description, the distance by which the pointer 301 is moved by the first or second operation may be set to be smaller as the distance between the HMD 100 and the target is shorter.
In the above description, the first operation is selected when the distance to the target is greater than the threshold value, and the second operation is selected when the distance to the target is less than or equal to the threshold value. In addition to this, the position of the pointer 301 may be moved by any one of the first operation and the second operation. That is, the pointer 301 may be operated by either the first operation or the second operation both when the distance to the target is larger than the threshold and when the distance is less than the threshold.
In the description of FIG. 20, the distance to the HMD 100 is estimated based on the radio wave intensity of the beacon signal transmitted from the target. However, when the HMD 100 includes a stereo camera as the camera 61, the distance from the target to the target is determined by the captured image of the camera 61. The distance may be obtained.

FIG. 21 is a flowchart showing the operation of the HMD 100.
The controller 150 determines whether or not a contact operation on the operation surface of the track pad 14 has been detected (step S1). The control unit 150 determines whether or not a contact operation is detected based on the operation signal input from the track pad 14 (step S1). If the contact operation is not detected (step S1 / NO), control unit 150 waits until a contact operation is detected, and executes this process if there is another process that can be executed. Moreover, the control part 150 moves the pointer 301, when a contact operation is detected (step S1 / YES) (step S2). The moving distance by which the control unit 150 moves the pointer 301 is a distance corresponding to the moving distance of the user's finger on the operation surface, and the moving direction corresponds to the moving direction of the user's finger on the operation surface. Direction.

Next, the control unit 150 determines whether or not the user's finger or the like is in contact with the operation surface (step S3). In this embodiment, while the contact with the operation surface such as a finger is continued, the pointer 301 is moved by the contact operation with respect to the operation surface which is the first operation. When contact with the operation surface such as a finger is no longer detected, the control unit 150 switches the operation of the pointer 301 from the first operation to the second operation. In the second operation, the control unit 150 detects the movement of the user's head and moves the pointer 301 according to the detected movement of the head.
When detecting a contact with the operation surface (step S3 / YES), the control unit 150 returns to step S1 to determine whether or not a contact operation has been detected (step S1), and the pointer 301 is determined according to the detected contact operation. Is moved (step S2).

In addition, when the contact with the operation surface cannot be detected (step S3 / NO), the control unit 150 switches the operation of the pointer 301 to the second operation. First, the control unit 150 acquires the detection value of the 6-axis sensor 235 (step S4).
The control unit 150 determines whether or not the movement of the user's head has been detected based on the acquired detection value of the 6-axis sensor 235 (step S5). When it is determined that the user's head is not tilted (step S5 / NO), the control unit 150 determines whether an end operation for ending the operation of the pointer 301 has been received (step S6). If the end operation has not been received (step S6 / NO), the control unit 150 returns to the determination of step S4 and continues to detect the movement of the user's head based on the detection value of the 6-axis sensor 235 (step S5). . Moreover, the control part 150 complete | finishes this process flow, when completion | finish operation is received (step S6 / YES).

  When it is determined that the user's head is tilted (step S5 / YES), the control unit 150 determines whether the image display unit 20 is tilted left or right (step S7). ). In this determination, it is determined whether or not the user has moved the head to the left or right. When the user's head tilts to the left or right, the target object 305 in the outside scene that the user visually recognizes through the image display unit 20 is viewed in a state tilted to the left or right. Therefore, when it is determined that the image display unit 20 is tilted left and right, the control unit 150 controls the movement of the pointer 301 so as to correspond to the tilt of the image display unit 20.

When it is determined that the image display unit 20 is not tilted left or right (NO in step S7), the control unit 150 detects the movement of the user's head based on the detection value of the 6-axis sensor 235, The pointer 301 is moved in the direction corresponding to the detected inclination (step S8).
For example, when an operation of turning the user's face upward is detected, the control unit 150 moves the pointer 301 upward, which is the positive direction of the z axis shown in FIG. In addition, when an operation of turning the user's face downward is detected, the control unit 150 moves the pointer 301 downward, which is the negative direction of the z axis.
When the user's face is turned to the left, that is, when a left rotation around the Z axis is detected, the control unit 150 moves the pointer 301 to the left, which is the positive direction of the x axis shown in FIG. When the user's face is turned to the right, that is, when a right rotation around the Z axis is detected, the control unit 150 moves the pointer 301 to the right, which is the negative direction of the x axis shown in FIG.
The control unit 150 may move the pointer 301 horizontally based on the detection value of the magnetic sensor 237 when the pointer 301 is moved with reference to the target object 305.

  Further, when the control unit 150 determines that the image display unit 20 is tilted to the left or right (step S7 / YES), the control unit 150 determines whether the operation mode of the control unit 150 is the first operation mode. (Step S9). In the first operation mode, when the pointer 301 is moved in the horizontal direction, the pointer 301 is moved so as to be horizontal with respect to the outside scene. In a state where the user's head is tilted to the left or right, the target object 305 in the outside scene seen through the image display unit 20 also appears to tilt. Therefore, when the operation mode is the first operation mode (step S9 / YES), the control unit 150 moves the pointer 301 with reference to the target object 305 (step S10). That is, the control unit 150 adjusts the display so that the pointer 301 moves horizontally with respect to the target object 305 in the outside scene (step S10). The control unit 150 detects the rotation angle around the Y axis based on the detection value of the 6-axis sensor 235 and determines the tilt of the image display unit 20. Based on the determined inclination of the image display unit 20, the control unit 150 adjusts the display so that the pointer 301 moves horizontally with respect to the target object 305 (step S10).

  Further, when determining that the operation mode is not the first operation mode (step S9 / NO) and the second operation mode, the control unit 150 moves the pointer 301 with reference to the display image (step S11). . That is, the control unit 150 adjusts the display so that the pointer 301 is horizontal with respect to the display image. When the control unit 150 moves the pointer 301, the control unit 150 returns to step S4 to newly acquire the detection value of the 6-axis sensor 235 and detects the movement of the head (step S5).

  When the head movement is not detected in the determination in step S5 (step S5 / NO), the control unit 150 determines whether a confirmation operation has been accepted (step S6). For example, the control unit 150 determines that the confirmation operation has been received when an operation of hitting the operation surface of the track pad 14 is received. When accepting the confirmation operation, the control unit 150 determines that the target at the position overlapping the pointer 301 has been selected, and executes processing according to the selected target. When the target is the target object 305, the control unit 150 acquires a parameter generated by calibration, and based on the acquired parameter, the coordinate of the display area 310 where the pointer 301 is displayed is the coordinate on the captured image data. Convert to Then, the control unit 150 identifies the target object 305 shown at the coordinates on the converted captured image data, and executes processing according to the identified target object 305. For example, when a product button of the vending machine 305A is selected as a target, the control unit 150 performs a process of purchasing the product of the selected product button by performing wireless communication with the vending machine 305A. In addition, when a display image is selected as the target, the control unit 150 executes processing associated with the display image. For example, when the display image is a menu screen and an item on the menu screen is selected by the pointer 301, the control unit 150 executes processing corresponding to the selected item.

FIG. 22 is a layout diagram illustrating a layout of the detection device mounted on the shoe sole or the shoe interior of the user.
The embodiment described above is configured to detect the first operation and the second operation by the HMD 100, but may be configured to detect the first operation or the second operation by an external device. FIG. 22 shows a configuration in which a detection device 330 is arranged in the shoe sole or inside of a user and the first operation or the second operation is detected by the detection device 330. The detection device 330 is, for example, a pressure sensor that detects pressure generated on the sole, and a communication device that performs wireless communication with the HMD 100 by short-range wireless communication and transmits a detection value detected by the pressure sensor to the HMD 100 (both not shown). ).

FIG. 22 shows an example in which the detection device 330 is arranged in the center of the shoe on the right shoe sole or inside the shoe. The user puts weight on the toe side (front side), the heel side (rear side), the left side, or the right side according to the direction in which the pointer 301 is moved. The detection device 330 detects the direction with the highest pressure among the four directions, front, rear, left, and right, and transmits a signal indicating the detected direction to the HMD 100. Based on the signal transmitted from the detection device 330, the HMD 100 determines which one of the four directions, front, rear, left, and right has been selected. The HMD 100 moves the pointer 301 in a direction corresponding to the determined direction.
For example, when the user puts his / her weight on the toe side, the control unit 150 moves the pointer 301 upward, which is the positive direction of the z axis shown in FIG. When the user puts his / her weight on the heel side, the control unit 150 moves the pointer 301 downward, which is the negative direction of the z axis. When the user puts his / her weight on the left side, the control unit 150 moves the pointer 301 to the left, which is the positive direction of the x axis shown in FIG. When the user puts his / her weight on the right side, the control unit 150 moves the pointer 301 to the right, which is the negative direction of the x axis shown in FIG. For example, when the pointer 301 is desired to be moved in the depth direction (Y-axis direction) of the display area, the circular motion is performed so that the weight is applied in order in the four directions of front, rear, left and right. For example, when front, right, rear, and left-right rotations are detected, the control unit 150 moves the pointer 301 to the near side that is the positive direction of the y axis illustrated in FIG. In addition, for example, when front, left, rear, and right left rotations are detected, the control unit 150 moves the pointer 301 to the back side that is the negative direction of the y-axis illustrated in FIG. 9.

FIG. 23 is a diagram illustrating another arrangement example of the detection devices.
FIG. 22 shows an example in which one detection device 330 is provided in the shoe sole or the inside of the shoe. However, as shown in FIG. 23, one detection device 331 is arranged in the center of the shoe sole or the inside of the shoe, and the toe side is arranged. Two detectors 332 and 333 may be arranged in the same. The configuration of the detection devices 331, 332, and 333 is the same as that of the detection device 330 shown in FIG. The detected pressure value is transmitted to the HMD 100.
When the user puts weight on the front (toe side), the detection values of the detection devices 332 and 333 increase, and the detection value of the detection device 331 decreases. Further, when the user puts his / her weight on the back (heel side), the detection value of the detection device 331 increases and the detection values of the detection devices 332 and 333 decrease.
Further, when the user puts his / her weight on the left side or the right side, the detection value of the detection device 332 and the detection value of the detection device 333 become significantly different values. When the user puts his / her weight on the left side, the detection value of the detection device 332 increases and the detection value of the detection device 333 decreases. When the user puts his / her weight on the right side, the detection value of the detection device 332 decreases and the detection value of the detection device 333 increases.

FIG. 24 is a diagram illustrating another example of the detection device.
The arrangement example of the detection device shown in FIG. 24 shows an example in which the detection devices 334 and 335 are arranged in the soles of the left and right feet or inside the shoes, respectively. The configuration of the detection devices 334 and 335 is the same as that of the detection device 330 shown in FIG. 22, detects the direction with the highest pressure among the four directions, front, rear, left and right, and transmits a signal indicating the detected direction to the HMD 100.
For example, it is assumed that the user inputs an operation for enlarging and displaying the display image displayed at the display position of the pointer 301. In this case, the user puts weight on the inside of the toes of both feet and performs an operation of spreading the toes of both feet, that is, an operation of releasing the toes of both feet. In this case, the left foot detection device 334 determines that the pressure on the inside (right side) of the left foot is high, and transmits a signal indicating that the pressure on the right foot is high to the HMD 100. Also, the right foot detection device 335 determines that the pressure on the inside (left side) of the right foot is high, and transmits a signal indicating that the pressure on the left foot is high to the HMD 100.
When the HMD 100 determines that the pressure on the right side of the left foot is high based on the signal received from the detection device 334 for the left foot and determines that the pressure on the left side of the right foot is high based on the signal received from the detection device 335 on the right foot, the pinch out It is determined that the operation is input. In this case, the control unit 150 of the HMD 100 enlarges and displays the display image displayed at a position overlapping the pointer 301, for example. In this embodiment, an example in which a pinch-out operation is associated with enlargement of a display image has been described. However, a selection operation for selecting a display image displayed at a position overlapping the pointer 301 and a confirmation operation for confirming the operation are supported. May be attached.

For example, it is assumed that the user inputs an operation for reducing the display image displayed at the display position of the pointer 301. In this case, the user puts weight on the inside of the toes of both feet and performs an operation of closing the toes of the spread both feet. In this case, the left foot detection device 334 determines that the pressure on the outside (left side) of the left foot is high and transmits a signal indicating that the left pressure is high to the HMD 100. The right foot detection device 335 determines that the pressure on the outside (right side) of the right foot is high, and transmits a signal indicating that the pressure on the right foot is high to the HMD 100.
The HMD 100 determines that the pressure on the left side of the left foot is high based on the signal received from the detection device 334 on the left foot, and determines that the pressure on the right side of the right foot is high based on the signal received from the detection device 335 on the right foot. It is determined that an operation has been input. In this case, the control unit 150 of the HMD 100 reduces the display image displayed at a position overlapping the pointer 301, for example. In this embodiment, the example in which the pinch-in operation is associated with the enlargement of the display image has been described. However, the operation is associated with the selection operation for selecting the display image displayed at the position overlapping the pointer 301 and the confirmation operation for confirming the operation. May be.

As described above in detail, the HMD 100 of the first embodiment includes the image display unit 20, the display control unit 153, and the detection unit.
The image display unit 20 displays a display image in the display area 310 so that it can be visually recognized together with the outside scene. The display control unit 153 causes the image display unit 20 to display the pointer 301 indicating the designated position on the display area 310. The detection unit includes a 6-axis sensor 235 and a detection control unit 151, and detects an operation.
The display control unit 153 includes a first operation mode in which the position of the pointer 301 is moved corresponding to the outside scene by an operation detected by the detection unit, and a movement of the position of the pointer 301 corresponding to the outside scene by an operation detected by the detection unit. Is switched to the second operation mode that moves in a different manner.
Therefore, it is possible to switch between the first operation mode in which the position of the pointer 301 is moved corresponding to the outside scene and the second operation mode in which the pointer 301 is moved in a manner different from the movement corresponding to the outside scene. For this reason, the pointer 301 can be moved in a plurality of modes, the operability of the pointer 301 can be improved, and the usability of the HMD 100 can be improved.

Further, the 6-axis sensor 235 and the detection control unit 151 serving as the detection unit detect the first operation and the second operation as operations. The display control unit 153 moves the position of the pointer 301 according to the first operation detected by the detection unit. The display control unit 153 is configured such that after the pointer 301 is moved according to the first operation, the position of the pointer 301 is different from the movement of the pointer 301 according to the first operation according to the second operation detected by the detection unit. Move with.
Therefore, the position of the pointer 301 can be moved by a plurality of operations of the first operation and the second operation, and the pointer 301 can be moved in a different manner between the first operation and the second operation. For this reason, the operability of the pointer 301 can be improved, and the usability of the head-mounted display device can be improved.

In addition, the display control unit 153 starts the position of the pointer 301 moved by the first operation as a starting point, and moves the position of the pointer 301 in a manner different from the first operation according to the second operation.
Therefore, the position of the pointer 301 can be moved by the second operation, starting from the position of the pointer 301 moved by the first operation.

Further, the 6-axis sensor 235 and the detection control unit 151 serving as a detection unit detect a user's operation as at least one of the first operation and the second operation.
Therefore, since the user's motion is detected as at least one of the first operation and the second operation, the position of the pointer 301 can be moved by the motion.

Further, the 6-axis sensor 235 and the detection control unit 151 as the detection unit detect at least one of the inclination of the head relative to the user's body axis and the rotation around the body axis of the head as the user's operation. .
The display control unit 153 determines the amount of movement of the pointer 301 in one of the vertical direction and the horizontal direction of the display region 310 based on at least one of the tilt of the head detected by the detection unit and the rotation around the body axis. decide. The display control unit 153 moves the position of the pointer 301 based on the determined movement amount.
Therefore, the pointer 301 can be moved by tilting the user's head or rotating the head around the body axis.

The HMD 100 also includes an operation unit 110 and an operation control unit 149, a 6-axis sensor 235, and a detection control unit 151 as detection units.
The HMD 100 detects an operation received by the operation unit 110 and the operation control unit 149 as a first operation, and detects a head movement detected by the 6-axis sensor 235 and the detection control unit 151 as a second operation.
Therefore, the position of the pointer 301 can be moved by the operation of the operation unit and the movement of the head.

In addition, the HMD 100 includes a camera 61 as an imaging unit that images an outside scene.
The display control unit 153 identifies a target object that is visually recognized by overlapping the pointer 301 based on the captured image of the camera 61 during execution of the first operation mode. In addition, the display control unit 153 identifies a display image corresponding to the position of the pointer 301 from the plurality of display images displayed in the display area 310 during execution of the second operation mode.
Therefore, the target to be identified can be switched by changing the operation mode to the first operation mode or the second operation mode.

In addition, when the first operation or the second operation is detected by the detection unit during the execution of the first operation mode, the display control unit 153 displays the pointer 301 according to the first operation or the second operation by the user. Move based on the outside scene to be viewed. In addition, when the first operation or the second operation is detected by the detection unit during the execution of the second operation mode, the display control unit 153 displays the pointer 301 according to the first operation or the second operation by the user. The display image to be visually recognized is moved as a reference.
Therefore, by switching the operation mode, the pointer 301 can be easily aligned with the position of the target object or the display image.

The display control unit 153 moves the pointer 301 according to the second operation when moving the pointer 301 according to the second operation, and moves when the pointer 301 is moved according to the first operation. Make it smaller than the amount. Thereby, the movement of the pointer 301 according to the first operation and the movement of the pointer 301 according to the second operation are moved in different modes.
Therefore, the positioning of the pointer 301 to the target object 305 or the display image can be facilitated, and the number of operations for moving the pointer 301 to the position of the target object or the display image can be reduced.

In addition, the HMD 100 includes a beacon receiving unit 119 as a distance detecting unit that detects a distance to a target object that is viewed by the user through the image display unit 20. The display control unit 153 moves the pointer 301 in a different manner according to the distance to the target object detected by the beacon receiving unit 119.
Accordingly, since the pointer 301 can be moved in different modes according to the distance to the target object, the operability of the operation of the pointer 301 can be improved and the usability of the head-mounted display device can be improved.

In addition, the display control unit 153 sets the operation detected by the detection unit as the first operation when the distance to the target object 305 detected by the beacon receiving unit 119 is larger than the threshold value. The display control unit 153 sets the operation detected by the detection unit as the second operation when the distance to the target object 305 detected by the beacon receiving unit 119 is equal to or less than the threshold value. In addition, the display control unit 153 makes the movement amount for moving the pointer 301 by the first operation smaller than the movement amount for moving the pointer 301 by the second operation.
Therefore, the operation used for the movement of the pointer 301 can be changed according to the distance to the target object 305, and the movement amount of the pointer 301 can be changed. For this reason, the operability of the operation of the pointer 301 can be improved, and the usability of the HMD 100 can be improved.

The HMD 100 also includes an operation unit 110 that receives operations.
The display control unit 153 detects a moving object that moves based on the captured image of the camera 61, and displays the pointer 301 at the position of the display area 310 corresponding to the detected moving object.
Therefore, the pointer 301 can be displayed at the position of the display area 310 corresponding to the moving object.

In addition, a camera 61 and an imaging control unit 147 that capture an image of the QR code 340 disposed on the target object 305 and detect the QR code 340 are provided.
The display control unit 153 displays the pointer 301 at a position on the target object 305 where the QR code 340 detected by the camera 61 and the imaging control unit 147 is arranged. The display control unit 153 moves the position of the pointer 301 according to the operation detected by the operation unit 110, the six-axis sensor 235, and the detection control unit 151.
Therefore, the pointer 301 can be easily aligned.

Further, the beacon receiving unit 119 receives a beacon signal transmitted from the beacon transmitter 345 arranged on the target object 305. The display control unit 153 connects to the beacon transmitter 345 based on the beacon signal received by the beacon receiving unit 119, receives additional information transmitted from the beacon transmitter 345, and causes the display area 310 to display the additional information.
Therefore, additional information can be acquired from the beacon transmitter 345 and displayed.

[Second Embodiment]
A second embodiment of the present invention will be described.
The configuration of this embodiment is the same as that of the first embodiment described above. For this reason, the description about the structure of HMD100 of 2nd Embodiment is abbreviate | omitted.

  For example, when an operation for invalidating the operation of the pointer 301 is received by the operation unit 110, the control unit 150 of the present embodiment causes the storage unit 125 to store display position information indicating the display position of the pointer 301. At this time, it is assumed that the coordinates of the display area 310 are stored in the storage unit 125 as display position information. In this case, for example, when the pointer 301 is displayed at or near the position of the target object 305 existing in the outside scene, if the user moves the head, the position of the pointer 301 also moves. Become. For this reason, when the operation of the invalidated pointer 301 is validated and the pointer 301 is tried to be manipulated, the pointer 301 may move from the display position before the invalidation.

Therefore, in this embodiment, when receiving an operation for invalidating the operation of the pointer 301, the control unit 150 associates the display position of the pointer 301 with the coordinates of the three-dimensional space and stores the coordinates of the three-dimensional space in the storage unit. 125 is stored.
When receiving an operation for invalidating the operation of the pointer 301, the control unit 150 detects the azimuth angle of the HMD 100 (user's head) based on the detection value of the magnetic sensor 237 (or 113). Further, the control unit 150 detects the elevation angle of the HMD 100 (user's head) based on the detection value of the 6-axis sensor 235. Based on the detected azimuth angle and elevation angle, the control unit 150 obtains coordinates (azimuth angle and elevation angle) in the three-dimensional space that associates the display position of the pointer 301 with each other. The control unit 150 stores the obtained azimuth angle and elevation angle in the storage unit 125 as display position information.

  The control unit 150 may erase the display of the pointer 301 from the display area 310 when the operation of the pointer 301 is set to be invalid. Further, the control unit 150 may continue to display the pointer 301 at the position displayed when the operation is invalidated without deleting the display of the pointer 301. In this case, when the user moves his / her head or body and the display position of the pointer 301 deviates from the display area 310, the pointer 301 is not displayed in the display area 310. However, when the user moves his / her head or body so that the coordinates of the pointer 301 indicated by the display position information are located in the display area 310, the pointer 301 is displayed again in the display area 310.

  Further, when the user operates the pointer 301 again, for example, the user operates the operation unit 110 to set the operation of the pointer 301 to be valid. The user moves the head and body so that the coordinates of the three-dimensional space stored in the storage unit 125 as the display position information are located in the display area 310. The control unit 150 obtains the azimuth angle and the elevation angle based on the detection values of the magnetic sensor 237 (or 113) and the six-axis sensor 235, and whether or not the coordinates of the pointer 301 indicated by the display position information are located in the display area 310 Determine. When determining that the coordinates of the pointer 301 indicated by the display position information are located in the display area 310, the control unit 150 displays the pointer 301 in the display area 310. In addition, the control unit 150 receives the first operation or the second operation, and moves the display position of the pointer 301 corresponding to the received operation.

  As described above, in the present embodiment, when the operation of the pointer 301 is set to be invalid, the display position of the pointer 301 is associated with the coordinates in the three-dimensional space and stored in the storage unit 125. For this reason, even if the user moves the head or body while the operation is disabled, the display position of the pointer 301 does not move. Further, when the operation of the pointer 301 is set to be valid, the pointer 301 can be displayed at the display position when the operation is set to be invalid.

In addition, this invention is not restricted to the structure of said each embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.
For example, in the above-described embodiment, the operation for moving the display position of the pointer 301 by operating the track pad 14 is the first operation, and the operation for moving the display position of the pointer 301 by moving the user's head is the first operation. It was described as two operations. Contrary to this description, the first operation is an operation for moving the display position of the pointer 301 by moving the user's head, and the second operation is an operation for moving the display position of the pointer 301 by operating the track pad 14. It is good also as operation.

  In the above-described embodiment, the configuration in which the control device 10 is connected to the image display unit 20 by wire is illustrated. However, the present invention is not limited to this, and the image display unit 20 is wirelessly connected to the control device 10. It may be a configuration. In this case, the wireless communication method may be the method exemplified as the communication method supported by the communication unit 117, or may be another communication method.

In addition, when the HMD 100 is in a state of detecting the first operation, the control unit 150 generates an image or a message indicating that the HMD 100 is in a state of detecting the first operation, and the generated image or message is displayed in the image display unit. 20 is displayed. In addition, when the HMD 100 is in a state of detecting the second operation, the control unit 150 generates an image or a message indicating that the HMD 100 is in a state of detecting the second operation, and the generated image or message is displayed in the image display unit. 20 is displayed.
Further, when the operation mode of the HMD 100 is the first operation mode, the control unit 150 generates an image or message indicating that the operation mode is the first operation mode, and the generated image or message is displayed on the image display unit 20. To display. In addition, when the operation mode of the HMD 100 is the second operation mode, the control unit 150 generates an image or message indicating that the operation mode is the second operation mode, and the generated image or message is displayed on the image display unit 20. To display.

  In addition, some functions of the control device 10 may be provided in the image display unit 20, and the control device 10 may be realized by a plurality of devices. That is, the control device 10 is not limited to the configuration including the box-shaped case 10A. For example, instead of the control device 10, a wearable device that can be attached to a user's body, clothes, or an accessory worn by the user may be used. The wearable device in this case may be, for example, a watch-type device, a ring-type device, a laser pointer, a mouse, an air mouse, a game controller, a pen-type device, or the like.

  Further, in each of the above embodiments, the configuration in which the image display unit 20 and the control device 10 are separated and connected via the connection cable 40 has been described as an example. The present invention is not limited to this, and the control device 10 and the image display unit 20 may be configured integrally and mounted on the user's head.

  As the control device 10, a notebook computer, a tablet computer, or a desktop computer may be used. The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like.

  In each of the above embodiments, the configuration in which the user visually recognizes the outside scene through the display unit is not limited to the configuration in which the right light guide plate 26 and the left light guide plate 28 transmit external light. For example, the present invention can be applied to a display device that displays an image in a state where an outside scene cannot be visually recognized. Specifically, the present invention is applied to a display device that displays a captured image of the camera 61, an image or CG generated based on the captured image, video based on video data stored in advance or video data input from the outside, and the like. Can be applied. This type of display device can include a so-called closed display device in which an outside scene cannot be visually recognized. Further, as described in the above embodiment, the present invention is applied to a display device that does not perform processing such as AR display that displays an image superimposed on real space or MR (Mixed Reality) display that combines a captured real space image and a virtual image. The invention can be applied. The present invention can also be applied to a display device that does not perform processing such as VR (Virtual Reality) display for displaying a virtual image. For example, a display device that displays video data or an analog video signal input from the outside is naturally included as an application target of the present invention.

  Further, for example, instead of the image display unit 20, an image display unit of another method such as an image display unit worn like a hat may be adopted, and an image corresponding to the left eye LE of the user is displayed. What is necessary is just to provide the display part which displays, and the display part which displays an image corresponding to a user's right eye RE. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet. In this case, the part for positioning the position with respect to the user's body and the part positioned with respect to the part can be used as the mounting portion.

In addition, as an optical system that guides image light to the user's eyes, a configuration in which virtual images are formed by the half mirrors 261 and 281 on a part of the right light guide plate 26 and the left light guide plate 28 is illustrated. The present invention is not limited to this, and an image may be displayed on a display region having an area that occupies the entire or most of the right light guide plate 26 and the left light guide plate 28. In this case, a process of reducing the image may be included in the operation of changing the display position of the image.
Furthermore, the optical element of the present invention is not limited to the right light guide plate 26 and the left light guide plate 28 having the half mirrors 261 and 281, and may be any optical component that allows image light to enter the user's eyes. A diffraction grating, a prism, or a holographic display unit may be used.

  In addition, at least a part of the functional blocks shown in FIGS. 4 and 5 may be realized by hardware, or may be realized by cooperation of hardware and software. As described above, the present invention is not limited to a configuration in which independent hardware resources are arranged. Further, the program executed by the control unit 150 may be stored in the nonvolatile storage unit 121 or another storage device (not shown) in the control device 10. A program stored in an external device may be acquired and executed via the communication unit 117 or the external connector 184. Moreover, the operation part 110 may be formed as a user interface (UI) among the structures formed in the control apparatus 10.

  Further, the processing unit of the flowchart shown in FIG. 21 is divided according to the main processing contents in order to make the processing of the control unit 150 of the HMD 100 easy to understand. The invention is not limited. Further, the processing of the control unit 150 can be divided into more processing units according to the processing content, or can be divided so that one processing unit includes more processing. Further, the processing order of the above flowchart is not limited to the illustrated example.

  DESCRIPTION OF SYMBOLS 1 ... System, 10 ... Control apparatus, 10A ... Case, 11 ... Button, 12, 67 ... LED indicator, 14 ... Track pad, 15 ... Up / down key, 16 ... Changeover switch, 17 ... LED display part, 18 ... Power switch, DESCRIPTION OF SYMBOLS 19 ... Vibrator, 20 ... Image display part (display part), 21 ... Right holding part, 22 ... Right display unit, 23 ... Left holding part, 24 ... Left display unit, 26 ... Right light guide plate, 27 ... Front frame, 28 ... Left light guide plate, 30 ... Headset, 32 ... Right earphone, 34 ... Left earphone, 40 ... Connection cable, 46 ... Connector, 61 ... Camera (imaging part), 63 ... Microphone, 65 ... Illuminance sensor, 100 ... HMD (Receiving device, display device), 110 ... operation unit, 111 ... 6-axis sensor, 113 ... magnetic sensor, 115 ... GPS receiving unit, 117 ... communication unit, 119 ... bi CON receiver (distance detector), 120 ... controller board, 121 ... nonvolatile storage, 123 ... memory, 125 ... storage, 127 ... setting data, 128 ... content data, 130 ... power supply, 132 ... battery, 134 DESCRIPTION OF SYMBOLS ... Power supply control circuit 140 ... Main processor 143 ... Operating system 145 ... Image processing part 147 ... Imaging control part 149 ... Operation control part (detection part), 150 ... Control part, 151 ... Detection control part (detection part) 153, display control unit, 180, audio codec, 182 ... audio interface, 184 ... external connector, 186 ... external memory interface, 188 ... USB connector, 192 ... sensor hub, 194 ... FPGA, 196 ... interface, 200 ... beacon Device, 211 ... Interface 213 ... receiver, 215 ... EEPROM, 217 ... temperature sensor, 221 ... OLED unit, 223 ... OLED panel, 225 ... OLED drive circuit, 231 ... interface, 233 ... receiver, 235 ... 6-axis sensor (detector) 237 ... magnetic sensor, 239 ... temperature sensor, 241 ... OLED unit, 243 ... OLED panel, 245 ... OLED drive circuit, 251 ... right optical system, 252 ... left optical system, 261,281 ... half mirror, 300 ... field of view, 301 ... Pointer, 302 ... Display screen of word processing software, 303 ... Enlarged image, 304 ... Software keyboard, 304A ... First key screen, 304B ... Second key screen, 305 ... Target object, 305A ... Vending machine, 305B ... Refrigerated Showcase, 307, user image, 308 Pop display, 310 ... display area, from 330 to 335 ... detection device, 340 ... QR code.

Claims (21)

A head-mounted display device mounted on a user's head,
A display unit that displays a display image in a display area so that it can be viewed together with an outside scene;
A display control unit that causes the display unit to display a marker indicating the indicated position in the display area;
A detection unit for detecting an operation,
The display control unit includes a first operation mode in which the position of the marker is moved corresponding to the outside scene by the operation detected by the detection unit, and the position of the marker by the operation detected by the detection unit. A head-mounted display device that switches between a second operation mode that moves in a manner different from movement corresponding to the outside scene. The detection unit detects a first operation and a second operation as the operation,
The display control unit moves the position of the marker according to the first operation detected by the detection unit, and the second operation detected by the detection unit after the marker moves according to the first operation. The head-mounted display device according to claim 1, wherein the position of the marker is moved in a manner different from the movement of the marker according to the first operation.   The said display control part moves the position of the said marker in the aspect different from the said 1st operation according to the said 2nd operation from the position of the said marker moved by the said 1st operation as a starting point. Head-mounted display device.   The head-mounted display device according to claim 2, wherein the detection unit detects an operation of the user as at least one of the first operation and the second operation. The detection unit detects at least one of the inclination of the head relative to the user's body axis and the rotation of the head around the body axis as the user's motion,
The display control unit, based on at least one of the tilt of the head detected by the detection unit and the rotation around the body axis, the marker in the vertical direction or the horizontal direction of the display region The head-mounted display device according to claim 4, wherein a movement amount is determined and the position of the marker is moved based on the determined movement amount.   The detection unit detects an operation received by an operation unit included in the head-mounted display device as the first operation, and detects a movement of the head as the second operation. The head-mounted display device according to the item. An imaging unit for imaging the outside scene;
The display control unit identifies a target object that is visually recognized by overlapping the marker based on a captured image of the imaging unit during execution of the first operation mode, and the execution of the second operation mode The head-mounted display device according to claim 1, wherein the display image corresponding to the position of the marker is specified from the plurality of display images displayed in a display area. When the first operation or the second operation is detected by the detection unit during execution of the first operation mode, the display control unit is configured to select the marker according to the first operation or the second operation. Is moved with reference to the outside scene visually recognized by the user,
When the first operation or the second operation is detected by the detection unit during execution of the second operation mode, the user visually recognizes the marker according to the first operation or the second operation. The head-mounted display device according to claim 2, wherein the display image is moved with reference to the display image.   When the display control unit moves the marker according to the second operation, the display control unit moves the marker according to the second operation, and moves the marker according to the first operation. The head-mounted display device according to claim 2, wherein the movement of the marker according to the first operation and the movement of the marker according to the second operation are moved in a different manner by making the movement amount smaller than the movement amount. . A distance detection unit that detects a distance to a target object that the user sees through the display unit;
The head-mounted display device according to any one of claims 1 to 5, wherein the display control unit moves the marker in a different manner according to a distance to the target object detected by the distance detection unit. The display control unit detects the operation detected by the detection unit as the first operation when the distance to the target object detected by the distance detection unit is greater than a threshold value, and the distance detection unit detects the first operation. When the distance to the target object is less than or equal to a threshold value, the operation detected by the detection unit is the second operation,
The head-mounted display device according to claim 10, wherein a movement amount for moving the marker by the first operation is smaller than a movement amount for moving the marker by the second operation. A head-mounted display device mounted on a user's head,
A display unit that displays a display image in a display area so as to transmit the outside scene and be visible together with the outside scene;
An imaging unit for imaging the outside scene;
A display control unit that causes the display unit to display a marker indicating an indication position on the display area,
The display control unit, based on a captured image of the imaging unit, a first operation mode that identifies a target object that is visually recognized overlapping with the marker, and a plurality of display images displayed in the display area, A head-mounted display device that switches and executes a second operation mode for specifying the display image corresponding to a position. A detection unit for detecting operation;
The display control unit is based on the outside scene in which the user visually recognizes the marker according to the detected operation when an operation is detected by the detection unit during the execution of the first operation mode. Move
13. When an operation is detected by the detection unit during execution of the second operation mode, the marker is moved based on the display image visually recognized by the user according to the detected operation. The head-mounted display device described. The detection unit detects an operation received by an operation unit included in the head-mounted display device as a first operation, detects a movement of the user's head as a second operation,
The display control unit moves the position of the marker according to the first operation detected by the detection unit, and the detection unit detects the first position after movement of the marker position according to the first operation. 2 Move the position of the marker according to the operation,
The head-mounted display according to claim 12 or 13, wherein a movement amount when the marker is moved according to the second operation is smaller than a movement amount when the marker is moved according to the first operation. apparatus.   The display control unit displays a display indicating a moving direction of the marker with reference to the outside scene during execution of the first operation mode, and uses the display image as a reference during execution of the second operation mode. The head-mounted display device according to claim 13 or 14, wherein a display indicating a moving direction of the marker is displayed.   The display control unit manages the display position of the marker by coordinates in the display area, and when the operation detected by the detection unit is set to invalid, the display position of the marker corresponds to the real space. The head-mounted display device according to claim 12 or 13, which is managed in association with coordinates in a three-dimensional space. It has an operation unit that accepts operations,
The said display control part detects the moving object which moves based on the captured image of the said imaging part, and displays the said marker in the position of the said display area corresponding to the detected said moving object. A head-mounted display device according to one item. A mark detection unit for detecting an identification mark arranged on the target object;
The display control unit displays the marker at the position of the target object on which the identification mark detected by the mark detection unit is arranged, and moves the position of the marker according to an operation detected by the detection unit The head-mounted display device according to any one of claims 12 to 17. A receiving unit for receiving a beacon signal transmitted from a beacon transmitter disposed on the target object;
The display control unit is connected to the beacon transmitter based on the beacon signal received by the receiving unit, receives additional information transmitted from the transmitter, and displays the additional information in the display area. A head-mounted display device according to claim 1. A display unit that displays a display image in a display area so as to be visible through the outside scene and visible together with the outside scene, a display control unit that displays a marker indicating an indicated position in the display area by the display unit, and detection that detects an operation A program to be executed by a computer mounted on a head-mounted display device mounted on a user's head,
On the computer,
A procedure for moving the position of the marker corresponding to the outside scene by the operation detected by the detection unit;
A procedure for moving the position of the marker in a manner different from the movement corresponding to the outside scene by the operation detected by the detection unit;
A program for running A display unit that displays a display image in a display area so as to be visible through the outside scene and visible together with the outside scene, a display control unit that displays a marker indicating an indicated position in the display area by the display unit, and detection that detects an operation A head-mounted display device mounted on a user's head, comprising:
Moving the position of the marker corresponding to the outside scene by the operation detected by the detection unit;
Moving the position of the marker in a manner different from the movement corresponding to the outside scene by the operation detected by the detection unit;
Control method for a head-mounted display device having

Images