JP2018091882A - Head-mounted display, program, and method for controlling head-mounted display - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve operability to improve convenience of a head-mounted display.SOLUTION: An HMD 100 comprises: an image display part 20 that displays a display image in a display area 310 such that a user can visually recognize the display image while passing through an outer scenery and together with the outer scenery; a display control part 153 that displays a pointer 301 indicating an indication position in the display area 310 with the image display part 20; an operation control part 149 that detects a first operation; and a detection control part 151 that detects a second operation. The display control part 153 moves the position of the pointer 301 according to the first operation detected by the operation control part 149, and after moving the position of the pointer 301 according to the first operation, moves the position of the pointer 301 according to the second operation detected by the detection control part 151 in a mode different from that of the movement of the position of the pointer 301 according to the first operation.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a head-mounted display device, a program, and a control method for the head-mounted display device.

2. Description of the Related Art Conventionally, a pointing device that moves a marker to select a displayed menu or designate a displayed object is known. As a pointing device, a movement of a user's head is detected, and a marker is moved according to the detected movement of the head (for example, see Patent Document 1).
Patent Document 1 detects changes in the position and orientation of the head based on ultrasonic waves, electromagnetic waves, and the like transmitted from a plurality of transmitters mounted on the head, and detects the detected position and orientation of the head. A head-driven pointing device is disclosed that indicates a specific position on a display in response to a change.

JP-A-6-187092

By the way, when operating the marker by the movement of the head in the head-mounted display device, the user must keep the head tilted while moving the marker, and must maintain an unreasonable posture was there. Also, when operating the operation unit such as a trackpad to move the marker to the target position, if the operation unit is not fixed, it is not easy to move the marker as the user imagined, and operability There is a problem of lacking. For this reason, improvement in the operability of the head-mounted display device has been desired.
An object of the present invention is to improve operability and improve the usability of a head-mounted display device.

In order to solve the above-described problems, the present invention is a head-mounted display device that is worn on a user's head, and displays a display image in a display region so as to transmit the outside scene and be visible together with the outside scene. A display unit, a display control unit that displays a marker indicating an indication position on the display area by the display unit, a detection unit that detects a first operation and a second operation, and the display control unit includes: The position of the marker is moved according to the first operation detected by the detection unit, and after the movement of the position of the marker according to the first operation, according to the second operation detected by the detection unit, The marker position is moved in a manner different from the movement of the marker position according to the first operation.
According to the present invention, the position of the marker can be moved in a different manner between the first operation and the second operation. Therefore, since the marker can be moved in a plurality of modes, the operability of the marker operation can be improved, and the usability of the head-mounted display device can be improved.

In the present invention, the detection unit detects the user's movement as at least one of the first operation and the second operation.
According to the present invention, the position of the marker can be moved by the operation of the user.

In the present invention, the detection unit detects an operation received by an operation unit included in the head-mounted display device as a first operation, and detects a movement of the head as a second operation.
According to the present invention, the position of the marker can be moved by the operation of the operation unit and the movement of the head.

In addition, the present invention includes an imaging unit that images the outside scene, and the display control unit includes a first operation mode that identifies a target object that is visually recognized by overlapping the marker based on a captured image of the imaging unit. The second operation mode for specifying the display image corresponding to the position of the marker is switched and executed from the plurality of display images displayed in the display area.
According to the present invention, a target object can be specified by a marker, and a display image can be specified.

Further, according to the present invention, the display control unit moves the marker in a different manner between when the first operation mode is being executed and when the second operation mode is being executed.
According to the present invention, the marker can be moved in a different manner by changing the operation mode.

Further, the present invention provides the display control unit when the first operation or the second operation is detected by the detection unit during execution of the first operation mode. In response to two operations, the marker is moved with reference to the outside scene visually recognized by the user, and the first operation or the second operation is detected by the detection unit while executing the second operation mode. In this case, the marker is moved with reference to the display image visually recognized by the user according to the first operation or the second operation.
According to the present invention, when the first operation mode or the second operation is detected when the operation mode of the head-mounted display device is the first operation mode, the marker can be moved with reference to the outside scene. . Further, when the first operation or the second operation is detected when the operation mode of the head-mounted display device is the second operation mode, the marker can be moved with reference to the display image.

In the present invention, the display control unit makes a movement amount when the marker is moved according to the second operation smaller than a movement amount when the marker is moved according to the first operation. .
According to the present invention, the marker can be moved by the second operation in which the amount of movement of the marker is smaller than that of the first operation after the marker is moved by the first operation in which the amount of movement of the marker is larger than that of the second operation. . Therefore, the number of operations for bringing the marker closer to the target object or the display image can be reduced, and the marker can be finely adjusted to the position of the target object or the display image by the second operation with a small amount of movement of the marker.

The present invention may further include a distance detection unit that detects a distance to the target object that the user sees through the display unit, and the display control unit detects the target object detected by the distance detection unit. The marker is moved in a different manner depending on the distance up to.
According to the present invention, the marker can be moved in different manners depending on the distance to the target object. Therefore, the operability of the marker operation can be improved and the usability of the head-mounted display device can be improved.

Further, in the present invention, the display control unit sets the operation detected by the detection unit when the distance to the target object detected by the distance detection unit is larger than a threshold as the first operation, When the distance to the target object detected by the distance detection unit is equal to or less than a threshold value, the operation detected by the detection unit is the second operation, and the amount of movement by which the marker is moved by the first operation, The amount of movement is smaller than the amount of movement of the marker by the second operation.
According to the present invention, when the distance to the target object is larger than the threshold value, the operation detected by the detection unit is the first operation in which the amount of movement of the marker is larger than the second operation, and the first operation The marker can be moved. Further, when the distance to the target object is equal to or less than the threshold value, the operation detected by the detection unit is the second operation in which the amount of movement of the marker is smaller than the first operation, and the marker is moved by the second operation. be able to.
Accordingly, the number of operations for bringing the marker closer to the target object can be reduced, and the marker can be finely adjusted to the position of the target object or the display image by the second operation in which the amount of movement of the marker is small.

In order to solve the above-described problems, the present invention is a head-mounted display device that is worn on a user's head, and displays a display image in a display region so as to transmit the outside scene and be visible together with the outside scene. A display unit; an imaging unit that images the outside scene; and a display control unit that causes the display unit to display a marker indicating an indication position in the display area. The display control unit includes a captured image of the imaging unit. Based on the first operation mode for identifying a target object that is visually recognized by overlapping with the marker, and for identifying the display image corresponding to the position of the marker from the plurality of display images displayed in the display area. The operation mode is switched between and executed.
According to the present invention, the marker can be moved in a different manner by changing the operation mode.

Further, the present invention includes a detection unit that detects an operation, and the display control unit is configured to detect the operation when the operation is detected by the detection unit during execution of the first operation mode. In response, the marker is moved with reference to the outside scene visually recognized by the user, and when the operation is detected by the detection unit during execution of the second operation mode, according to the detected operation The marker is moved with reference to the display image visually recognized by the user.
According to the present invention, when an operation is detected when the operation mode of the head-mounted display device is the first operation mode, the marker can be moved with reference to the outside scene. In addition, when an operation is detected when the operation mode of the head-mounted display device is the second operation mode, the marker can be moved with reference to the display image.

Further, according to the present invention, the detection unit detects an operation received by an operation unit included in the head-mounted display device as a first operation, detects a movement of the user's head as a second operation, and displays the display. The control unit moves the position of the marker according to the first operation detected by the detection unit, and the second operation detected by the detection unit after the position of the marker is moved according to the first operation. The position of the marker is moved according to the movement, and the movement amount when moving the marker according to the second operation is made smaller than the movement amount when moving the marker according to the first operation.
According to the present invention, the marker can be moved by the second operation in which the amount of movement of the marker is smaller than that of the first operation after the marker is moved by the first operation in which the amount of movement of the marker is larger than that of the second operation. . Therefore, the number of operations for bringing the marker closer to the target object or the display image can be reduced, and the marker can be finely adjusted to the position of the target object or the display image by the second operation with a small amount of movement of the marker.

In order to solve the above problems, the present invention provides a display unit that displays a display image in a display area so that it can be seen through the outside scene and is visible together with the outside scene, and a marker that indicates an indicated position is displayed in the display area by the display unit. A program that is executed by a computer that is mounted on a head-mounted display device that is mounted on a user's head, the display control unit being configured to detect the first operation, and a detection unit that detects the second operation Then, the detection unit detects after the procedure of causing the computer to move the position of the marker according to the first operation detected by the detection unit and the movement of the marker position according to the first operation. In accordance with the second operation, a procedure for moving the marker position in a manner different from the movement of the marker position according to the first operation is executed.
According to the present invention, the position of the marker can be moved in a different manner between the first operation and the second operation. Therefore, since the marker can be moved in a plurality of modes, the operability of the marker operation can be improved, and the usability of the head-mounted display device can be improved.

In order to solve the above problems, the present invention provides a display unit that displays a display image in a display area so that it can be seen through the outside scene and is visible together with the outside scene, and a marker that indicates an indicated position is displayed in the display area by the display unit. A control method for a head-mounted display device mounted on a user's head, comprising: a display control unit to be operated; a detection unit that detects a first operation and a second operation; The step of moving the position of the marker according to the first operation detected by the method, and the movement of the position of the marker according to the first operation, and the second operation detected by the detection unit, Moving the marker position in a manner different from the movement of the marker position according to the first operation.
According to the present invention, the position of the marker can be moved in a different manner between the first operation and the second operation. Therefore, since the marker can be moved in a plurality of modes, the operability of the marker operation can be improved, and the usability of the head-mounted display device can be improved.

FIG. The principal part top view which shows the structure of the optical system of HMD. The perspective view which shows the structure of an image display part. The block diagram of HMD. The functional block diagram of a control apparatus. The figure which shows a user's visual field. The figure which shows the visual field seen in the state which the user inclined the neck. The figure which shows the visual field seen in the state which the user inclined the neck. The figure which shows the enlarged image displayed when a pointer is moved to the vicinity of a target. The figure which shows the relationship between the distance to a target, and the moving distance of a pointer. The flowchart which shows operation | movement of HMD. The layout of the detection apparatus with which the user's shoe sole or shoes were mounted | worn. The layout of the detection apparatus with which the user's shoe sole or shoes were mounted | worn. The layout of the detection apparatus with which the user's shoe sole or shoes were mounted | worn.

FIG. 1 is an external view showing an external configuration of an HMD (Head Mounted Display) 100 to which the present invention is applied.
HMD100 is a display apparatus provided with the image display part 20 (display part) which makes a user visually recognize a virtual image in the state with which the user (user) was mounted | worn, and the control apparatus 10 which controls the image display part 20. It is.

  The control device 10 includes a flat box-shaped case 10A (also referred to as a housing or a main body) as shown in FIG. The case 10 </ b> A includes buttons 11, an LED indicator 12, a track pad 14, an up / down key 15, a changeover switch 16, and a power switch 18. The button 11, the track pad 14, the up / down key 15, the changeover switch 16 and the power switch 18 are operated parts operated by the user. In addition, the LED indicator 12 functions as a sub display unit that indicates an operation state of the HMD 100, for example. The user can operate the HMD 100 by operating the operated part. The control device 10 functions as a controller of the HMD 100.

The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 in a main body having a right holding unit 21, a left holding unit 23, and a front frame 27.
Each of the right holding unit 21 and the left holding unit 23 extends rearward from both ends of the front frame 27, and holds the image display unit 20 on the user's head like a temple of glasses. Here, of both ends of the front frame 27, an end located on the right side of the user in the mounted state of the image display unit 20 is defined as an end ER, and an end located on the left side of the user is defined as an end EL. To do. The right holding unit 21 extends from the end ER of the front frame 27 to a position corresponding to the right side of the user when the image display unit 20 is mounted. The left holding unit 23 is provided to extend from the end EL to a position corresponding to the left side of the user when the image display unit 20 is mounted.

  The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is positioned in front of the right eye of the user when the image display unit 20 is mounted, and causes the right eye to visually recognize the image. The left light guide plate 28 is positioned in front of the left eye of the user when the image display unit 20 is mounted, and causes the left eye to visually recognize the image.

  The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other, and this connection position is in a wearing state in which the user wears the image display unit 20. Corresponding to the eyebrows. The front frame 27 may be provided with a nose pad portion that comes into contact with the user's nose when the image display unit 20 is mounted at a connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the user's head by the nose pad, the right holding unit 21 and the left holding unit 23. Further, a belt (not shown) that contacts the back of the user when the image display unit 20 is mounted may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 is used by the belt. Can be held on the person's head.

  The right display unit 22 realizes image display by the right light guide plate 26. The right display unit 22 is provided in the right holding unit 21 and is located in the vicinity of the right side of the user in the mounted state. The left display unit 24 realizes image display by the left light guide plate 28. The left display unit 24 is provided in the left holding part 23 and is located in the vicinity of the left side of the user in the wearing state.

The right light guide plate 26 and the left light guide plate 28 of the present embodiment are optical units formed of a light-transmitting resin or the like, and are, for example, prisms, and output image light output from the right display unit 22 and the left display unit 24. Lead to the eyes of the user.
A light control plate (not shown) may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The light control plate is an optical element on a thin plate having different transmittance depending on the wavelength region of light, and functions as a so-called wavelength filter. For example, the light control plate is disposed so as to cover the front side of the front frame 27 which is the side opposite to the eye side of the user. By appropriately selecting the optical characteristics of the light control plate, the transmittance of light in an arbitrary wavelength region such as visible light, infrared light, and ultraviolet light can be adjusted. The amount of external light incident on the light plate 28 and transmitted through the right light guide plate 26 and the left light guide plate 28 can be adjusted.

  The image display unit 20 guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, respectively. The image light guided to the right light guide plate 26 and the left light guide plate 28 enters the right eye and the left eye of the user and makes the user visually recognize a virtual image. Thereby, the image display unit 20 displays an image.

  When the external light enters the user's eyes through the right light guide plate 26 and the left light guide plate 28 from the front of the user, the image light and the external light constituting the virtual image enter the user's eyes. In other words, the visibility of the virtual image is affected by the intensity of external light. For this reason, for example, by attaching a light control plate to the front frame 27 and appropriately selecting or adjusting the optical characteristics of the light control plate, the ease of visual recognition of the virtual image can be adjusted. In a typical example, it is possible to use a light control plate having a light transmittance that allows a user wearing the HMD 100 to visually recognize at least the outside scenery. In addition, when the dimming plate is used, it is possible to protect the right light guide plate 26 and the left light guide plate 28 and to suppress the damage to the right light guide plate 26 and the left light guide plate 28, the adhesion of dirt, and the like. The light control plate may be detachable with respect to the front frame 27 or each of the right light guide plate 26 and the left light guide plate 28, or may be mounted by replacing a plurality of types of light control plates. May be omitted.

The camera 61 is disposed on the front frame 27 of the image display unit 20. The configuration and arrangement of the camera 61 are determined so as to capture an outside scene direction visually recognized by the user while wearing the image display unit 20. For example, the camera 61 is provided on the front surface of the front frame 27 at a position that does not block external light that passes through the right light guide plate 26 and the left light guide plate 28. In the example of FIG. 1, the camera 61 is disposed on the end ER side of the front frame 27, but the camera 61 may be disposed on the end EL side, and the right light guide plate 26 and the left light guide plate 28. You may arrange | position in the connection part.
The camera 61 corresponds to an “imaging unit” of the present invention.

The camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like. The camera 61 of this embodiment is a monocular camera, but may be a stereo camera. The camera 61 images at least a part of the outside scene (real space) in the front side direction of the HMD 100, in other words, the user's viewing direction when the HMD 100 is worn. In another expression, the camera 61 captures a range or direction that overlaps the user's field of view, and captures a direction in which the user gazes. The direction and width of the angle of view of the camera 61 can be set as appropriate. In the present embodiment, as will be described later, the angle of view of the camera 61 includes the outside world visually recognized by the user through the right light guide plate 26 and the left light guide plate 28. More preferably, the angle of view of the camera 61 is set so that the entire field of view of the user who can see through the right light guide plate 26 and the left light guide plate 28 can be imaged.
The camera 61 performs imaging according to the control of the imaging control unit 147 included in the control unit 150 (FIG. 5). The camera 61 outputs captured image data to the control unit 150 via an interface 211 described later.

  The HMD 100 may include a distance sensor (not shown) that detects a distance to a measurement object positioned in a preset measurement direction. The distance sensor can be disposed, for example, at a connection portion between the right light guide plate 26 and the left light guide plate 28 in the front frame 27. In this case, when the image display unit 20 is mounted, the position of the distance sensor is substantially in the middle between the user's eyes in the horizontal direction and above the user's eyes in the vertical direction. The measurement direction of the distance sensor can be, for example, the front side direction of the front frame 27, in other words, the direction overlapping the imaging direction of the camera 61. The distance sensor can be configured to include, for example, a light source such as an LED or a laser diode, and a light receiving unit that receives reflected light that is reflected from the light to be measured. The distance sensor may perform a triangulation process or a distance measurement process based on a time difference in accordance with the control of the control unit 150. The distance sensor may be configured to include a sound source that emits ultrasonic waves and a detection unit that receives ultrasonic waves reflected by the measurement object. In this case, the distance sensor may perform the distance measurement process based on the time difference until the reflection of the ultrasonic wave according to the control of the control unit 150.

FIG. 2 is a principal plan view showing the configuration of the optical system provided in the image display unit 20. FIG. 2 shows the user's left eye LE and right eye RE for explanation.
As shown in FIG. 2, the right display unit 22 and the left display unit 24 are configured symmetrically. The right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 that emits image light and a lens group that guides the image light L emitted from the OLED unit 221 as a configuration that allows the user's right eye RE to visually recognize an image. Right optical system 251. The image light L is guided to the right light guide plate 26 by the right optical system 251.

The OLED unit 221 includes an OLED panel 223 and an OLED drive circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-luminous display panel configured by arranging, in a matrix, light emitting elements that emit light of R (red), G (green), and B (blue) by emitting light by organic electroluminescence. It is. The OLED panel 223 includes a plurality of pixels, each of which includes one R, G, and B element, and forms an image with pixels arranged in a matrix. The OLED drive circuit 225 selects a light emitting element included in the OLED panel 223 and energizes the light emitting element under the control of the control unit 150 (FIG. 5), and causes the light emitting element of the OLED panel 223 to emit light. The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back surface of the light emitting surface by bonding or the like. The OLED drive circuit 225 may be formed of a semiconductor device that drives the OLED panel 223, for example, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 223. A temperature sensor 217 is mounted on this substrate.
Note that the OLED panel 223 may have a configuration in which light emitting elements that emit white light are arranged in a matrix, and color filters corresponding to R, G, and B colors are stacked. Further, an OLED panel 223 having a WRGB configuration provided with a light emitting element that emits W (white) light in addition to the light emitting elements that respectively emit R, G, and B color light may be used.

  The right optical system 251 includes a collimator lens that converts the image light L emitted from the OLED panel 223 into a parallel light flux. The image light L converted into a parallel light beam by the collimator lens enters the right light guide plate 26. A plurality of reflecting surfaces that reflect the image light L are formed in an optical path that guides light inside the right light guide plate 26. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 (reflection surface) located in front of the right eye RE is formed on the right light guide plate 26. The image light L is reflected by the half mirror 261 and emitted from the right light guide plate 26 toward the right eye RE. The image light L forms an image on the retina of the right eye RE, and allows the user to visually recognize the image.

  In addition, the left display unit 24 has a left optical system including an OLED unit 241 that emits image light and a lens group that guides the image light L emitted from the OLED unit 241 as a configuration that allows the user's left eye LE to visually recognize an image. 252. The image light L is guided to the left light guide plate 28 by the left optical system 252.

  The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. The OLED panel 243 is a self-luminous display panel configured similarly to the OLED panel 223. The OLED driving circuit 245 causes the light emitting elements of the OLED panel 243 to emit light by selecting a light emitting element included in the OLED panel 243 and energizing the light emitting element according to the control of the control unit 150 (FIG. 5). The OLED drive circuit 245 is fixed to the back surface of the OLED panel 243, that is, the back surface of the light emitting surface by bonding or the like. The OLED drive circuit 245 may be configured by a semiconductor device that drives the OLED panel 243, for example, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 243. A temperature sensor 239 is mounted on this substrate.

  The left optical system 252 includes a collimating lens that converts the image light L emitted from the OLED panel 243 into a light beam in a parallel state. The image light L converted into a parallel light beam by the collimator lens enters the left light guide plate 28. The left light guide plate 28 is an optical element formed with a plurality of reflecting surfaces that reflect the image light L, and is, for example, a prism. The image light L is guided to the left eye LE side through a plurality of reflections inside the left light guide plate 28. The left light guide plate 28 is formed with a half mirror 281 (reflection surface) located in front of the left eye LE. The image light L is reflected by the half mirror 281 and emitted from the left light guide plate 28 toward the left eye LE. The image light L forms an image on the retina of the left eye LE, and allows the user to visually recognize the image.

According to this configuration, the HMD 100 functions as a see-through display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 are incident on the user's right eye RE. Further, the image light L reflected by the half mirror 281 and the external light OL transmitted through the half mirror 281 are incident on the left eye LE. In this way, the HMD 100 superimposes the image light L of the internally processed image and the external light OL so as to enter the user's eyes, and for the user, the outside light can be seen through the right light guide plate 26 and the left light guide plate 28. The image by the image light L is visually recognized over the outside scene.
The half mirrors 261 and 281 are image extraction units that extract image by reflecting image light output from the right display unit 22 and the left display unit 24, respectively, and can be called display units.

  The left optical system 252 and the left light guide plate 28 are collectively referred to as a “left light guide”, and the right optical system 251 and the right light guide plate 26 are collectively referred to as a “right light guide”. The configuration of the right light guide and the left light guide is not limited to the above example, and any method can be used as long as a virtual image is formed in front of the user's eyes using image light. It may be used, or a semi-transmissive reflective film may be used.

Returning to FIG. 1, the control device 10 and the image display unit 20 are connected by a connection cable 40. The connection cable 40 is detachably connected to a connector (not shown) provided at the lower part of the case 10 </ b> A, and is connected to various circuits provided in the image display unit 20 from the tip of the left holding unit 23. The connection cable 40 includes a metal cable that transmits digital data or an optical fiber cable, and may include a metal cable that transmits an analog signal. A connector 46 is provided in the middle of the connection cable 40. The connector 46 is a jack for connecting a stereo mini plug, and the connector 46 and the control device 10 are connected by a line for transmitting an analog audio signal, for example. In the configuration example shown in FIG. 1, a headset 30 having a right earphone 32 and a left earphone 34 and a microphone 63 constituting a stereo headphone is connected to a connector 46.
The control device 10 and the image display unit 20 may be wirelessly connected. For example, the control device 10 and the image display unit 20 mutually transmit and receive control signals and data by wireless communication complying with standards such as Bluetooth (registered trademark) and wireless LAN (including Wi-Fi (registered trademark)). It is good also as a structure.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collection unit of the microphone 63 faces the user's line of sight, collects sound, and outputs a sound signal to the sound interface 182 (FIG. 4). To do. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.

  The control device 10 includes a button 11, an LED indicator 12, a track pad 14, an up / down key 15, a changeover switch 16, and a power switch 18 as operated parts operated by a user. These operated parts are arranged on the surface of the case 10A.

The button 11 includes keys and switches for operating the control device 10, and these keys and switches are displaced by a pressing operation. For example, the button 11 includes a menu key, a home key, and a “return” key for performing operations related to the operating system 143 (see FIG. 5) executed by the control device 10.
The LED indicator 12 lights up or blinks in accordance with the operating state of the HMD 100. The up / down key 15 is used to input an instruction to increase / decrease the volume output from the right earphone 32 and the left earphone 34 and to input an instruction to increase / decrease the display brightness of the image display unit 20. The changeover switch 16 is a switch for changing an input corresponding to the operation of the up / down key 15. The power switch 18 is a switch that switches on / off the power of the HMD 100, and is, for example, a slide switch.

  The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. The detection method on the operation surface is not limited, and an electrostatic method, a pressure detection method, an optical method, or other methods can be adopted. Contact (touch operation) to the track pad 14 is detected by a touch sensor (not shown). An LED display unit 17 is installed on the track pad 14. The LED display unit 17 includes a plurality of LEDs, and light of each LED is transmitted through the track pad 14 to display an operation icon or the like. This icon or the like functions as a software button. The track pad 14 corresponds to the “detection unit” of the present invention.

FIG. 3 is a perspective view showing a configuration of the image display unit 20 and shows a main configuration of the image display unit 20 as viewed from the user's head side. FIG. 3 shows the side of the image display unit 20 that contacts the user's head, in other words, the side that can be seen by the user's right eye RE and left eye LE. In other words, the back sides of the right light guide plate 26 and the left light guide plate 28 are visible.
In FIG. 3, the half mirror 261 that irradiates the user's right eye RE with the image light and the half mirror 281 that irradiates the left eye LE with the image light appear as substantially rectangular areas. Further, the entire right light guide plate 26 and left light guide plate 28 including the half mirrors 261 and 281 transmit external light as described above. For this reason, the user sees the outside scene through the entire right light guide plate 26 and the left light guide plate 28, and sees a rectangular display image at the positions of the half mirrors 261 and 281.

  The camera 61 is disposed at the right end of the image display unit 20 and images the direction in which both eyes of the user face, that is, the front for the user. The optical axis of the camera 61 is a direction including the line-of-sight directions of the right eye RE and the left eye LE. The outside scene that the user can visually recognize in a state where the HMD 100 is worn is not always at infinity. For example, when the user gazes at a target object located in front of the user with both eyes, the distance from the user to the target object is often about 30 cm to 10 m, and is about 1 m to 4 m. But more. Therefore, for the HMD 100, an upper limit and a lower limit guideline for the distance from the user to the target object during normal use may be set. This standard may be obtained by investigation or experiment, or may be set by the user. The optical axis and the angle of view of the camera 61 are set so that the distance to the target object during normal use corresponds to the set upper limit guideline and the lower limit guideline. It is preferable to be set so as to be included.

  In general, the viewing angle of a human is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Among them, the effective field of view with excellent information receiving ability is about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction. Furthermore, the stable gaze field in which the gaze point that the human gazes at appears to be quickly and stably is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction. When the gazing point is a target object positioned in front of the user, in the user's field of view, about 30 degrees in the horizontal direction and about 20 degrees in the vertical direction are effective around the line of sight of each of the right eye RE and the left eye LE. It is a field of view. Further, the stable viewing field is about 60 to 90 degrees in the horizontal direction and about 45 to 70 degrees in the vertical direction, and the viewing angle is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Furthermore, the actual visual field that the user sees through the right light guide plate 26 and the left light guide plate 28 can be referred to as a real field of view (FOV: Field Of View). In the configuration of the present embodiment shown in FIGS. 1 and 2, the actual visual field corresponds to an actual visual field that is transmitted through the right light guide plate 26 and the left light guide plate 28 and visually recognized by the user. The real field of view is narrower than the viewing angle and stable field of view, but wider than the effective field of view.

  The field angle of the camera 61 is preferably capable of capturing an image in a wider range than the user's field of view, and specifically, the field angle is preferably at least wider than the user's effective field of view. Further, it is more preferable that the angle of view is wider than the actual field of view of the user. More preferably, the angle of view is wider than the stable viewing field of the user, and most preferably, the angle of view is wider than the viewing angle of both eyes of the user.

  The camera 61 may include a so-called wide-angle lens as an imaging lens so that a wide angle of view can be captured. The wide-angle lens may include a lens called a super-wide-angle lens or a quasi-wide-angle lens, may be a single focus lens or a zoom lens, and the camera 61 includes a lens group including a plurality of lenses. There may be.

FIG. 4 is a block diagram illustrating a configuration of each unit included in the HMD 100.
The control device 10 includes a main processor 140 that executes a program and controls the HMD 100. A non-volatile storage unit 121 and a memory 123 are connected to the main processor 140. The main processor 140 is connected to the track pad 14 and the operation unit 110 as input devices. In addition, a 6-axis sensor 111 and a magnetic sensor 113 are connected to the main processor 140 as sensors. The main processor 140 is connected to a GPS receiver 115, a communication unit 117, a beacon receiver 119, an audio codec 180, an external connector 184, an external memory interface 186, a USB connector 188, a sensor hub 192, and an FPGA 194. These function as an interface with the outside.

  The main processor 140 is mounted on the controller board 120 built in the control device 10. In addition to the main processor 140, a memory 123, a beacon receiving unit 119, a nonvolatile storage unit 121, and the like may be mounted on the controller board 120. In the present embodiment, a six-axis sensor 111, a magnetic sensor 113, a GPS receiver 115, a communication unit 117, a memory 123, a nonvolatile storage unit 121, an audio codec 180, and the like are mounted on the controller board 120. Further, the external connector 184, the external memory interface 186, the USB connector 188, the sensor hub 192, the FPGA 194, and the interface 196 may be mounted on the controller board 120.

  The memory 123 constitutes a work area that temporarily stores a program to be executed and data to be processed when the main processor 140 executes the program. The non-volatile storage unit 121 is configured by a flash memory or an eMMC (embedded Multi Media Card). The nonvolatile storage unit 121 stores a program executed by the main processor 140 and various data processed by the main processor 140 executing the program.

The main processor 140 detects a touch operation on the operation surface of the track pad 14 based on an operation signal input from the track pad 14 and acquires an operation position.
The operation unit 110 includes a button 11 and an LED display unit 17. The operation unit 110 outputs an operation signal corresponding to the operated operation element to the main processor 140 when an operation element such as a button or a switch included in the button 11 is operated.

  The LED display unit 17 controls the turning on and off of the LED indicator 12 according to the control of the main processor 140. Further, the LED display unit 17 may have a configuration including an LED (not shown) arranged immediately below the track pad 14 (FIG. 1) and a drive circuit for lighting the LED. In this case, the LED display unit 17 turns on, blinks, and turns off the LED according to the control of the main processor 140.

The 6-axis sensor 111 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 111 may employ an IMU (Internal Measurement Unit) in which the above sensors are modularized.
The magnetic sensor 113 is, for example, a triaxial geomagnetic sensor.
The 6-axis sensor 111 and the magnetic sensor 113 output the detection value to the main processor 140 according to a sampling period specified in advance. The 6-axis sensor 111 and the magnetic sensor 113 output detection values to the main processor 140 at a timing designated by the main processor 140 in response to a request from the main processor 140.

  The GPS receiving unit 115 includes a GPS antenna (not shown) and receives a GPS signal transmitted from a GPS satellite. The GPS receiving unit 115 outputs the received GPS signal to the main processor 140. Further, the GPS receiving unit 115 measures the signal strength of the received GPS signal and outputs it to the main processor 140. Information such as received signal strength (RSSI: Received Signal Strength Indication), electric field strength, magnetic field strength, signal to noise ratio (SNR) can be used as the signal strength.

  The communication unit 117 performs wireless communication with an external device. The communication unit 117 includes an antenna, an RF circuit, a baseband circuit, a communication control circuit, and the like, or a device in which these are integrated. The communication unit 117 performs wireless communication complying with standards such as Bluetooth and wireless LAN (including Wi-Fi), for example.

The beacon receiving unit 119 receives a beacon signal transmitted from a beacon signal transmission device (not shown). The beacon receiving unit 119 detects a beacon ID included in the received beacon signal. The beacon ID is identification information of the transmission device. Further, the beacon receiving unit 119 detects the received signal strength indication (RSSI) of the received beacon signal, and estimates the distance from the transmission device based on the detected radio field strength. As the beacon signal, for example, a signal conforming to a short-range wireless communication standard such as NFC (Near Field Communication), Bluetooth, Bluetooth Low Energy, Bluetooth Smart, or iBeacon (registered trademark) is used.
The beacon receiving unit 119 corresponds to the “distance detection unit” of the present invention.

  The audio interface 182 is an interface for inputting and outputting audio signals. In the present embodiment, the audio interface 182 includes a connector 46 (FIG. 1) provided on the connection cable 40. The connector 46 is connected to the headset 30. The audio signal output from the audio interface 182 is input to the right earphone 32 and the left earphone 34, whereby the right earphone 32 and the left earphone 34 output audio. Further, the microphone 63 provided in the headset 30 collects sound and outputs a sound signal to the sound interface 182. An audio signal input from the microphone 63 to the audio interface 182 is input to the external connector 184.

  The audio codec 180 is connected to the audio interface 182 and encodes and decodes an audio signal input / output via the audio interface 182. The audio codec 180 may include an A / D converter that converts analog audio signals to digital audio data, and a D / A converter that performs the reverse conversion. For example, the HMD 100 according to the present embodiment outputs sound by the right earphone 32 and the left earphone 34 and collects sound by the microphone 63. The audio codec 180 converts the digital audio data output from the main processor 140 into an analog audio signal and outputs the analog audio signal via the audio interface 182. The audio codec 180 converts an analog audio signal input to the audio interface 182 into digital audio data and outputs the digital audio data to the main processor 140.

  The external connector 184 is a connector that connects an external device that communicates with the main processor 140. The external connector 184 is an interface for connecting an external device when, for example, an external device is connected to the main processor 140 to debug a program executed by the main processor 140 or collect a log of the operation of the HMD 100. It is.

  The external memory interface 186 is an interface to which a portable memory device can be connected. For example, the external memory interface 186 includes a memory card slot in which a card type recording medium is mounted and data can be read and an interface circuit. In this case, the size, shape and standard of the card type recording medium are not limited and can be changed as appropriate.

  The USB (Universal Serial Bus) connector 188 includes a connector conforming to the USB standard and an interface circuit. The USB connector 188 can connect a USB memory device, a smartphone, a computer, or the like. The size and shape of the USB connector 188 and the compatible USB standard version can be appropriately selected and changed.

  The sensor hub 192 and the FPGA 194 are connected to the image display unit 20 via an interface (I / F) 196. The sensor hub 192 acquires detection values of various sensors included in the image display unit 20 and outputs them to the main processor 140. The FPGA 194 executes processing of data transmitted / received between the main processor 140 and each unit of the image display unit 20 and transmission via the interface 196.

  The HMD 100 includes a vibrator 19. The vibrator 19 includes a motor and an eccentric rotor (both not shown), and may have other necessary configurations. The vibrator 19 generates vibrations by rotating the motor according to the control of the main processor 140. For example, the HMD 100 generates vibration by the vibrator 19 with a predetermined vibration pattern when an operation on the operation unit 110 is detected, when the power of the HMD 100 is turned on / off, or in other cases.

  The right display unit 22 and the left display unit 24 of the image display unit 20 are each connected to the control device 10. As shown in FIG. 1, in the HMD 100, a connection cable 40 is connected to the left holding unit 23, and wirings connected to the connection cable 40 are laid inside the image display unit 20, and each of the right display unit 22 and the left display unit 24. Is connected to the control device 10.

The right display unit 22 includes a display unit substrate 210. On the display unit substrate 210, an interface (I / F) 211 connected to the interface 196, a receiving unit (Rx) 213 that receives data input from the control device 10 via the interface 211, and an EEPROM 215 are mounted. .
The interface 211 connects the receiving unit 213, the EEPROM 215, the temperature sensor 217, the camera 61, the illuminance sensor 65, and the LED indicator 67 to the control device 10.

  An EEPROM (Electrically Erasable Programmable Read-Only Memory) 215 stores various data so that the main processor 140 can read the data. The EEPROM 215 stores, for example, data relating to light emission characteristics and display characteristics of the OLED units 221 and 241 provided in the image display unit 20, data relating to characteristics of sensors provided in the right display unit 22 or the left display unit 24, and the like. Specifically, parameters relating to gamma correction of the OLED units 221, 241 and data for compensating the detection values of the temperature sensors 217, 239 are stored. These data are generated by the factory inspection of the HMD 100 and written in the EEPROM 215. After the shipment, the main processor 140 can perform processing using the data in the EEPROM 215.

The camera 61 executes imaging according to a signal input via the interface 211 and outputs captured image data or a signal indicating the imaging result to the control device 10.
As shown in FIG. 1, the illuminance sensor 65 is provided at the end ER of the front frame 27 and is arranged to receive external light from the front of the user wearing the image display unit 20. The illuminance sensor 65 outputs a detection value corresponding to the amount of received light (received light intensity).
As shown in FIG. 1, the LED indicator 67 is disposed near the camera 61 at the end ER of the front frame 27. The LED indicator 67 is lit during execution of imaging by the camera 61 to notify that imaging is in progress.

  The temperature sensor 217 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 217 is mounted on the back side of the OLED panel 223 (FIG. 2). For example, the temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223.

  The receiving unit 213 receives data transmitted from the main processor 140 via the interface 211. When receiving the image data of the image displayed on the OLED unit 221, the receiving unit 213 outputs the received image data to the OLED drive circuit 225 (FIG. 2).

The left display unit 24 includes a display unit substrate 210. On the display unit substrate 210, an interface (I / F) 231 connected to the interface 196 and a receiving unit (Rx) 233 that receives data input from the control device 10 via the interface 231 are mounted. In addition, a six-axis sensor 235 and a magnetic sensor 237 are mounted on the display unit substrate 210.
The interface 231 connects the receiving unit 233, the six-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 to the control device 10.

The 6-axis sensor 235 is a motion sensor (inertial sensor) including a 3-axis acceleration sensor and a 3-axis gyro (angular velocity) sensor. The 6-axis sensor 235 may employ an IMU in which the above sensors are modularized. The 6-axis sensor 235 corresponds to the “detection unit” of the present invention.
The magnetic sensor 237 is, for example, a triaxial geomagnetic sensor.

The temperature sensor 239 detects the temperature and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 239 is mounted on the back side of the OLED panel 243 (FIG. 2). The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243.
Further, the temperature sensor 239 may be incorporated in the OLED panel 243 or the OLED drive circuit 245. The substrate may be a semiconductor substrate. Specifically, when the OLED panel 243 is mounted as an Si-OLED as an integrated circuit on an integrated semiconductor chip together with the OLED drive circuit 245 and the like, the temperature sensor 239 may be mounted on the semiconductor chip.

The camera 61, the illuminance sensor 65, the temperature sensor 217, and the 6-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 included in the left display unit 24 are connected to the sensor hub 192. The sensor hub 192 sets and initializes the sampling period of each sensor according to the control of the main processor 140. The sensor hub 192 executes energization to each sensor, transmission of control data, acquisition of a detection value, and the like in accordance with the sampling period of each sensor. The sensor hub 192 outputs detection values of the sensors included in the right display unit 22 and the left display unit 24 to the main processor 140 at a preset timing. The sensor hub 192 may have a function of temporarily holding the detection value of each sensor in accordance with the output timing to the main processor 140. In addition, the sensor hub 192 may have a function of converting the data of the output value of each sensor or the data format into data of a unified data format and outputting the data to the main processor 140.
The sensor hub 192 starts and stops energization of the LED indicator 67 according to the control of the main processor 140, and lights or blinks the LED indicator 67 in accordance with the timing when the camera 61 starts and ends imaging.

  The control device 10 includes a power supply unit 130 and operates with power supplied from the power supply unit 130. The power supply unit 130 includes a rechargeable battery 132 and a power supply control circuit 134 that detects the remaining capacity of the battery 132 and controls charging of the battery 132. The power control circuit 134 is connected to the main processor 140 and outputs a detected value of the remaining capacity of the battery 132 or a detected value of the voltage to the main processor 140. Further, the power may be supplied from the control device 10 to the image display unit 20 based on the power supplied from the power supply unit 130. The power supply state from the power supply unit 130 to each unit of the control device 10 and the image display unit 20 may be configured to be controllable by the main processor 140.

  FIG. 5 is a functional block diagram of the storage unit 125 and the control unit 150 that constitute the control system of the control device 10. The storage unit 125 illustrated in FIG. 5 is a logical storage unit configured by the nonvolatile storage unit 121 (FIG. 4), and may include an EEPROM 215. The control unit 150 and various functional units included in the control unit 150 are formed by cooperation of software and hardware by the main processor 140 executing a program. Each functional unit constituting the control unit 150 and the control unit 150 includes, for example, a main processor 140, a nonvolatile storage unit 121, and a memory 123.

The control unit 150 executes various processes using data stored in the storage unit 125 and controls the HMD 100.
The storage unit 125 stores various data processed by the control unit 150. The storage unit 125 stores setting data 127 and content data 128.
The setting data 127 includes various setting values that set the operation of the HMD 100. When the control unit 150 controls the HMD 100, when using parameters, determinants, arithmetic expressions, LUT (LookUp Table), etc., these may be included in the setting data 127.

The content data 128 is content data including images and videos displayed by the image display unit 20 under the control of the control unit 150, and includes image data or video data. Further, the content data 128 may include audio data. The content data 128 may include image data of a plurality of images. In this case, the plurality of images are not limited to images displayed on the image display unit 20 at the same time.
The content data 128 is an interactive content in which when the content is displayed on the image display unit 20, the control device 10 receives a user operation and the control unit 150 executes a process according to the received operation. It may be. In this case, the content data 128 may include image data of a menu screen that is displayed when an operation is accepted, data that defines processing corresponding to items included in the menu screen, and the like.

The control unit 150 has functions of an operating system (OS) 143, an image processing unit 145, an imaging control unit 147, an operation control unit 149, a detection control unit 151, and a display control unit 153. The operation control unit 149 and the detection control unit 151 correspond to the “detection unit” of the present invention.
The functions of the operating system 143 are functions of a control program stored in the storage unit 125, and the other units are functions of application programs executed on the operating system 143.

The image processing unit 145 generates a signal to be transmitted to the right display unit 22 and the left display unit 24 based on image data of an image or video displayed by the image display unit 20. The signal generated by the image processing unit 145 may be a vertical synchronization signal, a horizontal synchronization signal, a clock signal, an analog image signal, or the like.
Further, the image processing unit 145 may perform resolution conversion processing for converting the resolution of the image data to a resolution suitable for the right display unit 22 and the left display unit 24 as necessary. The image processing unit 145 also performs image adjustment processing for adjusting brightness and saturation of image data, 2D image data is created from 3D image data, or 2D / 3D conversion processing for generating 3D image data from 2D image data, etc. May be executed. When these image processes are executed, the image processing unit 145 generates a signal for displaying an image based on the processed image data, and transmits the signal to the image display unit 20 via the connection cable 40.
The image processing unit 145 may be configured by hardware different from the main processor 140 (for example, a digital signal processor (DSP)), in addition to the configuration realized by the main processor 140 executing a program.

  The imaging control unit 147 controls the camera 61 to execute imaging, generates captured image data, and temporarily stores it in the storage unit 125. When the camera 61 is configured as a camera unit including a circuit that generates captured image data, the imaging control unit 147 acquires captured image data from the camera 61 and temporarily stores the captured image data in the storage unit 125.

The operation control unit 149 detects an operation on the track pad 14 and the operation unit 110 and outputs operation data corresponding to the operation. For example, when a button or the like of the operation unit 110 is operated, the operation control unit 149 generates operation data indicating the operation content and outputs the operation data to the display control unit 153. The display control unit 153 changes the display state according to the operation data input from the operation control unit 149.
Further, the operation control unit 149 acquires the coordinates of the operation position on the track pad 14 when the operation on the track pad 14 is detected. The operation control unit 149 generates a locus of the operation position. The locus of the operation position is a locus of the operation position while the touch operation of the track pad 14 is not released, that is, while the state where the user's hand or finger operation body is in contact with the track pad 14 continues. The operation control unit 149 outputs operation data indicating the locus of the detected operation position to the display control unit 153. The operation control unit 149 corresponds to the “detection unit” of the present invention.

The detection control unit 151 detects the movement of the user's head based on the detection value of the 6-axis sensor 235. The 6-axis sensor 235 performs sampling every predetermined cycle (for example, 50 msec), and outputs an analog voltage value indicating a detection result. The output voltage value is converted into a digital signal by an A / D converter (not shown) and output to the control unit 150.
The angular velocity sensor of the 6-axis sensor 235 is configured to rotate around the X axis (pitch angle), rotate around the Y axis (yaw angle), and rotate around the Z axis (shown in FIG. 1) at the measurement reference point of the built-in detection mechanism. Roll angle) is detected. The X-axis, Y-axis, and Z-axis are three axial directions orthogonal to each other as shown in FIG. 1, the Z-axis direction corresponds to the vertical direction, and the X-axis direction corresponds to the left-right direction of the user's head. The Y-axis direction corresponds to the front-rear direction of the user's head.
The detection control unit 151 detects the rotation (pitch angle) around the X axis based on the detection value of the 6-axis sensor 235 and detects the movement of the user raising or lowering the face. Further, the detection control unit 151 detects a rotation (yaw angle) around the Y axis based on a detection value of the 6-axis sensor 235, and detects a movement in which the user tilts the neck toward the left shoulder or the right shoulder. Further, the detection control unit 151 detects a rotation (roll angle) around the Z axis based on a detection value of the 6-axis sensor 235, and detects a movement of the user turning the face left or right. Hereinafter, the movement of raising and lowering the face, the movement of tilting the neck toward the left shoulder or the right shoulder, and the movement of turning the face left or right are collectively referred to as the tilt of the head.
The detection control unit 151 generates information indicating the direction of head tilt and the angle of the tilted head as operation data based on the detection value of the 6-axis sensor 235, and the generated operation data is displayed on the display control unit 153. Output to.

  The display control unit 153 generates control signals for controlling the right display unit 22 and the left display unit 24, and controls the generation and emission of image light by the right display unit 22 and the left display unit 24 based on the control signals. . Specifically, the display control unit 147 controls the OLED drive circuits 225 and 245 to cause the OLED panels 223 and 243 to display an image. The display control unit 147 controls the timing at which the OLED drive circuits 225 and 245 draw on the OLED panels 223 and 243, controls the luminance of the OLED panels 223 and 243, and the like based on the signal output from the image processing unit 145.

Further, the display control unit 153 moves the position of the pointer 301 (FIG. 6) displayed on the display area 310 according to the operation data input from the operation control unit 149. A pointer 301 indicates a designated position in the display area 310 and corresponds to a “marker” of the present invention. Further, instead of the pointer 301, a cursor, a mark, a character, a symbol, a sign, an arrow, or a game character can be used as a marker.
The display control unit 153 receives data indicating the movement distance on the operation surface and the movement direction as operation data from the operation control unit 149. The display control unit 153 generates a control signal indicating a moving direction and a moving distance for moving the pointer 301 based on the input operation data, and outputs the generated control signal to the right display unit 22 and the left display unit 24. To do.
Further, the display control unit 153 moves the position of the pointer 301 (FIG. 6) displayed in the display area 310 according to the operation data input from the detection control unit 151. Data indicating the direction in which the head tilt is detected and the angle of the tilted head are input to the display control unit 153 as operation data from the detection control unit 151. The display control unit 153 generates a control signal indicating a moving direction and a moving distance for moving the pointer 301 based on the input operation data.
For example, the display control unit 153 specifies a direction in which the head tilt is detected from the input operation data, and generates a control signal for moving the pointer 301 in the direction of the display area 310 corresponding to the specified direction. May be. Further, the display control unit 153 may specify the direction in which the tilt of the head is detected from the operation data and also specify the angle of the tilted head. Then, the display control unit 153 generates a control signal for controlling the pointer 301 so that the pointer 301 is moved in the direction of the display area 310 corresponding to the specified direction and moved at a speed corresponding to the specified head angle.
The display control unit 153 outputs the generated control signal to the right display unit 22 and the left display unit 24.

In addition, the display control unit 153 performs calibration and generates a parameter that associates the coordinates of the display area 310 with the captured image of the camera 61.
The display control unit 153 recognizes and tracks the reference real object corresponding to the calibration image from the captured image data of the camera 61 while displaying the calibration image on the image display unit 20. In this state, the user moves the position of the calibration image on the image display unit 20 via the user interface such as the track pad 14. Then, at the timing when the user perceives that the calibration image and the reference real object overlap (at least one of position, size, and orientation substantially coincides), the display control unit 153 is connected via the user interface. Notice. In accordance with the acquired notification, the display control unit 153 sets a parameter that associates the detection result of the reference real object (position on the captured image data) at the timing with the display position of the calibration image in the display area 310. Generate. The display control unit 153 stores the generated parameter in the storage unit 125.
This parameter is used when the target object selected by the pointer 301 is specified. The control unit 150 acquires captured image data of the camera 61, converts the coordinates of the display area 310 on which the pointer 301 is displayed, into coordinates on the captured image data, and identifies the target object.

  The HMD 100 may include an interface (not shown) for connecting various external devices that are content supply sources. For example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface may be used, or a wireless communication interface may be used. The external device in this case is an image supply device that supplies an image to the HMD 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used. In this case, the HMD 100 can output images and sounds based on content data input from these external devices.

FIG. 6 is a diagram illustrating a field of view 300 that the user visually recognizes through the image display unit 20.
The user includes a person existing in the real world (hereinafter referred to as an outside scene), a building, a road, and other objects (hereinafter collectively referred to as a target object) and a display image displayed by the image display unit 20. It is visually recognized through the image display unit 20. The display image displayed by the image display unit 20 includes, for example, a pointer 301 and contents such as images and videos. When the image display unit 20 reproduces the content data under the control of the control unit 150, the content is reproduced in the display area 310. A display area 310 indicates an area in which the image display unit 20 can display a display image.

  The user performs the first operation and the second operation to move the pointer 301 to a target position (hereinafter referred to as a target). For example, when the user purchases a beverage from the vending machine 320, the target is a product button of the product to be purchased. The user moves the pointer 301 displayed in the display area 310 to the target position by the first operation and the second operation.

In the present embodiment, the position of the pointer 301 is moved according to the first operation, and the position of the pointer 301 is moved according to the second operation. For example, when the pointer 301 is moved according to the first operation and when the pointer 301 is moved according to the second operation, the amount of movement of the pointer 301 (for example, movement distance, movement speed, acceleration, etc.) is changed. change.
When the movement distance for moving the pointer 301 by operation is set small or when the movement speed is set slow, if the distance between the position of the pointer 301 and the target is far, the pointer 301 is moved to the vicinity of the target. take time. In addition, when the movement distance for moving the pointer 301 by an operation is set to be large or the movement speed is set to be high, it may be difficult to finely adjust the pointer 301 to the target position.
For this reason, the HMD 100 according to the present embodiment moves the pointer 301 to the target position by two operations of the first operation and the second operation in which at least one of the moving distance, moving speed, and acceleration of the pointer 301 is different. . The target may be a display image displayed by the image display unit 20 or a target object visually recognized as an outside scene.

The first operation is performed before the second operation. The first operation is an operation for moving the pointer 301 to the vicinity of the target in a short time. The second operation is an operation for finely adjusting the position of the pointer 301 and moving the pointer 301 to the target position.
For example, it is assumed that the first operation and the second operation are pressing operations of switches and buttons. In this case, the movement distance, moving speed, or acceleration for moving the pointer 301 when the pressing operation by the first operation is received once, the moving distance by which the pointer 301 is moved when the pressing operation by the second operation is received once. Set larger than the moving speed or acceleration. The movement distance, the movement speed, or the acceleration corresponds to the “movement amount” of the present invention.
Further, it is assumed that the first operation and the second operation are contact operations on the operation surface, and are operations for moving the pointer 301 according to the detected contact distance of the finger on the operation surface. In this case, the movement distance, movement speed, or acceleration for moving the pointer 301 corresponding to the finger contact distance as the first operation is the movement distance for moving the pointer 301 corresponding to the finger contact distance as the second operation. Set larger or faster than the moving speed or acceleration.
In addition, the first operation and the second operation are a touch operation on the operation surface or an operation of moving the pointer 301 by detecting the tilt of the user's head, and detecting the touch operation or the tilt of the head being detected. It is assumed that the operation is to move the pointer 301 by a distance corresponding to time. In this case, the moving distance, moving speed, or acceleration for moving the pointer 301 corresponding to the detection time of the first operation is obtained from the moving distance, moving speed, or acceleration for moving the pointer 301 corresponding to the detection time of the second operation. Also set larger or faster.

In the present embodiment, an operation for moving the pointer 301 in accordance with the operation of the track pad 14 is a first operation. When the pointer 301 is moved to the target position by operating the track pad 14, it is not easy to move the pointer 301 as the user imagines if the track pad 14 is not fixed. For this reason, in this embodiment, the operation of the track pad 14 is used as an operation of moving the pointer 301 in the vicinity of the target.
In the present embodiment, the operation of detecting the tilt of the user's head as an operation and moving the pointer 301 in accordance with the detected operation is referred to as a second operation. The operation of the pointer 301 performed by the user tilting his / her head corresponds to the “user operation” of the present invention. When operating the pointer 301 with the head tilted, the user has to keep tilting the head until the operation is completed, and sometimes has to maintain an unreasonable posture. However, in the present embodiment, since the pointer 301 has already been moved in the vicinity of the target by the first operation performed before the second operation, the time for performing the second operation can be shortened.

6 represents the horizontal direction of the display area 310, the y axis represents the depth direction of the display area 310, and the z axis represents the vertical direction (vertical direction) of the display area 310.
When rotation around the X axis of the user's head shown in FIG. 1 is detected, for example, the control unit 150 moves the pointer 301 in the z axis direction shown in FIG. 6, that is, in the vertical direction of the display area 310. Further, when the rotation of the user's head around the Y axis shown in FIG. 1 is detected, the control unit 150 moves the pointer 301 in the y axis direction shown in FIG. Let 1 is detected, the control unit 150 moves the pointer 301 in the x-axis direction shown in FIG. 6, that is, in the horizontal direction of the display area 310, for example. Let
The association between the direction in which the user tilts the head and the moving direction in which the pointer 301 is moved is not limited to the above-described one, and can be arbitrarily set.

In the present embodiment, the timing for switching the operation for moving the pointer 301 from the first operation to the second operation is detected whether or not the user's hand or finger (hereinafter referred to as a finger) has moved away from the track pad 14. Do that. The control unit 150 determines that the user intends to operate the track pad 14 while detecting contact with the operation surface of the track pad 14, and moves the pointer 301 by operating the track pad 14. In addition, when the contact with the operation surface cannot be detected and the finger is separated from the operation surface, the control unit 150 determines that the willingness to operate the track pad 14 is lost, and performs an operation for moving the pointer 301 as the second operation. Switch to.
Further, switching from the first operation to the second operation is not limited to whether or not contact with the operation surface is detected. For example, when the preset button 11, the up / down key 15, the changeover switch 16 is operated, or when an operation of tapping the track pad 14 for a preset number of times is accepted, the first operation is switched to the second operation. May be. In addition, the first operation may be switched to the second operation by sound input from the microphone 63. For example, when a finger snap sound or a word uttered by the user is input from the microphone 63 and the input voice data matches the preset voice data, the first operation may be switched to the second operation. .

In this embodiment, the operation of the pointer 301 by the track pad 14 is the first operation, and the operation of the pointer 301 based on the tilt of the user's head is the second operation. However, the first operation and the second operation are It is not limited to the operation of. For example, the operation of moving the pointer 301 by detecting the tilt of the user's head may be the first operation, and the operation of the pointer 301 by the track pad 14 may be the second operation. Moreover, you may use a hand sign and a gesture for any one of 1st operation and 2nd operation. The hand sign and the gesture correspond to the “user action” of the present invention. When the first or second operation is performed by hand sign or gesture, a dictionary used for specifying the hand sign or gesture is stored in the storage unit 125 in advance. For example, data indicating the position and shape of the finger is registered in the dictionary for each hand sign. In addition, if the gesture is to perform preset hand shapes (forms) in the preset order, data indicating the finger position and shape in each shape (form) is registered in the dictionary for each gesture. The The control unit 150 detects a region where the user's hand or finger is imaged (hereinafter referred to as a finger region) from the captured image of the camera 61, detects the shape or position of the finger from the detected finger region, and stores it in the dictionary. Recognize hand signs and gestures by matching with registered data.
In addition, the control unit 150 detects the movement of the fingernail of the user's hand, the ring fitted on the finger of the user, the bracelet fitted on the arm, the watch, or the band as one of the first operation and the second operation. May be. The control unit 150 detects movements of the user's fingernails, a ring fitted on the finger of the user, a bracelet fitted on the arm, a watch, and a band from the captured image of the camera 61, and according to the detected movement. The pointer 301 is moved.

The control unit 150 includes a first operation mode and a second operation mode as operation modes, and switches between these operation modes for execution. The first operation mode is a mode in which a target object (a vending machine 320 in FIG. 7) included in an outside scene visually recognized by the user through the image display unit 20 is a target, and the target object as a target is indicated by the pointer 301. is there. The second operation mode is a mode in which a plurality of display images displayed by the image display unit 20 are targeted, and the display image as the target is designated by the pointer 301.
Switching of the operation mode between the first operation mode and the second operation mode may be performed, for example, when an operation of the button 11, the up / down key 15, the changeover switch 16, or the like is received. The operation mode may be switched when an operation of hitting the track pad 14 a predetermined number of times is received.

Further, the HMD 100 is equipped with a line-of-sight sensor (not shown), and the first operation mode and the second operation mode are determined depending on whether the target object exists in the line-of-sight direction detected by the line-of-sight sensor or the display image exists. May be switched.
Here, the line-of-sight sensor will be described.
A pair of line-of-sight sensors 68 are provided at the center positions of the right light guide plate 26 and the left light guide plate 28 so as to correspond to the right eye RE and the left eye LE of the user, respectively. The line-of-sight sensor includes, for example, a pair of cameras that capture images of the right eye RE and left eye LE of the user. The line-of-sight sensor captures images according to the control of the control unit 150, and the control unit 150 detects reflected light and pupil images on the eyeball surfaces of the right eye RE and the left eye LE from the captured image data, and identifies the line-of-sight direction.
When the target object exists in the line-of-sight direction detected by the line-of-sight sensor, the control unit 150 sets the operation mode of the HMD 100 to the first operation mode. Moreover, the control part 150 sets the operation mode of HMD100 to a 2nd operation mode, when a display image exists in the gaze direction detected by a gaze sensor.

7 and 8 are views showing the field of view 300 viewed by the user with the neck tilted to the left. FIG. 7 shows a state where the vending machine 320 can be seen as a target object in the outside scene, and FIG. 8 shows a state where a display screen 302 of word processor software can be seen as a display image displayed by the image display unit 20.
When the user operates the pointer 301 by the second operation, the user may perform an operation of tilting the neck to the left shoulder side or the right shoulder side. At this time, the display area 310 is also tilted according to the tilt of the user's neck, but the target object seen through the image display unit 20 is not tilted. Therefore, as shown in FIG. 320 appears to tilt.
Further, the control unit 150 adjusts the inclination of the display image displayed by the image display unit 20 based on the detection value of the 6-axis sensor 235. That is, the display image is tilted according to the tilt of the head as shown in FIG. 8, and the display image looks horizontal to the user.

When the operation mode is the first operation mode and the inclination of the image display unit 20 is detected based on the detection value of the six-axis sensor 235, the control unit 150 moves the pointer 301 to the target object according to the detected inclination. And move it horizontally. That is, in the first operation mode, the pointer 301 is moved with reference to the target object. FIG. 7 shows a state in which the pointer 301 is moved horizontally with respect to the vending machine 320 that is the target object.
Further, when the operation mode is the second operation mode and the inclination of the image display unit 20 is detected based on the detection value of the 6-axis sensor 235, the control unit 150 moves the pointer 301 in accordance with the detected inclination. That is, in the second operation mode, the pointer 301 is moved with reference to the display image. FIG. 8 shows a state in which the pointer 301 is moved in accordance with the inclination of the display screen 302 of the word processor software that is a display image.

FIG. 9 is a diagram illustrating a display image displayed when the pointer 301 is moved to the vicinity of the target by the first operation.
When the pointer 301 is operated by the first operation to move the pointer 301 close to the target, the target existing near the pointer 301 is enlarged and displayed as shown in FIG.
The control unit 150 acquires captured image data captured by the camera 61 and detects a target object from the acquired captured image data. When the control unit 150 detects a target object within a preset range from the display position of the pointer 301, the control unit 150 extracts an image of the target object from the captured image data, enlarges the extracted target object image, and displays the display region 310. To display.
In addition, when the target object is detected within a preset range from the display position of the pointer 301, the control unit 150 causes the display area 310 to display a display for inquiring whether to enlarge the target object image. May be. For example, when the instruction to enlarge the target object image is received by the button 11 or the track pad 14, the control unit 150 enlarges the target object image and displays it on the display area 310. An image of the target object displayed in an enlarged manner is referred to as an enlarged image 303.

The user finely adjusts the position of the pointer 301 by the second operation so that the pointer 301 overlaps the target while the enlarged image 303 is displayed. When the pointer 301 overlaps the target, the user performs a determination operation. When accepting the confirmation operation, control unit 150 determines that the target overlapping the display position of pointer 301 has been selected. For the confirmation operation, an operation other than an operation for detecting the tilt of the user's head and moving the pointer 301 as the second operation is used. The confirmation operation may be, for example, an operation of hitting the operation surface of the track pad 14, or an operation of the button 11, the up / down key 15, the changeover switch 16, and the like.
Further, the HMD 100 may determine that the confirmation operation has been received when receiving a signal transmitted from an external device wirelessly connected. For example, a wearable terminal on which a user's arm or the like is worn can be used as the external device. When the pointer 301 overlaps the target, the user inputs an operation preset by the wearable terminal. When the wearable terminal receives a preset operation, the wearable terminal transmits a predetermined signal to the HMD 100. When receiving a predetermined signal from the wearable terminal, the control unit 150 of the HMD 100 determines that a confirmation operation has been received.

FIG. 10 is a diagram illustrating the relationship between the distance to the target and the movement distance of the pointer 301.
In the above description, the moving distance, moving speed, and acceleration of the pointer 301 are changed between the case where the pointer 301 is operated by the first operation and the case where the pointer 301 is operated by the second operation.
As other operations, the movement distance, the movement speed, and the acceleration, which are modes of movement of the pointer 301, may be changed according to the distance between the HMD 100 and the target. For example, when the target includes a beacon transmission unit 321 that transmits a beacon signal as illustrated in FIG. 10, the HMD 100 receives a beacon signal transmitted from the target by the beacon reception unit 119. The control unit 150 estimates the distance to the target based on the radio wave intensity of the received beacon signal. The control unit 150 changes the moving distance, moving speed, and acceleration of the pointer 301 that is moved by the first or second operation according to the estimated distance to the target.

In the present embodiment, as shown in FIG. 10, the movement distance of the pointer 301 when the distance between the HMD 100 and the target is a distance A is larger than the movement distance of the pointer 301 when the distance to the target is a distance B. Set smaller. Assume that distance A is greater than the threshold and distance B is less than the threshold.
For example, the control unit 150 compares the distance estimated based on the radio wave intensity of the beacon signal with a preset threshold value, and the distance between the HMD 100 and the target is greater than the distance set as the threshold value. Determine whether it is larger.
When the distance between the HMD 100 and the target is larger (separated) than the threshold value, the control unit 150 detects the operation of the track pad 14 as the first operation, and the pointer 301 of the pointer 301 is detected according to the detected first operation. Move position. In addition, when the distance from the target is larger (separated) from the target, the control unit 150 sets the moving distance of the pointer 301 moved by the first operation as the first distance.
Further, when the distance from the target is equal to or less than the threshold value (close), the control unit 150 detects the tilt of the head, which is the second operation, and moves the position of the pointer 301 according to the detected tilt. In addition, when the distance from the target is equal to or smaller than the threshold value, the control unit 150 sets the distance of the pointer 301 moved by the second operation as the second distance.
The second distance is set to a value larger than the first distance. That is, the moving distance of the pointer 301 is set to be larger as the distance between the HMD 100 and the target is shorter.

  When the HMD 100 and the target are separated from each other, the size of the target (the vending machine 320 or the product button) in the user's field of view 300 is small. If the moving distance of the pointer 301 by the first operation is set to be large, the pointer 301 may be greatly deviated from the target. For this reason, when the distance between the HMD 100 and the target is larger than the threshold value, the moving distance of the pointer 301 is set small. When the HMD 100 and the target are close to each other, the size of the target in the user's field of view 300 is large. For this reason, even if the moving distance of the pointer 301 is set to be large, the pointer 301 is hardly deviated from the target. For this reason, when the distance between the HMD 100 and the target is closer than the threshold value, the moving distance of the pointer 301 is set large.

Further, the moving distance of the pointer 301 may be continuously changed according to the distance between the target and the HMD 100. The control unit 150 estimates the distance to the target based on the radio wave intensity of the beacon signal, and changes the moving distance for moving the pointer 301 by the first or second operation according to the estimated distance. The control unit 150 sets the distance to move the pointer 301 by the first or second operation as the distance from the target is shorter.
In contrast to the above description, the distance by which the pointer 301 is moved by the first or second operation may be set to be smaller as the distance between the HMD 100 and the target is shorter.
In the above description, the first operation is selected when the distance to the target is greater than the threshold value, and the second operation is selected when the distance to the target is less than or equal to the threshold value. In addition to this, the position of the pointer 301 may be moved by any one of the first operation and the second operation. That is, the pointer 301 may be operated by either the first operation or the second operation both when the distance to the target is larger than the threshold and when the distance is less than the threshold.
In the description of FIG. 10, the distance from the HMD 100 is estimated based on the radio wave intensity of the beacon signal transmitted from the target. However, when the HMD 100 includes a stereo camera as the camera 61, the distance from the target to the target is determined by the captured image of the camera 61. The distance may be obtained.

FIG. 11 is a flowchart showing the operation of the HMD 100.
The controller 150 determines whether or not a contact operation on the operation surface of the track pad 14 has been detected (step S1). The control unit 150 determines whether or not a contact operation is detected based on the operation signal input from the track pad 14 (step S1). If the contact operation is not detected (step S1 / NO), control unit 150 waits until a contact operation is detected, and executes this process if there is another process that can be executed. Moreover, the control part 150 moves the pointer 301, when a contact operation is detected (step S1 / YES) (step S2). The moving distance by which the control unit 150 moves the pointer 301 is a distance corresponding to the moving distance of the user's finger on the operation surface, and the moving direction corresponds to the moving direction of the user's finger on the operation surface. Direction.

Next, the control unit 150 determines whether or not the user's finger or the like is in contact with the operation surface (step S3). In this embodiment, while the contact with the operation surface such as a finger is continued, the pointer 301 is moved by the contact operation with respect to the operation surface which is the first operation. When contact with the operation surface such as a finger is no longer detected, the control unit 150 switches the operation of the pointer 301 from the first operation to the second operation. In the second operation, the control unit 150 detects the tilt of the user's head and moves the pointer 301 according to the detected tilt of the head.
When detecting a contact with the operation surface (step S3 / YES), the control unit 150 returns to step S1 to determine whether or not a contact operation has been detected (step S1), and the pointer 301 is determined according to the detected contact operation. Is moved (step S2).

In addition, when the contact with the operation surface cannot be detected (step S3 / NO), the control unit 150 switches the operation of the pointer 301 to the second operation. First, the control unit 150 acquires the detection value of the 6-axis sensor 235 (step S4).
The control unit 150 determines whether or not the tilt of the user's head has been detected based on the acquired detection value of the 6-axis sensor 235 (step S5). When it is determined that the user's head is not tilted (step S5 / NO), the control unit 150 determines whether an end operation for ending the operation of the pointer 301 has been received (step S6). If the end operation has not been received (step S6 / NO), the control unit 150 returns to the determination of step S4 and continues to detect the tilt of the user's head based on the detection value of the 6-axis sensor 235 (step S5). . Moreover, the control part 150 complete | finishes this process flow, when completion | finish operation is received (step S6 / YES).

  When it is determined that the user's head is tilted (step S5 / YES), the control unit 150 determines whether the image display unit 20 is tilted left or right (step S7). ). In this determination, it is determined whether or not the user has moved the head to the left or right. When the user's head tilts to the left or right, the target object in the outside scene that the user visually recognizes through the image display unit 20 is viewed in a state tilted to the left or right. Therefore, when it is determined that the image display unit 20 is tilted left and right, the control unit 150 controls the movement of the pointer 301 so as to correspond to the tilt of the image display unit 20.

When it is determined that the image display unit 20 is not tilted left or right (step S7 / NO), the control unit 150 detects the tilt of the user's head based on the detection value of the six-axis sensor 235, The pointer 301 is moved in the direction corresponding to the detected inclination (step S8).
For example, when an operation of turning the user's face upward is detected, the control unit 150 moves the pointer 301 upward, which is the positive direction of the z axis shown in FIG. In addition, when an operation of turning the user's face downward is detected, the control unit 150 moves the pointer 301 downward, which is the negative direction of the z axis.
When the user's face is turned to the left, that is, when a left rotation around the Z axis is detected, the control unit 150 moves the pointer 301 to the left, which is the positive direction of the x axis shown in FIG. When the user's face is turned to the right, that is, when a right rotation around the Z axis is detected, the control unit 150 moves the pointer 301 to the right, which is the negative direction of the x axis shown in FIG.
In addition, when moving the pointer 301 with reference to the target object, the control unit 150 may move the pointer 301 horizontally based on the detection value of the magnetic sensor 237.

  Further, when the control unit 150 determines that the image display unit 20 is tilted left or right (step S7 / YES), it is determined whether or not the operation mode of the control unit 150 is the first operation mode. Determine (step S9). In the first operation mode, when the pointer 301 is moved in the horizontal direction, the pointer 301 is moved so as to be horizontal with respect to the outside scene. In the state where the user's head is tilted to the left or right, the target object in the outside scene that can be seen through the image display unit 20 also appears to tilt. Therefore, when the operation mode is the first operation mode (step S9 / YES), the control unit 150 moves the pointer 301 with reference to the target object (step S10). That is, the control unit 150 adjusts the display so that the pointer 301 moves horizontally with respect to the target object in the outside scene (step S10). The control unit 150 detects the rotation angle around the Y axis based on the detection value of the 6-axis sensor 235 and determines the tilt of the image display unit 20. Based on the determined inclination of the image display unit 20, the control unit 150 adjusts the display so that the pointer 301 moves horizontally with respect to the target object (step S10).

  Further, when determining that the operation mode is not the first operation mode (step S9 / NO) and the second operation mode, the control unit 150 moves the pointer 301 with reference to the display image (step S9 / NO). S11). That is, the control unit 150 adjusts the display so that the pointer 301 is horizontal with respect to the display image. When the pointer 301 is moved, the control unit 150 returns to step S4 to newly acquire the detection value of the 6-axis sensor 235 and detects the tilt of the head (step S5).

  When the head tilt is not detected in the determination in step S5 (step S5 / NO), the control unit 150 determines whether or not a confirmation operation has been accepted (step S6). For example, the control unit 150 determines that the confirmation operation has been received when an operation of hitting the operation surface of the track pad 14 is received. When accepting the confirmation operation, the control unit 150 determines that the target at the position overlapping the pointer 301 has been selected, and executes processing according to the selected target. When the target is the target object, the control unit 150 acquires the parameter generated by the calibration, and based on the acquired parameter, the coordinate of the display area 310 on which the pointer 301 is displayed is the coordinate on the captured image data. Convert. And the control part 150 specifies the target object reflected in the coordinate on the converted captured image data, and performs the process according to the specified target object. For example, when the product button of the vending machine 320 is selected as a target, the control unit 150 performs processing for purchasing the product of the selected product button by performing wireless communication with the vending machine 320. In addition, when a display image is selected as the target, the control unit 150 executes processing associated with the display image. For example, when the display image is a menu screen and an item on the menu screen is selected by the pointer 301, the control unit 150 executes processing corresponding to the selected item.

FIG. 12 is a layout diagram illustrating the layout of the detection device mounted on the shoe sole or the shoe interior of the user.
The embodiment described above is configured to detect the first operation and the second operation by the HMD 100, but may be configured to detect the first operation or the second operation by an external device. FIG. 12 shows a configuration in which a detection device 330 is arranged in the shoe sole or inside of a user and the first operation or the second operation is detected by the detection device 330. The detection device 330 is, for example, a pressure sensor that detects pressure generated on the sole, and a communication device that performs wireless communication with the HMD 100 by short-range wireless communication and transmits a detection value detected by the pressure sensor to the HMD 100 (both not shown). ).

FIG. 12 shows an example in which the detection device 330 is arranged in the center of the shoe, in the sole of the right foot or inside the shoe. The user puts weight on the toe side (front side), the heel side (rear side), the left side, or the right side according to the direction in which the pointer 301 is moved. The detection device 330 detects the direction with the highest pressure among the four directions, front, rear, left, and right, and transmits a signal indicating the detected direction to the HMD 100. Based on the signal transmitted from the detection device 330, the HMD 100 determines which one of the four directions, front, rear, left, and right has been selected. The HMD 100 moves the pointer 301 in a direction corresponding to the determined direction.
For example, when the user puts his / her weight on the toe side, the control unit 150 moves the pointer 301 upward, which is the positive direction of the z axis shown in FIG. When the user puts his / her weight on the heel side, the control unit 150 moves the pointer 301 downward, which is the negative direction of the z axis. When the user puts his / her weight on the left side, the control unit 150 moves the pointer 301 to the left which is the positive direction of the x axis shown in FIG. When the user puts his / her weight on the right side, the control unit 150 moves the pointer 301 in the right direction, which is the negative direction of the x axis shown in FIG. For example, when the pointer 301 is desired to be moved in the depth direction (Y-axis direction) of the display area, the circular motion is performed so that the weight is applied in order in the four directions of front, rear, left and right. For example, when front, right, rear, and left rotations are detected, the control unit 150 moves the pointer 301 to the front side that is the positive direction of the y-axis illustrated in FIG. In addition, for example, when front, left, rear, and right left rotations are detected, the control unit 150 moves the pointer 301 to the back side, which is the negative direction of the y-axis illustrated in FIG.

FIG. 13 is a diagram illustrating another arrangement example of the detection devices.
FIG. 12 shows an example in which one detection device 330 is provided in the shoe sole or the inside of the shoe. However, as shown in FIG. 13, one detection device 331 is arranged in the center of the shoe sole or the inside of the shoe, and the toe side is arranged. Two detectors 332 and 333 may be arranged in the same. The configuration of the detection devices 331, 332, and 333 is the same as that of the detection device 330 shown in FIG. 12, but the detection devices 331, 332, and 333 are detected by pressure sensors included in the detection devices 331, 332, and 333, respectively. The detected pressure value is transmitted to the HMD 100.
When the user puts weight on the front (toe side), the detection values of the detection devices 332 and 333 increase, and the detection value of the detection device 331 decreases. Further, when the user puts his / her weight on the back (heel side), the detection value of the detection device 331 increases and the detection values of the detection devices 332 and 333 decrease.
Further, when the user puts his / her weight on the left side or the right side, the detection value of the detection device 332 and the detection value of the detection device 333 become significantly different values. When the user puts his / her weight on the left side, the detection value of the detection device 332 increases and the detection value of the detection device 333 decreases. When the user puts his / her weight on the right side, the detection value of the detection device 332 decreases and the detection value of the detection device 333 increases.

FIG. 14 is a diagram illustrating another example of the detection device.
The arrangement example of the detection device shown in FIG. 14 shows an example in which the detection devices 334 and 335 are arranged in the shoe soles of the left and right feet or inside the shoes, respectively. The configuration of the detection devices 334 and 335 is the same as that of the detection device 330 shown in FIG. 12, but the direction with the highest pressure is detected among the four directions of front, rear, left and right as in the detection device 330 shown in FIG. A signal indicating the selected direction is transmitted to the HMD 100.
For example, it is assumed that the user inputs an operation for enlarging and displaying the display image displayed at the display position of the pointer 301. In this case, the user puts weight on the inside of the toes of both feet and performs an operation of spreading the toes of both feet, that is, an operation of releasing the toes of both feet. In this case, the left foot detection device 334 determines that the pressure on the inside (right side) of the left foot is high, and transmits a signal indicating that the pressure on the right foot is high to the HMD 100. Also, the right foot detection device 335 determines that the pressure on the inside (left side) of the right foot is high, and transmits a signal indicating that the pressure on the left foot is high to the HMD 100.
When the HMD 100 determines that the pressure on the right side of the left foot is high based on the signal received from the detection device 334 for the left foot and determines that the pressure on the left side of the right foot is high based on the signal received from the detection device 335 on the right foot, the pinch out It is determined that the operation is input. In this case, the control unit 150 of the HMD 100 enlarges and displays the display image displayed at a position overlapping the pointer 301, for example. In this embodiment, an example in which a pinch-out operation is associated with enlargement of a display image has been described. However, a selection operation for selecting a display image displayed at a position overlapping the pointer 301 and a confirmation operation for confirming the operation are supported. May be attached.

For example, it is assumed that the user inputs an operation for reducing the display image displayed at the display position of the pointer 301. In this case, the user puts weight on the inside of the toes of both feet and performs an operation of closing the toes of the spread both feet. In this case, the left foot detection device 334 determines that the pressure on the outside (left side) of the left foot is high and transmits a signal indicating that the left pressure is high to the HMD 100. The right foot detection device 335 determines that the pressure on the outside (right side) of the right foot is high, and transmits a signal indicating that the pressure on the right foot is high to the HMD 100.
The HMD 100 determines that the pressure on the left side of the left foot is high based on the signal received from the detection device 334 on the left foot, and determines that the pressure on the right side of the right foot is high based on the signal received from the detection device 335 on the right foot. It is determined that an operation has been input. In this case, the control unit 150 of the HMD 100 reduces the display image displayed at a position overlapping the pointer 301, for example. In this embodiment, the example in which the pinch-in operation is associated with the enlargement of the display image has been described. However, the operation is associated with the selection operation for selecting the display image displayed at the position overlapping the pointer 301 and the confirmation operation for confirming the operation. May be.

As described above in detail, the HMD 100 according to the present embodiment includes the image display unit 20, the display control unit 153, the track pad 14, the operation control unit 149, the six-axis sensor 235, and the detection control unit 151. Prepare.
The image display unit 20 displays the display image in the display area 310 so that the outside scene is transmitted and can be visually recognized together with the outside scene. The display control unit 153 causes the image display unit 20 to display the pointer 301 indicating the designated position on the display area 310. The track pad 14 and the operation control unit 149 detect an operation on the operation surface of the track pad 14 that is the first operation, and output operation data corresponding to the operation. The 6-axis sensor 235 and the detection control unit 151 detect the tilt of the user's head as the second operation, and output operation data corresponding to the detected tilt of the head. The display control unit 153 moves the position of the pointer 301 according to the first operation detected by the detection unit. In addition, the display control unit 153 changes the position of the pointer 301 according to the second operation detected by the detection control unit 151 after the movement of the position of the pointer 301 according to the first operation to the pointer 301 according to the first operation. It is moved in a manner different from the movement of the position.
Accordingly, since the manner of movement of the pointer 301 is increased, the operability when the pointer 301 is operated can be improved, and the usability of the HMD 100 can be improved.

Moreover, although the 6-axis sensor 235 and the detection control part 151 detect a user's operation | movement as 2nd operation, it is also possible to detect a user's operation | movement as 1st operation.
Therefore, the position of the pointer 301 can be moved by the user's action.

The HMD 100 also includes a camera 61 that captures an outside scene.
The display control unit 163 includes a first operation mode that identifies a target object that is visually recognized by overlapping with the pointer 301 based on a captured image of the camera 61. In addition, the display control unit 163 includes a second operation mode for specifying a display image corresponding to the position of the pointer 301 from a plurality of display images displayed in the display area 310. The display control unit 163 performs switching between the first operation mode and the second operation mode.
Therefore, the target object can be specified by moving the pointer 301, and the display image can be specified.

Further, according to the present invention, the display control unit 153 moves the position of the pointer 301 in a different manner between when the first operation mode is being executed and when the second operation mode is being executed.
Therefore, the manner of movement of the position of the pointer 301 can be changed according to whether the target specified by the pointer 301 is a target object or a display image.

The 6-axis sensor 235 and the detection control unit 151 detect the tilt of the user's head as the second operation. When the detection control unit 151 detects the tilt of the head during execution of the first operation mode, the display control unit 153 moves the pointer 301 with reference to the outside scene visually recognized by the user. In addition, when the detection control unit 151 detects the tilt of the head during the execution of the second operation mode, the display control unit 153 moves the pointer 301 with reference to the display image visually recognized by the user. .
Therefore, during execution of the first operation mode, the position of the pointer 301 can be moved with reference to the outside scene even if the user tilts the head by the second operation. Further, during execution of the second operation mode, the position of the pointer 301 can be moved with reference to the display image.

In addition, the display control unit 153 makes the movement amount when the pointer 301 is moved according to the second operation smaller than the movement amount when the pointer 301 is moved according to the first operation.
Therefore, after the pointer 301 is moved by the first operation having a larger movement amount than the second operation, the pointer 301 can be moved by the second operation having the smaller movement amount than the first operation. Therefore, the pointer 301 can be moved by the first operation, and then the position of the pointer 301 can be adjusted by the second operation.

Moreover, HMD100 is provided with the beacon receiving part 119 which detects the distance to a target object with a beacon signal.
The display control unit 153 moves the pointer 301 in a different manner according to the distance to the target object detected by the beacon receiving unit 119.
Therefore, the operability of the operation of the pointer 301 can be improved and the usability of the HMD 100 can be improved.

Further, when the distance to the target object is greater than the threshold value, the display control unit 153 sets the operation detected by the operation control unit 149 as the first operation, and the distance to the target object is equal to or less than the threshold value. In addition, the operation detected by the detection control unit 151 is defined as a second operation.
The display control unit 153 makes the movement amount for moving the pointer 301 by the first operation smaller than the movement amount for moving the pointer 301 by the second operation.
Therefore, when the distance to the target object is greater than the threshold value, the amount of movement of the pointer 301 can be reduced to prevent the position of the pointer 301 from moving greatly due to the first operation.

In addition, this invention is not restricted to the structure of said each embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.
For example, in the above-described embodiment, the configuration in which the control device 10 is wired to the image display unit 20 is illustrated. However, the present invention is not limited to this, and the image display unit 20 is wirelessly connected to the control device 10. It may be a configuration. In this case, the wireless communication method may be the method exemplified as the communication method supported by the communication unit 117, or may be another communication method.

In addition, when the HMD 100 is in a state of detecting the first operation, the control unit 150 generates an image or a message indicating that the HMD 100 is in a state of detecting the first operation, and the generated image or message is displayed in the image display unit. 20 is displayed. In addition, when the HMD 100 is in a state of detecting the second operation, the control unit 150 generates an image or a message indicating that the HMD 100 is in a state of detecting the second operation, and the generated image or message is displayed in the image display unit. 20 is displayed.
In addition, when the operation mode of the HMD 100 is the first operation mode, the control unit 150 generates an image or message indicating that the operation mode is the first operation mode, and displays the generated image or message as an image. Displayed by the unit 20. In addition, when the operation mode of the HMD 100 is the second operation mode, the control unit 150 generates an image or message indicating that the operation mode is the second operation mode, and displays the generated image or message as an image. Displayed by the unit 20.

  In addition, some functions of the control device 10 may be provided in the image display unit 20, and the control device 10 may be realized by a plurality of devices. That is, the control device 10 is not limited to the configuration including the box-shaped case 10A. For example, instead of the control device 10, a wearable device that can be attached to a user's body, clothes, or an accessory worn by the user may be used. The wearable device in this case may be, for example, a watch-type device, a ring-type device, a laser pointer, a mouse, an air mouse, a game controller, a pen-type device, or the like.

  Further, in each of the above embodiments, the configuration in which the image display unit 20 and the control device 10 are separated and connected via the connection cable 40 has been described as an example. The present invention is not limited to this, and the control device 10 and the image display unit 20 may be configured integrally and mounted on the user's head.

  As the control device 10, a notebook computer, a tablet computer, or a desktop computer may be used. The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like.

  In each of the above embodiments, the configuration in which the user visually recognizes the outside scene through the display unit is not limited to the configuration in which the right light guide plate 26 and the left light guide plate 28 transmit external light. For example, the present invention can be applied to a display device that displays an image in a state where an outside scene cannot be visually recognized. Specifically, the present invention is applied to a display device that displays a captured image of the camera 61, an image or CG generated based on the captured image, video based on video data stored in advance or video data input from the outside, and the like. Can be applied. This type of display device can include a so-called closed display device in which an outside scene cannot be visually recognized. Further, as described in the above embodiment, the present invention is applied to a display device that does not perform processing such as AR display that displays an image superimposed on real space or MR (Mixed Reality) display that combines a captured real space image and a virtual image. The invention can be applied. The present invention can also be applied to a display device that does not perform processing such as VR (Virtual Reality) display for displaying a virtual image. For example, a display device that displays video data or an analog video signal input from the outside is naturally included as an application target of the present invention.

  Further, for example, instead of the image display unit 20, an image display unit of another method such as an image display unit worn like a hat may be adopted, and an image corresponding to the left eye LE of the user is displayed. What is necessary is just to provide the display part which displays, and the display part which displays an image corresponding to a user's right eye RE. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet. In this case, the part for positioning the position with respect to the user's body and the part positioned with respect to the part can be used as the mounting portion.

In addition, as an optical system that guides image light to the user's eyes, a configuration in which virtual images are formed by half mirrors 261 and 281 on part of the right light guide plate 26 and the left light guide plate 28 is illustrated. The present invention is not limited to this, and an image may be displayed on a display region having an area that occupies the entire or most of the right light guide plate 26 and the left light guide plate 28. In this case, a process of reducing the image may be included in the operation of changing the display position of the image.
Furthermore, the optical element of the present invention is not limited to the right light guide plate 26 and the left light guide plate 28 having the half mirrors 261 and 281, and may be any optical component that allows image light to enter the user's eyes. A diffraction grating, a prism, or a holographic display unit may be used.

  In addition, at least a part of the functional blocks shown in FIGS. 4 and 5 may be realized by hardware, or may be realized by cooperation of hardware and software. As described above, the present invention is not limited to a configuration in which independent hardware resources are arranged. Further, the program executed by the control unit 150 may be stored in the nonvolatile storage unit 121 or another storage device (not shown) in the control device 10. A program stored in an external device may be acquired and executed via the communication unit 117 or the external connector 184. Moreover, the operation part 110 may be formed as a user interface (UI) among the structures formed in the control apparatus 10.

  Further, the processing unit of the flowchart shown in FIG. 11 is divided according to main processing contents in order to make the processing of the control unit 150 of the HMD 100 easy to understand. The invention is not limited. Further, the processing of the control unit 150 can be divided into more processing units according to the processing content, or can be divided so that one processing unit includes more processing. Further, the processing order of the above flowchart is not limited to the illustrated example.

  DESCRIPTION OF SYMBOLS 1 ... System, 10 ... Control apparatus, 10A ... Case, 11 ... Button, 12, 67 ... LED indicator, 14 ... Track pad, 15 ... Up / down key, 16 ... Changeover switch, 17 ... LED display part, 18 ... Power switch, DESCRIPTION OF SYMBOLS 19 ... Vibrator, 20 ... Image display part (display part), 21 ... Right holding part, 22 ... Right display unit, 23 ... Left holding part, 24 ... Left display unit, 26 ... Right light guide plate, 27 ... Front frame, 28 ... Left light guide plate, 30 ... Headset, 32 ... Right earphone, 34 ... Left earphone, 40 ... Connection cable, 46 ... Connector, 61 ... Camera (imaging part), 63 ... Microphone, 65 ... Illuminance sensor, 100 ... HMD (Receiving device, display device), 110 ... operation unit, 111 ... 6-axis sensor, 113 ... magnetic sensor, 115 ... GPS receiving unit, 117 ... communication unit, 119 ... bi CON receiver (distance detector), 120 ... controller board, 121 ... nonvolatile storage, 123 ... memory, 125 ... storage, 127 ... setting data, 128 ... content data, 130 ... power supply, 132 ... battery, 134 DESCRIPTION OF SYMBOLS ... Power supply control circuit 140 ... Main processor 143 ... Operating system 145 ... Image processing part 147 ... Imaging control part 149 ... Operation control part (detection part), 150 ... Control part, 151 ... Detection control part (detection part) 153, display control unit, 180, audio codec, 182 ... audio interface, 184 ... external connector, 186 ... external memory interface, 188 ... USB connector, 192 ... sensor hub, 194 ... FPGA, 196 ... interface, 200 ... beacon Device, 211 ... Interface 213 ... receiver, 215 ... EEPROM, 217 ... temperature sensor, 221 ... OLED unit, 223 ... OLED panel, 225 ... OLED drive circuit, 231 ... interface, 233 ... receiver, 235 ... 6-axis sensor (detector) 237 ... magnetic sensor, 239 ... temperature sensor, 241 ... OLED unit, 243 ... OLED panel, 245 ... OLED drive circuit, 251 ... right optical system, 252 ... left optical system, 261,281 ... half mirror, 300 ... field of view, DESCRIPTION OF SYMBOLS 301 ... Pointer, 303 ... Enlarged image, 310 ... Display area, 320 ... Vending machine, 330-335 ... Detection apparatus.

Claims (14)

A head-mounted display device mounted on a user's head,
A display unit that displays a display image in a display area so as to transmit the outside scene and be visible together with the outside scene;
A display control unit that causes the display unit to display a marker indicating the indicated position in the display area;
A detection unit for detecting a first operation and a second operation,
The display control unit moves the position of the marker according to the first operation detected by the detection unit, and the detection unit detects the first position after movement of the marker position according to the first operation. A head-mounted display device that moves the position of the marker according to two operations in a manner different from the movement of the position of the marker according to the first operation.   The head-mounted display device according to claim 1, wherein the detection unit detects an operation of the user as at least one of the first operation and the second operation.   The head-mounted type according to claim 1 or 2, wherein the detection unit detects an operation received by an operation unit included in the head-mounted display device as a first operation and detects a movement of the head as a second operation. Display device. An imaging unit for imaging the outside scene;
The display control unit includes: a first operation mode that identifies a target object that is visually recognized by overlapping the marker based on a captured image of the imaging unit; and the marker from a plurality of the display images displayed in the display area. The head-mounted display device according to any one of claims 1 to 3, wherein the second operation mode for specifying the display image corresponding to the position is switched and executed.   5. The head according to claim 4, wherein the display control unit moves the marker in a different manner between when the first operation mode is being executed and when the second operation mode is being executed. 6. Wearable display device. When the first operation or the second operation is detected by the detection unit during the execution of the first operation mode, the display control unit performs the operation according to the first operation or the second operation. Move the marker with reference to the outside scene visually recognized by the user,
When the first operation or the second operation is detected by the detection unit during the execution of the second operation mode, the user can select the marker according to the first operation or the second operation. The head-mounted display device according to claim 5, wherein the display is moved with reference to the visually recognized display image.   7. The display control unit according to claim 1, wherein the display control unit makes a movement amount when the marker is moved according to the second operation smaller than a movement amount when the marker is moved according to the first operation. The head-mounted display device according to any one of the preceding claims. A distance detection unit that detects a distance to the target object that the user sees through the display unit;
The head-mounted display device according to claim 4, wherein the display control unit moves the marker in a different manner according to a distance to the target object detected by the distance detection unit. The display control unit detects the operation detected by the detection unit as the first operation when the distance to the target object detected by the distance detection unit is greater than a threshold value, and the distance detection unit detects the first operation. When the distance to the target object is less than or equal to a threshold value, the operation detected by the detection unit is the second operation,
The head-mounted display device according to claim 8, wherein a movement amount for moving the marker by the first operation is smaller than a movement amount for moving the marker by the second operation. A head-mounted display device mounted on a user's head,
A display unit that displays a display image in a display area so as to transmit the outside scene and be visible together with the outside scene;
An imaging unit for imaging the outside scene;
A display control unit that causes the display unit to display a marker indicating an indication position on the display area,
The display control unit includes: a first operation mode that identifies a target object that is visually recognized by overlapping the marker based on a captured image of the imaging unit; and the marker from a plurality of the display images displayed in the display area. A head-mounted display device that switches between and executes a second operation mode for specifying the display image corresponding to the position of the head. A detection unit for detecting operation;
The display control unit is configured to reference the outside scene where the user visually recognizes the marker according to the detected operation when an operation is detected by the detection unit during the execution of the first operation mode. Move as
When an operation is detected by the detection unit during execution of the second operation mode, the marker is moved based on the display image visually recognized by the user according to the detected operation. The head-mounted display device according to 10. The detection unit detects an operation received by an operation unit included in the head-mounted display device as a first operation, detects a movement of the user's head as a second operation,
The display control unit moves the position of the marker according to the first operation detected by the detection unit, and the detection unit detects the first position after movement of the marker position according to the first operation. 2 Move the position of the marker according to the operation,
The head-mounted display device according to claim 11, wherein a movement amount when the marker is moved according to the second operation is smaller than a movement amount when the marker is moved according to the first operation. A display unit that transmits an outside scene and displays a display image in a display area so as to be visible together with the outside scene, a display control unit that displays a marker indicating an indication position on the display area by the display unit, a first operation, A detection unit that detects a second operation, and a program that is executed by a computer mounted on a head-mounted display device that is mounted on a user's head,
On the computer,
A procedure for moving the position of the marker according to the first operation detected by the detection unit;
After the movement of the position of the marker according to the first operation, the movement of the marker according to the first operation is changed from the position of the marker according to the second operation detected by the detection unit. Procedures to move in different ways;
A program for running A display unit that transmits an outside scene and displays a display image in a display area so as to be visible together with the outside scene, a display control unit that displays a marker indicating an indication position on the display area by the display unit, a first operation, A detection unit that detects a second operation, and a control method for a head-mounted display device mounted on a user's head,
Moving the position of the marker according to the first operation detected by the detection unit;
After the movement of the position of the marker according to the first operation, the movement of the marker according to the first operation is changed from the position of the marker according to the second operation detected by the detection unit. Moving in different ways;
A control method for a head-mounted display device, comprising:

Images