JP6439448B2 - Display system, program, and control method for display device - Google Patents

Description

  The present invention relates to a display system including a display device, a program, and a control method for the display device.

  2. Description of the Related Art Conventionally, as an input operation method for a display device such as a head mounted display (HMD) used by a user wearing on the head, the device is attached to a finger, and the movement of the finger is detected by this device. A method has been proposed (see, for example, Patent Documents 1 and 2). Patent Document 1 discloses a configuration for inputting text using a device worn on a finger. Patent Document 2 discloses a configuration in which an operation equivalent to that of a mouse is performed using an HMD and a light emitter that is worn on a finger.

JP 2000-284886 A JP 2000-29619 A

When an input operation equivalent to a text input or a mouse is performed in the HMD, if there is an erroneous input, a result contrary to the user's intention is brought about. For this reason, the user pays attention to the input character or the cursor indicating the indicated position, and performs an operation so as to be accurate and error-free. For this reason, the user is required to pay careful attention to the display on the display device and carefully operate it, and a method capable of controlling the HMD with a more intuitive operation is desired.
The present invention has been made in view of the above circumstances, and provides a display system capable of controlling a display device mounted on a user's head by a more intuitive input operation, and a display device control method. With the goal.

In order to achieve the above object, a display system according to the present invention includes a display device that is mounted on a user's head and has a transmissive display unit that allows the user to see through the outside scene, and the display unit. And an operation device configured as a separate body, the operation device includes an operation unit that receives an operation of the user, and the display device transmits the display unit based on an operation of the operation device. A control unit that performs AR display related to an outside scene by the display unit is provided.
According to the present invention, when the display device mounted on the user's head performs AR display related to the outside scene, the operation of the operation device separate from the display device is reflected. For this reason, the AR display can be controlled by an intuitive input operation.

According to the present invention, in the display system, the control unit of the display device displays an AR display related to the outside scene that is transmitted through the display unit in a mode associated with the operation unit based on an operation of the operation device. Is performed by the display unit.
According to the present invention, the operation on the operation device can be reliably detected in the display device. Further, it is possible to reduce the load on the display device related to the operation detection.

According to the present invention, in the display system, the control unit detects an operation of the operation unit based on data transmitted from the operation device, and the display unit displays the operation corresponding to the detected operation. It is characterized by changing the AR display.
According to the present invention, the AR display can be changed by operating the operation device while the AR display is being performed.

In the display system according to the present invention, the control unit may be capable of setting the allocation of the AR display to the operation unit.
In this case, the control unit may be capable of executing a process for updating the allocation of the AR display to the operation unit.

In the display system according to the present invention, the control unit of the display device obtains a relative position between the operation device and the display device, and performs AR display on the display unit based on the obtained relative position. And
According to the present invention, display can be controlled corresponding to the relative position between the operation device and the display device.

In the display system according to the aspect of the invention, the operation unit includes a touch operation unit that detects a contact operation, the touch operation unit includes a plurality of operation areas, and the control unit is operated by the touch operation unit. The AR display assigned to the operation area is displayed by the display unit.
According to the present invention, various operations for AR display can be performed by performing a touch operation with an operation device.

The present invention is characterized in that, in the display system, the operation unit of the operation device detects a movement of the operation device as an operation.
According to the present invention, the display device can be operated by moving the operation device.

The present invention is characterized in that, in the display system, the operation device includes an imaging unit, and data including a captured image of the imaging unit is transmitted to the display device by the communication unit.
According to the present invention, it is possible to capture an image with an operation device separate from the display device and display the captured image on the display device.

In order to achieve the above object, the present invention is a program executable by a computer for controlling a display device having a display unit mounted on a user's head, which is separate from the display unit. When a configured operation device accepts an operation of the user, the display unit performs AR display related to an outside scene that is transmitted through the display unit based on the operation of the operation device.
According to the present invention, when the display device mounted on the user's head performs AR display related to the outside scene, the operation of the operation device separate from the display device is reflected. For this reason, the AR display can be controlled by an intuitive input operation.

In order to achieve the above object, a method for controlling a display device according to the present invention includes a display device having a display unit mounted on a user's head as an operation device configured separately from the display unit. In the control method controlled by using the operation device, the operation unit of the operation device receives an operation of the user, and the display device displays an AR display related to an outside scene that is transmitted through the display unit based on the operation of the operation device. It is performed by the display unit.
According to the present invention, when the display device mounted on the user's head performs AR display related to the outside scene, the operation of the operation device separate from the display device is reflected. For this reason, the AR display can be controlled by an intuitive input operation.

  The present invention can also be realized in various forms other than the head-mounted display device. For example, a head-mounted display device control method, an information system having a head-mounted display device, a head-mounted display device control method, a computer program for realizing the information system, and the computer program recorded The present invention can be realized in the form of a recording medium, a server device for distributing a computer program, a data signal including the computer program and embodied in a carrier wave.

Explanatory drawing which shows schematic structure of the display system which concerns on embodiment. Explanatory drawing which shows the structure of an operation device. The functional block diagram of an operation device. The functional block diagram of a head mounted type display apparatus. The flowchart which shows operation | movement of an operation device. The flowchart which shows operation | movement of a head mounting | wearing type display apparatus. Explanatory drawing which shows the example of AR display for a setting. The figure which shows the example of the process performed according to operation of an operation device. The flowchart which shows operation | movement of a head mounting | wearing type display apparatus. The flowchart which shows another example of operation | movement of a head mounted type display apparatus.

[System configuration]
FIG. 1 is a diagram showing a schematic configuration of a display system 1 according to an embodiment to which the present invention is applied.
The display system 1 includes a head-mounted display device 100 and an operation device 3 that are worn on a user's body.
As shown in FIG. 1, the head-mounted display device 100 includes an image display unit 20 that is mounted on the user's head. The head-mounted display device 100 includes an image display unit 20 (display unit) that allows the user to visually recognize a virtual image while being mounted on the user's head, and a control device 10 that controls the image display unit 20. I have. The image display unit 20 is a mounting body to be mounted on the user's head, and has a glasses-shaped frame 2 (main body) in the present embodiment. The control device 10 also functions as a controller for the user to operate the head-mounted display device 100. The control device 10 is accommodated in the pocket of the user's clothes or attached to the belt on the user's waist.
The operation device 3 is a so-called wearable device that can be worn on the user's body. In the present embodiment, the operation device 3 has a watch-like shape that is worn on the user's arm.

[Operation device configuration]
FIG. 2 is an explanatory diagram showing a configuration of the operation device 3, FIG. 2A is a front view of a main part of the operation device 3, and FIGS. 2B and 2C are diagrams showing display examples on the LCD 303. . FIG. 3 is a functional block diagram of the operation device 3. The configuration of the operation device 3 will be described with reference to FIGS.

  The operation device 3 has a band portion 300 having a shape similar to that of a wristwatch band. The band part 300 has a fixing part such as a buckle (not shown), and can be wound and fixed around, for example, a user's forearm part. In the band portion 300 of the operation device 3, a substantially disc-shaped plane portion 300A is formed at a position corresponding to a dial of a watch. The plane portion 300A is provided with a bezel 301, an LCD 303, a button 305, a crown-type operation element 307, a plurality of buttons 309, and an optical unit 310.

  The bezel 301 is a ring-shaped operation element, and is disposed on the peripheral portion of the flat surface portion 300A. The bezel 301 is provided so as to be rotatable in the circumferential direction with respect to the band unit 300. As will be described later, the operation device 3 includes a mechanism for detecting the rotation direction and the rotation amount of the bezel 301. The mechanism that instructs the band unit 300 to rotate the bezel 301 may include a memory that generates a notch sound when the bezel 301 is rotated.

  The LCD 303 is an LCD (Liquid Crystal Display) that displays characters and images. A touch panel 304 (touch operation unit) shown in FIG.

  The button 305 is a push button type switch disposed outside the bezel 301. For example, as shown in FIGS. 1 and 2A, the button 305 is positioned below the bezel 301 when viewed from the user in the mounted state. The button 305 is larger than the crown-type operation element 307 and the button 309 described later, and can be operated by groping.

  The crown-type operation element 307 is an operation element having a shape imitating a crown of a wristwatch, and can be rotated as indicated by an arrow in the drawing. The operation device 3 includes a mechanism for detecting the rotation direction and the rotation amount of the crown-type operation element 307 when the user rotates the crown-type operation element 307. In addition, the mechanism that instructs the band unit 300 to rotate the crown-type operation element 307 may include a memory that generates a notch sound when the crown unit 307 is rotated.

The button 309 is a push button type switch provided on the outer peripheral portion of the plane portion 300A. The number of buttons 309 is not particularly limited, and in the present embodiment, an example in which four buttons 309 are provided is shown.
Different functions can be assigned to the buttons 309, and the functions assigned to the buttons 309 can be displayed on the LCD 303.

  FIG. 2B shows an example in which the function of the button 309 is displayed on the LCD 303. In this example, function displays 331, 332, 333, and 334 that indicate functions assigned to the four buttons 309 are displayed on the LCD 303. The display positions of the function displays 331, 332, 333, and 334 correspond to the position of the function display target button 309. For example, a function display 331 displayed on the upper right portion of the LCD 303 indicates a function assigned to the button 309 located on the upper right portion of the plane portion 300A.

  Further, the LCD 303 can display selectable functions by rotating the bezel 301. FIG. 2C shows an example in which functions assigned to the operation of the bezel 301 are displayed on the LCD 303. In this example, function displays 341, 342, 343, and 344 indicating four functions that can be selected by rotating the bezel 301 are displayed on the LCD 303. The display positions of the function displays 341, 342, 343, and 344 correspond to the rotational position of the bezel 301. That is, when the bezel 301 is rotated, the function displays 341, 342, 343, and 344 are selected by switching in order. For example, when the bezel 301 is rotated clockwise (CW), the function displays 341, 342, 343, and 344 are sequentially switched and selected.

  A function display 345 indicates a function assigned to the button 305, which is a “decision” function in the example of FIG. When the button 305 is operated in a state where any of the function displays 341, 342, 343, and 344 is selected by the rotation of the bezel 301, the selection of the function displays 341, 342, 343, and 344 is determined. . Then, the function selected by the rotation of the bezel 301 is executed.

  The function assigned to the button 309 and the function selected by the rotation of the bezel 301 are not limited to one, and more functions can be switched and assigned. In this case, the operation device 3 can be configured to be able to switch functions. In this case, the function display 331, 332, 333, 334 or the function display 341, 342, 343, 344 of the LCD 303 may be switched in accordance with the function switching.

The optical unit 310 includes two cameras 311 (imaging unit) and an LED 313. The camera 311 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like. The imaging ranges of the two cameras 311 included in the operation device 3 may be different ranges or the same range. In the present embodiment, the two cameras 311 can capture both eyes of the user wearing the operation device 3, and can detect the user's line of sight based on the captured image. Therefore, one of the two cameras 311 is configured to capture the right eye of the user who views the plane portion 300A from the front, and the other can capture the left eye.
An LED (Light Emitting Diode) 313 is an infrared LED that emits infrared light. The LED 313 is used as illumination when the camera 311 captures an image. The operation device 3 can transmit an optical signal by turning on or blinking the LED 313.

FIG. 3 shows the configuration of the control system of the operation device 3.
The operation device 3 includes a control unit 350 that controls each unit of the operation device 3. The control unit 350 includes a CPU, ROM, RAM, and the like (not shown), and realizes the function of the operation device 3 by executing a program stored in the ROM.
The control unit 350 includes a wireless communication unit 361 (communication unit) and an NFC (Near Field Communication) unit 362 as functional units for communication. The wireless communication unit 361 performs wireless data communication conforming to a standard such as a wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), Bluetooth (registered trademark), or the like. The NFC unit 362 performs short-range wireless communication based on the NFC standard.

The control unit 350 is connected to the LCD 303, the touch panel 304, the two cameras 311, and the LED 313. The control unit 350 controls the LCD 303 to display images and characters. The touch panel 304 is arranged so as to overlap the surface of the LCD 303 and detects a touch operation on the surface of the LCD 303. For the touch panel 304, for example, a capacitance type sensor or a pressure sensitive sensor can be used. The control unit 350 detects a contact operation using the touch panel 304 and specifies an operation position. The touch panel 304 may detect a single contact operation or may correspond to a so-called multi-touch operation that can simultaneously detect operations at a plurality of points.
The control unit 350 controls each of the two cameras 311 to execute imaging and generate captured image data. Further, the control unit 350 turns on / off the current flowing through the LED 313, and turns on or blinks the LED 313 at an arbitrary timing.

  A microphone 365, a button operation unit 366, a rotation detection unit 367, a 9-axis sensor 368, a vibrator 369, and a speaker 370 are connected to the control unit 350. The microphone 365 may be, for example, a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone. When the microphone 365 has directivity, the microphone 365 collects sound in the front direction of the plane portion 300A, for example, at or near the plane portion 300A (FIG. 2). The microphone 365 may be a bone conduction microphone that contacts the user's body. The control unit 350 acquires an analog audio signal of the audio collected by the microphone 365 and generates digital audio data.

  The button operation unit 366 detects the operation of the buttons 305 and 309 and outputs an operation signal corresponding to the operated button to the control unit 350. The rotation detection unit 367 detects the rotation of the bezel 301 and the rotation of the crown type manipulator 307. The rotation detection unit 367 outputs a signal indicating the rotation direction and the rotation amount of the bezel 301 to the control unit 350, and similarly outputs a signal indicating the rotation direction and the rotation amount to the control unit 350 for the crown-shaped operation element 307. Here, the rotation detection unit 367 may output a signal indicating the rotation direction every time the bezel 301 rotates by a predetermined amount. In this case, since the control unit 350 counts the output signal of the rotation detection unit 367, the control unit 350 can acquire the rotation amount, and the rotation detection unit 367 does not need to integrate the rotation amount. Similarly, for the rotation operation of the crown-type operation element 307, the rotation detection unit 367 may output a signal indicating the rotation direction every time the crown-type operation element 307 rotates by a predetermined amount.

The 9-axis sensor 368 is a unit including an inertial sensor, and in this embodiment, includes a 3-axis acceleration sensor, a 3-axis angular velocity sensor, and a 3-axis magnetic sensor. The axis on which the nine-axis sensor 368 detects can be set, for example, in the vertical direction, the horizontal direction, and the direction orthogonal to the plane including the plane portion 300A in the plane including the plane portion 300A.
Vibrator 369 includes a motor (not shown) that rotates under the control of control unit 350. A weight is eccentrically attached to the rotation shaft of the motor, and the operation device 3 is vibrated as the motor rotates. The speaker 370 outputs audio based on the audio signal output from the control unit 350.

  Power is supplied to each unit of the operation device 3 including the control unit 350 from a power supply unit 360 having a primary battery or a secondary battery. The power supply unit 360 may include a circuit that controls charging of the secondary battery together with the secondary battery.

The control unit 350 realizes the functions of the operation detection unit 351, the output control unit 352, the imaging control unit 353, the communication control unit 354, and the sound processing unit 355 by executing a program.
The operation detection unit 351 detects the operation of the operator included in the operation device 3 based on the signals input from the button operation unit 366 and the rotation detection unit 367, and specifies the operation content. Specifically, the operation detection unit 351 detects the operation of the button 305, the crown type operation element 307, and the button 309. Further, the operation detection unit 351 detects an operation such as shaking or moving the operation device 3 based on a signal input from the 9-axis sensor 368. Further, the operation detection unit 351 detects a touch operation on the LCD 303 based on a signal input from the touch panel 304 and specifies an operation position. Here, the operation detection unit 351 can specify the function assigned to the operation position by specifying the image displayed on the LCD 303 at the operation position.

The output control unit 352 executes output in response to data received by the communication control unit 354 by the wireless communication unit 361 or the NFC unit 362 or an operation detected by the operation detection unit 351. The output control unit 352 displays characters and images on the LCD 303. The output control unit 352 operates the motor of the vibrator 369 to vibrate the operation device 3.
The imaging control unit 353 controls the camera 311 to execute imaging, and acquires captured image data.
The communication control unit 354 transmits / receives data to / from an external device through the wireless communication unit 361 or the NFC unit 362. The communication control unit 354 can execute data communication with the head-mounted display device 100 through the wireless communication unit 361. Further, when the NFC unit 362 detects a device capable of near field communication, the communication control unit 354 transmits and receives data to and from this device. Further, the communication control unit 354 can transmit data using an optical signal by blinking the LED 313. In this case, the communication control unit 354 may be configured to detect and receive an optical signal with the camera 311.

  The sound processing unit 355 generates digital sound data of the sound collected by the microphone 365 and performs analysis as necessary. For example, the voice processing unit 355 may analyze the user's voice collected by the microphone 365 and output an operation instruction to the operation detection unit 351 when detecting a voice instructing the operation. In this case, the user can operate the operation device 3 by voice. In addition, the audio processing unit 355 outputs an audio signal to the speaker 370 to output audio.

  The control unit 350 can execute a plurality of operation modes including a clock mode and a UI mode as operation modes of the operation device 3. The clock mode is an operation mode in which the control unit 350 measures the current time and displays the current time on the LCD 303. The clock mode has an advantage that the power consumption is small because the operation device 3 is operated as a clock. The UI mode is an operation mode in which the operation device 3 functions as a user interface (UI) for operating the head-mounted display device 100. In the UI mode, the operation device 3 transmits data to the head-mounted display device 100 based on an operation on the operation device 3, and the head-mounted display device 100 operates according to this data. Switching from the clock mode to the UI mode and switching from the UI mode to the clock mode are performed under the control of the control unit 350, as will be described later.

[Configuration of head-mounted display device]
In the head-mounted display device 100, the frame 2 of the image display unit 20 includes a right holding unit 21 and a left holding unit 23. The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 contacts the right ear or the vicinity thereof on the user's head, and the left holding unit 23 contacts the user's left ear or the vicinity thereof and holds the image display unit 20 on the user's head. To do.

The frame 2 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61 ( An imaging unit) and a microphone 63 are provided.
The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 and the left display drive unit 24 are disposed on the side facing the user's head when the user wears the image display unit 20. The right display drive unit 22 and the left display drive unit 24 are collectively referred to simply as “display drive unit”, and the right optical image display unit 26 and the left optical image display unit 28 are simply referred to as “optical image display unit”. Also called.

The display driving units 22 and 24 include liquid crystal displays 241 and 242 (Liquid Crystal Display, hereinafter referred to as “LCDs 241 and 242”), projection optical systems 251 and 252, and the like.
The right optical image display unit 26 and the left optical image display unit 28 include light guide plates 261 and 262 (FIG. 2) and a light control plate 20A. The light guide plates 261 and 262 are formed of a light transmissive resin or the like, and guide the image light output from the display driving units 22 and 24 to the user's eyes. The light control plate 20A is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 which is the side opposite to the user's eye side. As the light control plate 20A, various materials such as a material that has almost no light transmission, a material that is nearly transparent, a material that attenuates the amount of light and transmits light, and a material that attenuates or reflects light of a specific wavelength may be used. it can. By appropriately selecting the optical characteristics (light transmittance, etc.) of the light control plate 20A, the external light amount incident on the right optical image display unit 26 and the left optical image display unit 28 from the outside is adjusted to visually recognize the virtual image. You can adjust the ease. In the present embodiment, a case will be described in which at least a light control plate 20A having such a light transmittance that a user wearing the head-mounted display device 100 can visually recognize an outside scene is used. The light control plate 20A protects the right light guide plate 261 and the left light guide plate 262, and suppresses damage to the right light guide plate 261 and the left light guide plate 262, adhesion of dirt, and the like.
The dimming plate 20A may be detachable from the right optical image display unit 26 and the left optical image display unit 28, or may be mounted by replacing a plurality of types of dimming plates 20A, or may be omitted. .
Then, the head-mounted display device 100 causes the image light of the image to be processed inside and the external light to overlap and enter the user's eyes, and the user can see the outside scene through the dimming plate 20A, An image of image light is visually recognized over the outside scene. Thus, the head-mounted display device 100 functions as a see-through display device.

The camera 61 is disposed at the boundary between the right optical image display unit 26 and the left optical image display unit 28. In a state where the user wears the image display unit 20, the position of the camera 61 is substantially in the middle of both eyes of the user in the horizontal direction and above the eyes of the user in the vertical direction. The camera 61 is a digital camera that includes an imaging element such as a CCD or CMOS, an imaging lens, and the like, and may be a monocular camera or a stereo camera.
The camera 61 images at least a part of the outside view in the front side direction of the head-mounted display device 100, in other words, the user's view direction when the head-mounted display device 100 is mounted. The width of the angle of view of the camera 61 can be set as appropriate, but the imaging range of the camera 61 is a range that includes the outside world visually recognized by the user through the right optical image display unit 26 and the left optical image display unit 28. preferable. Furthermore, it is more preferable that the imaging range of the camera 61 is set so that the entire field of view of the user through the light control plate 20A can be imaged.
The camera 61 executes imaging under the control of the control unit 140 (FIG. 4), and outputs captured image data to the control unit 140.

  The image display unit 20 is connected to the control device 10 via the connection unit 40. The connection unit 40 includes a main body cord 48, a right cord 42, a left cord 44, and a connecting member 46 that are connected to the control device 10. The right cord 42 and the left cord 44 are codes in which the main body cord 48 is branched into two. The right cord 42 is inserted into the housing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and is connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The connecting member 46 is provided at a branch point between the main body cord 48, the right cord 42 and the left cord 44, and has a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A microphone 63 is provided in the vicinity of the earphone plug 30. The earphone plug 30 to the microphone 63 are combined into one cord, and the cord branches from the microphone 63 and is connected to each of the right earphone 32 and the left earphone 34.

  For example, as shown in FIG. 1, the microphone 63 is arranged so that the sound collection unit of the microphone 63 faces the user's line of sight, collects sound, and sends the sound signal to the sound processing unit 187 (FIG. 3). Output. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.

  The right cord 42, the left cord 44, and the main body cord 48 only need to be capable of transmitting digital data, and can be constituted by, for example, a metal cable or an optical fiber. Further, the right cord 42 and the left cord 44 may be combined into a single cord.

  The image display unit 20 and the control device 10 transmit various signals via the connection unit 40. The end of the main body cord 48 opposite to the connecting member 46 and the control device 10 are provided with connectors (not shown) that fit together. By fitting the connector of the main body cord 48 and the connector of the control device 10 or removing this fitting, the control device 10 and the image display unit 20 can be contacted and separated.

  The control device 10 controls the head-mounted display device 100. The control device 10 includes switches including an enter key 11, a lighting unit 12, a display switching key 13, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. In addition, the control device 10 includes a track pad 14 that is operated by a user with fingers.

  The determination key 11 detects a pressing operation and outputs a signal for determining the content operated by the control device 10. The lighting unit 12 includes a light source such as an LED, and notifies the operating state (for example, power ON / OFF) of the head-mounted display device 100 according to the lighting state of the light source. The display switching key 13 outputs, for example, a signal instructing switching of the image display mode in response to the pressing operation.

  The track pad 14 has an operation surface for detecting a contact operation, and outputs an operation signal according to an operation on the operation surface. The detection method on the operation surface is not limited, and an electrostatic method, a pressure detection method, an optical method, or the like can be adopted. The luminance switching key 15 outputs a signal instructing increase / decrease of the luminance of the image display unit 20 in response to the pressing operation. The direction key 16 outputs an operation signal in response to a pressing operation on a key corresponding to the up / down / left / right direction. The power switch 18 is a switch for switching on / off the power of the head-mounted display device 100.

FIG. 4 is a functional block diagram of each part constituting the head-mounted display device 100.
The head-mounted display device 100 includes an interface 125 for connecting various external devices OA that are content supply sources. As the interface 125, for example, an interface corresponding to wired connection such as a USB interface, a micro USB interface, a memory card interface, or the like may be used, and the interface 125 may be configured by a wireless communication interface. The external device OA is an image supply device that supplies an image to the head-mounted display device 100, and a personal computer (PC), a mobile phone terminal, a portable game machine, or the like is used.

The control device 10 includes a control unit 140, an input information acquisition unit 110, a storage unit 120, a transmission unit (Tx) 51, and a transmission unit (Tx) 52.
The input information acquisition unit 110 is connected to the operation unit 111. The operation unit 111 is connected to each operation element of the control device 10, detects an operation of each operation element, and outputs an operation signal corresponding to the detected operation element to the input information acquisition unit 110. The input information acquisition unit 110 acquires the input content input by the operation of the control device 10 based on the signal input from the operation unit 111.

The storage unit 120 is a nonvolatile storage device, and stores various computer programs and data related to these programs. The storage unit 120 may store still image data or moving image data to be displayed on the image display unit 20.
The control device 10 includes a power supply unit 130 including a primary battery and a secondary battery, and supplies power from the power supply unit 130 to each unit of the control device 10 and the image display unit 20.

  A three-axis sensor 113, a voice recognition unit 114, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 acquires the detection value of the triaxial sensor 113. The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and calculates the current position of the control device 10. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. The GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140.

The communication unit 117 executes wireless data communication conforming to a standard such as a wireless LAN (WiFi (registered trademark)), Miracast (registered trademark), or Bluetooth (registered trademark).
When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and causes the image display unit 20 to display an image. On the other hand, when the external device OA is wired to the interface 125, the control unit 140 acquires content data from the interface 125 and causes the image display unit 20 to display an image. The communication unit 117 and the interface 125 function as a data acquisition unit DA that acquires content data from the external device OA.

  The control unit 140 stores a CPU (not shown) for executing a program, a RAM (not shown) for temporarily storing a program and data executed by the CPU, and a basic control program and data executed by the CPU in a nonvolatile manner. ROM (not shown). The control unit 140 controls each unit of the head-mounted display device 100 by executing a control program by the CPU. In addition, the control unit 140 reads out and executes the computer program stored in the storage unit 120 and realizes various functions of the control unit 140. That is, they function as an operating system (OS) 150, an image processing unit 160, a display control unit 170, an imaging processing unit 181, a device control unit 183, an AR display control unit 185, and an audio processing unit 187.

The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit or the like (not shown) according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). The image processing unit 160 stores the converted digital image signal in the RAM of the control unit 140 for each frame as image data of the target image (Data in the figure). This image data is, for example, RGB data.
Note that the image processing unit 160 may perform a resolution conversion process for converting the resolution of the image data into a resolution suitable for the right display driving unit 22 and the left display driving unit 24 as necessary. The image processing unit 160 also performs image adjustment processing for adjusting the brightness and saturation of image data, 2D image data is created from 3D image data, or 2D / 3D conversion processing is performed to generate 3D image data from 2D image data. May be executed.

  The image processing unit 160 transmits the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data stored in the RAM via the transmission units 51 and 52, respectively. The transmission units 51 and 52 function as a transceiver and execute serial transmission between the control device 10 and the image display unit 20. The image data Data transmitted via the transmission unit 51 is referred to as “right eye image data”, and the image data Data transmitted via the transmission unit 52 is referred to as “left eye image data”.

  The display control unit 170 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24, and generates and generates image light by each of the right display drive unit 22 and the left display drive unit 24 based on the control signal. Control injection. Specifically, the right LCD 241 is controlled to be turned on / off by the right LCD control unit 211, and the right backlight 221 is turned on / off by the right backlight control unit 201. In addition, the display control unit 170 controls the driving of the left LCD 242 by the left LCD control unit 212 and the driving of the left backlight 222 by the left backlight control unit 202.

  The audio processing unit 187 acquires an audio signal included in the content, amplifies the acquired audio signal, and outputs the amplified audio signal to the right earphone 32 and the left earphone 34. In addition, the sound processing unit 187 acquires sound collected by the microphone 63 and converts it into digital sound data. The voice processing unit 187 may perform a preset process on the digital voice data.

  The image display unit 20 includes an interface 25, a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, and a left light guide plate 262 as a left optical image display unit 28. A camera 61, a vibration sensor 65, and a nine-axis sensor 66.

  The vibration sensor 65 is configured using an acceleration sensor, and is built in the vicinity of the end ER of the right optical image display unit 26 in the right holding unit 21 as shown in FIG. When the user performs an operation of hitting the end ER (knock operation), the vibration sensor 65 detects vibration caused by this operation and outputs the detection result to the control unit 140. Based on the detection result of the vibration sensor 65, the control unit 140 detects a knocking operation by the user.

The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). When the image display unit 20 is mounted on the user's head, the control unit 140 can detect the movement of the user's head based on the detection value of the 9-axis sensor 66. For example, the control unit 140 can estimate the magnitude of the tilt and the tilt direction of the image display unit 20 based on the detection value of the 9-axis sensor 66.
The detection axes of the acceleration sensor and the angular velocity sensor built in the nine-axis sensor 66 can be, for example, three axes of X, Y, and Z shown in FIG. In this example, the left-right direction is the X axis, the front-rear direction is the Y-axis, and the vertical direction is the Z-axis with respect to the head of the user wearing the image display unit 20. More specifically, when the head-mounted display device 100 is worn, the image display unit 20 is in a horizontal position that is perceived by the user with respect to the left and right eyes. In a standard mounting position of the head-mounted display device 100, the detection axis (X axis, Y axis, Z axis) of the 9-axis sensor 66 is aligned with the left, right, front, back, and top perceived by the user. is there. The acceleration sensor of the 9-axis sensor 66 detects acceleration in the X-axis direction, the Y-axis direction, and the Z-axis direction. The angular velocity sensor included in the nine-axis sensor 66 detects rotation (pitch) around the X axis, rotation (yaw) around the Y axis, and rotation (roll) around the Z axis.

The interface 25 includes a connector to which the right cord 42 and the left cord 44 are connected. The interface 25 outputs the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data transmitted from the transmission unit 51 to the corresponding reception units (Rx) 53 and 54. Further, the interface 25 outputs a control signal transmitted from the display control unit 170 to the corresponding reception units 53 and 54, the right backlight control unit 201, or the left backlight control unit 202.
The interface 25 is an interface for connecting the camera 61, the vibration sensor 65, and the 9-axis sensor 66. The detection result of vibration by the vibration sensor 65 and the detection result of acceleration (three axes), angular velocity (three axes), and geomagnetism (three axes) by the nine-axis sensor 66 are sent to the control unit 140 via the interface 25.

  The right display drive unit 22 includes the right backlight 221, the right LCD 241, and the right projection optical system 251 described above. The right display driving unit 22 includes a receiving unit 53, a right backlight (BL) control unit 201 that controls the right backlight (BL) 221, and a right LCD control unit 211 that drives the right LCD 241.

  The reception unit 53 operates as a receiver corresponding to the transmission unit 51 and executes serial transmission between the control device 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data input via the reception unit 53. .

  The left display drive unit 24 has the same configuration as the right display drive unit 22. The left display driving unit 24 includes the left backlight 222, the left LCD 242 and the left projection optical system 252 described above. The left display driving unit 24 includes a receiving unit 54, a left backlight control unit 202 that drives the left backlight 222, and a left LCD control unit 212 that drives the left LCD 242.

  The reception unit 54 operates as a receiver corresponding to the transmission unit 52 and executes serial transmission between the control device 10 and the image display unit 20. The left backlight control unit 202 drives the left backlight 222 based on the input control signal. The left LCD control unit 212 drives the left LCD 242 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right-eye image data Data input via the reception unit 54. . The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are collectively referred to as a right “image light generation unit”. Similarly, the left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are collectively referred to as a left “image light generation unit”.

The imaging processing unit 181 controls the camera 61 to execute imaging, and acquires captured image data.
The device control unit 183 controls the communication unit 117 to perform data communication with the operation device 3 and analyzes data received from the operation device 3. Further, when the blinking pattern of the LED 313 provided in the operation device 3 corresponds to a specific pattern based on the captured image data of the camera 61, the device control unit 183 decodes this pattern and receives data.

  The AR display control unit 185 controls the image processing unit 160 and the display control unit 170 based on the AR content data, and causes the image display unit 20 to display an image for AR display. For example, the storage unit 120 stores the AR content data. In addition, when the content data 123 includes audio data, the AR display control unit 185 controls the audio processing unit 187 to output the audio of the content from the right earphone 32 and the left earphone 34.

  The AR display control unit 185 displays the AR content while the user is looking at the object through the image display unit 20. The AR display control unit 185 provides information about the object by performing AR display that displays an image, characters, or the like at a position corresponding to the object, or the shape of the object seen through the image display unit 20 Change how it looks. The AR content includes image and character data displayed at a position corresponding to the object. In addition, the AR content may include data for specifying an object and data regarding the display position of an image or a character. The display position of the AR content may be a position overlapping the target object or may be around the target object. The object may be an object, may be a real estate such as a building, may be a moving body such as an automobile or a train, or may be a living organism such as a human being or an animal. The AR display control unit 185 detects an object located in the user's field of view from the captured image data acquired by the imaging processing unit 181. Then, the AR display control unit 185 determines the display position of the AR content corresponding to the detected object, and displays the AR content.

  It is desirable to display the AR content so as to overlap the position where the user visually recognizes the target object or according to the position where the user visually recognizes the target object. For this reason, the AR display control unit 185 detects the image of the target object from the captured image data of the imaging processing unit 181, and in the imaging range of the camera 61 based on the positional relationship between the detected target image and the entire captured image. Identify the position of the object. The AR display control unit 185 determines the display position of the AR content corresponding to the position of the object based on the positional relationship between the imaging range of the camera 61 and the display area of the image display unit 20.

  The audio processing unit 187 acquires an audio signal included in the content, amplifies the acquired audio signal, and outputs the amplified audio signal to the right earphone 32 and the left earphone 34. In addition, the sound processing unit 187 acquires sound collected by the microphone 63 and converts it into digital sound data. The voice processing unit 187 may perform a preset process on the digital voice data.

[Display system operation]
FIG. 5 is a flowchart showing the operation of the display system 1, and particularly shows the operation of the operation device 3.
The operation device 3 operates in the clock mode while the display system 1 is not operated (step S11). When an operation for instructing the transition to the UI mode is performed in the timepiece mode (step S12), the control unit 350 of the operation device 3 performs imaging with the cameras 311 and 311 to detect the user's line of sight (step S12). S13). The control unit 350 performs imaging and analysis of the captured image until a line of sight is detected (step S13; NO).

  When detecting the line of sight (step S13; YES), the controller 350 determines whether or not the head-mounted display device 100 is ON (step S14). In step S <b> 14, for example, the wireless communication unit 361 communicates with the communication unit 117 of the head-mounted display device 100 to detect the operating state of the head-mounted display device 100, thereby performing the determination. Further, for example, the camera 311 may perform imaging under the control of the imaging control unit 353, and the operation state of the head-mounted display device 100 may be determined based on the captured image. For example, when the image displayed by the image display unit 20 passes through the light control plate 20A and appears in the captured image, it may be determined that the head-mounted display device 100 is ON. In addition, the image display unit 20 may be provided with a light emitting unit that emits light according to the operating state, and the operating state may be determined by specifying the light emitting state of the light emitting unit from the captured image.

  When it is determined that the head-mounted display device 100 is not ON (step S14; NO), the control unit 350 performs communication by the wireless communication unit 361 and instructs the head-mounted display device 100 to start. Data is transmitted (step S15). The control unit 350 receives control data indicating activation from the head-mounted display device 100, and proceeds to step S17. When it is determined that the head-mounted display device 100 is ON (step S14; YES), the control unit 350 proceeds to step S17.

  In step S17, control unit 350 determines whether or not authentication of head-mounted display device 100 has been completed. When used in combination with the head-mounted display device 100, the control unit 350 authenticates the head-mounted display device 100 and operates as a user interface of the head-mounted display device 100 that has been successfully authenticated. . If it is determined that the authentication has not been completed (step S17; NO), the control unit 350 transmits / receives authentication data to / from the head-mounted display device 100 that has determined whether or not the authentication is ON in step S14. To authenticate (step S18). When data necessary for authentication is transmitted / received, the earphone plug 30 performs pairing with the head-mounted display device 100, that is, one-to-one association that operates as a user interface (step S19), and proceeds to step S20. . This association may be one-to-many. When the head-mounted display device 100 has been authenticated (step S17; YES), the control unit 350 proceeds to step S20.

In step S20, control unit 350 displays an operation screen on LCD 303 and stands by for the operation. The operation screen is, for example, the screen illustrated in FIG. 2B or 2C.
Control unit 350 waits for an operation, and when operation is detected by operation detection unit 351 (step S21), operation data indicating the detected operation is transmitted to head-mounted display device 100 (step S22).

Thereafter, the control unit 350 determines whether or not an end condition is satisfied (step S23). When the end condition is satisfied (step S23; YES), the control unit 350 shifts to the clock mode (step S24) and ends the process. If the end condition is not satisfied (step S23; NO), the process returns to step S21.
The end condition is, for example, that the operation for instructing the end of the UI mode is performed by the touch panel 304, the button 305, the crown-type operation element 307, or the button 309, and the control data for instructing the end from the head-mounted display device 100. , Etc.

In the display system 1, the head-mounted display device 100 can display an AR image corresponding to the operation by the operation device 3.
FIG. 6 is a flowchart showing the operation of the head-mounted display device 100, and particularly shows the operation of performing AR display corresponding to the operation device 3.
The control unit 140 causes the device control unit 183 to receive and analyze the operation data transmitted from the operation device 3 by the communication unit 117 (step S31), and specify the type of operation (step S32). The data received in step S31 is operation data (step S22 in FIG. 5) transmitted by the operation device 3 in response to the operation. The device control unit 183 determines whether or not the instruction corresponding to the operation indicated by the operation data is an instruction related to the operation setting of the operation device 3 (step S33). When the instruction is other than the operation setting (step S33; NO), the device control unit 183 executes a process corresponding to the operation specified in step S32 (step S34), and ends this process. In step S34, for example, the AR display control unit 185 starts / ends the AR display according to the process of starting the imaging of the camera 61 according to the operation data instructing the start of imaging or the operation data instructing the start or end of the AR display. .

  On the other hand, when the instruction corresponding to the operation indicated by the operation data is an instruction related to operation setting (step S33; YES), the AR display control unit 185 performs AR display for operation setting (step S35). Here, when the operation device 3 transmits operation data in response to an operation on the operation device 3 during AR display, the device control unit 183 receives the operation data, and sets and sets the operation data based on the received operation data. Update the settings.

7A and 7B show an example of AR display for operation setting, FIG. 7A shows an example of AR display for setting the operation of the bezel 301, and FIG. 7B shows the setting for operation of the button 309. An example of AR display is shown.
In the example of FIG. 7A, the actual operation device 3 is visually recognized in the user's visual field VR, and the AR content is displayed by the image display unit 20 at a position corresponding to the operation device 3. The AR contents are assignment candidate displays A1, A2, A3, A4 and setting candidates A11, A12, A13, A14. The assignment candidate displays A1, A2, A3, and A4 are images showing candidate functions to be assigned to the rotation of the bezel 301. In this example, the assignment candidate displays A1, A2, A3, and A4 include text indicating combinations of functions that can be selected by rotating the bezel 301. The assignment candidate display A1 includes four character strings (text) of “Home”, “Return”, “Menu”, and “App end”. The allocation candidate display A2 includes four character strings of “forward”, “return”, “imaging”, and “imaging end”. When the operation of rotating the bezel 301 is performed on the operation device 3, any of the allocation candidates A1 to A4 can be selected in order.

  In the example of FIG. 7A, the allocation candidate A2 is selected. For this reason, four character strings of “forward”, “return”, “imaging”, and “imaging end” are arranged so as to overlap the operation device 3 visible in the center of the visual field VR. These character strings are candidate functions to be assigned, that is, setting candidates A11, A12, A13, and A14. When the assignment candidate display A2 is switched to another assignment candidate display by operating the bezel 301, the character strings of the setting candidates A11, A12, A13, and A14 are changed. Therefore, the user can select and set the function assigned to the rotation operation of the bezel 301 from the assignment candidate displays A1, A2, A3, and A4. When the button 305 is operated on the operation device 3, the assignment candidate display selected by the operation of the bezel 301 is confirmed and the setting is updated.

  In the example of FIG. 7B, assignment candidate displays A21, A22, A23, and A24 and setting candidates A31, A32, A33, and A34 are displayed as AR contents at positions corresponding to the actual operation device 3. The assignment candidate displays A21, A22, A23, and A24 are images showing function candidates to be assigned to the four buttons 309. In this example, the functions executed when each of the four buttons 309 are operated are shown. Contains text. The allocation candidate display A21 includes four character strings (text) of “Home”, “Return”, “Menu”, and “App end”. The allocation candidate display A22 includes four character strings of “forward”, “return”, “imaging”, and “imaging end”. When the operation of rotating the bezel 301 is performed on the operation device 3, any of the allocation candidates A21 to A24 can be selected in order.

  In the example of FIG. 7B, allocation candidate A22 is selected. For this reason, four character strings of “forward”, “return”, “imaging”, and “imaging end” are arranged so as to overlap the operation device 3 visible in the center of the visual field VR. These character strings are candidates for functions to be assigned, that is, setting candidates A31, A32, A33, and A34, and are displayed at positions corresponding to buttons 309 to which functions are assigned. When the assignment candidate display A22 is switched to another assignment candidate display by the operation of the bezel 301, the character strings of the setting candidates A31, A32, A33, and A34 are changed. Therefore, the user can select and set the function assigned to the rotation operation of the bezel 301 from the assignment candidate displays A21, A22, A23, and A24. When the button 305 is operated on the operation device 3, the assignment candidate display selected by the operation of the bezel 301 is confirmed and the setting is updated.

  The examples of FIGS. 7A and 7B show an operation of setting a function to be executed by the head-mounted display device 100 corresponding to the operation data transmitted by the operation device 3. Since the assignment of functions can be set and updated, various functions of the head-mounted display device 100 can be operated by the operation device 3. Moreover, the setting can be easily performed by using the AR display.

  FIG. 8 is an explanatory diagram illustrating an example of an operation performed by the head-mounted display device 100 in response to an operation of the operation device 3. FIG. 8 shows an example in which the display mode of the AR display displayed on the image display unit 20 is changed according to the operation of the operation device 3 under the control of the AR display control unit 185. FIG. 8A shows an example of AR display by the image display unit 20, and FIG. 8B shows an example in which the display mode of the AR display is changed.

In the example of FIG. 8A, a stage, performers, and audience seats are visible as an actual outside scene in the visual field VR. In this example, the character string A51 related to the performance being performed on the stage is displayed so as to overlap the actual outside scene. The character string A51 is, for example, text indicating a dialogue of the stage, and AR is displayed at a position overlapping the stage.
The head-mounted display device 100 changes the display size of the character string A51 according to the operation of the operation device 3, and displays the reduced size as shown in FIG. 8B, for example. In this operation, the device control unit 183 analyzes the operation data transmitted by the operation device 3, and when the device control unit 183 detects that the operation corresponding to the reduction of the AR display has been performed, the AR display control unit 185 Change the display size.

  In the head-mounted display device 100, as described with reference to FIGS. 6 and 7, the operation performed by the head-mounted display device 100 can be associated with the operation on the operation device 3. This association is not limited to the operation of the button 309 and the rotation operation of the bezel 301. For example, an operation of pressing the button 305 and an operation of rotating the crown-type operation element 307 may be used. Further, for example, an operation of touching the touch panel 304 with a finger may be included.

Specifically, the display image may be enlarged in response to an operation of turning the bezel 301 to the right, and the display image may be reduced in response to an operation of turning the bezel 301 to the left. Further, the display image is enlarged when a finger contact with the center of the touch panel 304 is detected, and the display image is reduced when a finger contact with the peripheral edge of the touch panel 304 (for example, near the lower left frame) is detected. Also good. Further, when the finger touching the touch panel 304 rotates in a clockwise direction or in a counterclockwise direction while being in contact, the display image may be rotated in accordance with the rotation.
In response to these operations, the head-mounted display device 100 performs not only enlargement / reduction of an image (including characters) displayed on the image display unit 20, but also rotation, copying, page turning, and the like. Also good.

  Furthermore, the operation device 3 can transmit the detection value of the 9-axis sensor 368 to the head-mounted display device 100. Further, the operation device 3 may include a GPS detection unit, and the current position detected by the GPS detection unit may be transmitted to the head-mounted display device 100. In this case, the control unit 140 can detect the relative position between the operation device 3 and the head-mounted display device 100 based on the detection value received from the operation device 3 and the detection value of the three-axis sensor 113.

  Then, the control unit 140 changes display such as enlargement, reduction, and rotation of the display image in response to the movement of the operation device 3 or the change in the relative position between the operation device 3 and the head-mounted display device 100. You may perform, and you may perform fade-in display and fade-out display of an image. In this case, the display can be changed by an operation of bringing the operation device 3 close to or away from the head-mounted display device 100.

Operation | movement of the display system 1 is not limited to said example, For example, the following operations can be performed.
By the operation of rotating the bezel 301 in the operation device 3, it is possible to realize an operation of sequentially switching and selecting a plurality of candidates. According to this operation, for example, when selecting a file or folder, or when selecting a virtual key on the virtual keyboard, the bezel 301 may be rotated. The bezel 301 can be operated intuitively and is located on the outer peripheral portion of the flat surface portion 300A of the operation device 3, so that it can be operated by groping. Therefore, the user can operate the bezel 301 while paying attention to the AR display displayed by the image display unit 20. Furthermore, since the selection can be confirmed by a button 305 which is provided at a position apart from the plane portion 300A and is larger than other switch type operation elements, the operation can be performed more easily. Since the position and size of the button 305 are different from those of the button 309 and the crown-type operation element 307, the operation by groping is easy like the bezel 301. Therefore, an intuitive and reliable operation can be performed while viewing the image displayed by the image display unit 20.

  Further, the control unit 140 may perform an operation of changing a screen displayed by the image display unit 20 in accordance with an operation of rotating the bezel 301. It is conceivable that the operating system 150 of the head-mounted display device 100 has a configuration in which a plurality of screens are switched and changed like iOS (registered trademark) or Android (registered trademark). In this case, the operation of the bezel 301 can be used more effectively by changing the screen according to the operation of rotating the bezel 301.

  In the operation of selecting one candidate from a plurality of candidates, instead of the operation of the bezel 301, it may be possible to select by voice. In this case, when a unique name, number, code, or the like assigned to the candidate is designated by voice, the operation device 3 detects this voice with the microphone 365 and transmits the detected voice data to the head-mounted display device 100. Then, the control unit 140 may perform the setting.

  It is also possible for the head-mounted display device 100 to operate the operation device 3 by transmitting control data to the operation device 3. For example, the device control unit 183 of the control unit 140 may transmit control data instructing to start detection of the touch panel 304 and the 9-axis sensor 368 or to shift to a non-detection state. In this case, execution / non-execution of sensing in the operation device 3 can be controlled by the head-mounted display device 100, so that an erroneous operation can be prevented and consumption of the power supply unit 360 can be prevented. In addition, the device control unit 183 may transmit control data instructing to start and stop imaging of the camera 311. In addition, the device control unit 183 may transmit control data for controlling lighting, extinguishing, and blinking of the LCD 303.

  The operation device 3 can also perform output based on data transmitted from the head-mounted display device 100. For example, the device control unit 183 may transmit text data and image data through the communication unit 117, and the operation device 3 may display text data and image data received from the head-mounted display device 100 on the LCD 303. In this case, the operation state (status) of the head-mounted display device 100 is displayed on the LCD 303 so that the user can easily grasp the operation state of the head-mounted display device 100. For example, it is useful when the user wants to know the operating state of the head-mounted display device 100 without wearing the head-mounted display device 100. The operation device 3 may output sound from the speaker 370 according to data received from the head-mounted display device 100, or may generate vibration by moving the vibrator 369.

  Further, the device control unit 183 transmits control data instructing execution of imaging and transmission of captured image data to the operation device 3, and the operation device 3 performs imaging and displays the captured image data on the image display unit 20. You can also In this case, imaging can be performed from an angle that cannot be captured by the camera 61 included in the head-mounted display device 100, and the captured image can be displayed on the image display unit 20. Further, the control unit 140 may analyze the captured image data.

  In the display system 1, even if the NFC unit 362 of the operation device 3 reads data recorded on a non-contact type IC card or an RFID tag, the read data is transmitted from the wireless communication unit 361 to the communication unit 117. Good. In this case, the balance or the like of the IC card type electronic money can be displayed by the image display unit 20.

  Thus, in the display system 1, the see-through head-mounted display device 100 and the clock-type operation device 3 are combined, and the head-mounted display device 100 is operated by operating the operation device 3. As a result, hands-free operation is realized except during operation, and operation can be performed with natural movement even during operation. The operation device 3 can be used as a wristwatch by operating in the clock mode. By using the operation device 3 as a user interface for operation, hands-free operation is realized except during operation, and operation can be performed with natural movement even during operation. In addition, the display on the image display unit 20 allows the user to see information related to the operation device 3, and it is not necessary to look directly at the LCD 303. Therefore, there is an advantage that it is not necessary to use a hand to view the display on the LCD 303.

  Further, when the touch panel 304 supports a so-called multi-touch operation, the display image of the head-mounted display device 100 is enlarged or reduced in response to an operation for expanding or reducing the interval between a plurality of operation points on the touch panel 304. Alternatively, a rotating operation may be performed.

  In the display system 1, the head-mounted display device 100 can detect a relative position with the operation device 3. Although various methods can be used for this position detection, in the present embodiment, the head-mounted display device 100 images the range including the operation device 3 and specifies the relative position based on the captured image data. An example will be described.

FIG. 9 is a flowchart showing the operation of the head-mounted display device 100, and particularly shows position detection processing for detecting the relative position between the head-mounted display device 100 and the operation device 3.
The control unit 140 controls the camera 61 by the imaging processing unit 181 and executes imaging (step S51). The control unit 140 acquires captured image data of the camera 61, analyzes the acquired captured image data, and detects an image of the operation device 3 (step S52). Thereby, in the captured image of the camera 61, the position where the operation device 3 is reflected and the shape of the image of the operation device 3 can be detected.

  Subsequently, the control unit 140 calculates the relative position between the operation device 3 and the camera 61 based on the position and size of the image of the operation device 3 in the captured image (step S53). That is, the control unit 140 calculates the position of the operation device 3 with respect to the central axis of the angle of view of the camera 61, for example, the optical axis of the imaging lens of the camera 61, from the image position of the operation device 3. In addition, the control unit 140 obtains the distance between the operation device 3 and the imaging surface of the camera 61 from the size of the image of the operation device 3 in the captured image. Here, the control unit 140 may perform the calculation using the imaging conditions (zoom magnification, etc.) of the camera 61.

In step S53, after obtaining the relative position between the camera 61 and the operation device 3, the control unit 140 converts the obtained relative position into the relative position between the head-mounted display device 100 and the operation device 3. Good.
Since the head-mounted display device 100 and the operation device 3 are larger than the coordinates in the captured image, it is realistic to set a reference point. That is, the relative position between the head-mounted display device 100 and the operation device 3 is obtained as the relative position between the reference point set in the head-mounted display device 100 and the reference point set in the operation device 3. . When the reference point in the head-mounted display device 100 is a point that overlaps the optical axis on the imaging surface of the camera 61, the relative position obtained in step S53 remains as it is between the operation device 3 and the head-mounted display device 100. Relative position. When the reference point of the head-mounted display device 100 is at a position different from the imaging surface of the camera 61, the control unit 140 may perform the above-described conversion. The reference point in the operation device 3 may be the center of the bezel 301, for example.

  The control unit 140 identifies the orientation of the LCD 303 of the operation device 3 based on the shape of the image of the operation device 3 in the captured image (step S54). In step S <b> 54, the control unit 140 can calculate the orientation based on the shape of the bezel 301 of the operation device 3 in the captured image, the entire shape of the operation device 3 including the band unit 300, or the shape of the LCD 303. Also, for example, as shown in FIG. 2A, when the operation device 3 displays a graphic such as an arrow on the LCD 303 or when the time is displayed on the LCD 303, the display image of the LCD 303 is used. You may ask for direction.

  Thereafter, the control unit 140 includes data indicating the relative position between the operation device 3 calculated in step S53 and the head-mounted display device 100, and the operation device 3 corresponding to the head-mounted display device 100 specified in step S54. Data indicating the direction is generated. The control unit 140 stores the relative position data including the generated data in the storage unit 120 (step S55).

For example, in the operation shown in step S35 of FIG. 6, the AR display control unit 185 causes the operation device 3 to be displayed while the user of the head-mounted display device 100 is looking at the operation device 3 through the image display unit 20. AR content is displayed so that it appears to overlap. Here, the AR display control unit 185 determines the relative position between the operation device 3 and the head-mounted display device 100 detected by the operation shown in FIG. 9 and the direction of the operation device 3 with respect to the head-mounted display device 100. Based on the above, the display position of the AR content is determined. In this process, the AR display control unit 185 determines the display position of the AR content corresponding to the position of the object based on the positional relationship between the imaging range of the camera 61 and the display area of the image display unit 20. This positional relationship can be obtained from the relative position data stored in the storage unit 120.
The AR display control unit 185 may adjust the display size of the AR content according to the distance between the operation device 3 and the head-mounted display device 100. In addition, the AR display control unit 185 may adjust the display position of the AR content in accordance with the direction of the LCD 303 of the operation device 3 with respect to the head-mounted display device 100, or deform an image included in the AR content. May be.

  The method for detecting the relative position of the head-mounted display device 100 with the operation device 3 is not limited to the method shown in FIG. 9. For example, the operation device 3 and the head-mounted display device 100 perform data communication. And mutual detection results may be used. An example of this operation is shown in FIG.

FIG. 10 is a flowchart showing the operation of the display system 1, (A) shows the operation of the operation device 3, and (B) shows the operation of the head-mounted display device 100.
The operation in FIG. 10 is executed when the operation device 3 is in the UI mode, for example.
The control unit 350 of the operation device 3 detects the user's line of sight by performing imaging with the cameras 311 and 311 (step S61). The control unit 350 performs imaging and analysis of the captured image until a line of sight is detected (step S61; NO).

  When the line of sight is detected (step S61; YES), the control unit 350 performs imaging with the cameras 311 and 311 (step S62), and acquires captured image data (step S63). The control unit 350 transmits the captured image data by the wireless communication unit 361 (step S64). In step S62, imaging may be performed using only one of the cameras 311 and 311. Alternatively, both of the two cameras 311 and 311 may be captured, and both captured image data may be acquired and transmitted. Good.

  The control unit 140 receives captured image data from the operation device 3 through the communication unit 117 (step S71), analyzes the received captured image data, and detects an image of the head-mounted display device 100 (step S72). . Thereby, in the captured image, the position where the head-mounted display device 100 is reflected and the shape of the image of the head-mounted display device 100 can be detected.

  Subsequently, the control unit 140 calculates the relative position between the operation device 3 and the head-mounted display device 100 based on the position and size of the image of the head-mounted display device 100 in the captured image (step S73). For example, the control unit 140 calculates the position of the head-mounted display device 100 with respect to the center axis of the angle of view of the cameras 311 and 311 from the position of the image of the head-mounted display device 100. In addition, the control unit 140 obtains the distance between the head-mounted display device 100 and the operation device 3 from the size of the image of the head-mounted display device 100 in the captured image. Here, the control unit 140 may perform the calculation using the imaging conditions (such as zoom magnification) of the cameras 311 and 311.

In step S <b> 73, the control unit 140 may perform detection with higher accuracy using captured images of the plurality of cameras 311 and 311. For example, the parallax of the plurality of captured images may be calculated, and the distance between the operation device 3 and the head-mounted display device 100 may be obtained from the parallax.
Thereafter, the control unit 140 generates data indicating the relative position between the operation device 3 calculated in step S73 and the head-mounted display device 100 (step S74). The control unit 140 stores relative position data including the generated data in the storage unit 120.

  As described above, the operation device 3 and the head-mounted display device 100 may be linked to perform processing for obtaining a relative position between the operation device 3 and the head-mounted display device 100. In the operation of FIG. 10, the processing of steps S72 to S73 may be executed by the control unit 350, and the control unit 350 may generate data indicating the relative position and transmit the data to the head-mounted display device 100.

  As described with reference to FIGS. 9 and 10, the head-mounted display device 100 can detect the position of the operation device 3 with respect to the head-mounted display device 100. The head-mounted display device 100 can display AR content corresponding to the position of the operation device 3. Furthermore, by repeatedly executing the detection of the relative position, the movement of the head-mounted display device 100 can be obtained with the operation device 3 as a reference. Further, for example, the screen displayed on the image display unit 20 may be changed in accordance with the position and orientation of the operation device 3. In this case, changing the relative position between the operation device 3 and the head-mounted display device 100 corresponds to an operation on the head-mounted display device 100.

For example, in step S35 of FIG. 6, the AR display control unit 185 appears to overlap the operation device 3 while the user of the head-mounted display device 100 is looking at the operation device 3 through the image display unit 20. Thus, the AR content is displayed. Here, the AR display control unit 185 displays the AR content display position based on, for example, the relative position between the operation device 3 and the head-mounted display device 100 and the orientation of the operation device 3 with respect to the head-mounted display device 100. To decide. In this process, the AR display control unit 185 determines the display position of the AR content corresponding to the position of the object based on the positional relationship between the imaging range of the camera 61 and the display area of the image display unit 20. This positional relationship can be obtained from relative position data stored in the storage unit 120, for example.
The AR display control unit 185 may adjust the display size of the AR content according to the distance between the operation device 3 and the head-mounted display device 100. In addition, the AR display control unit 185 may adjust the display position of the AR content in accordance with the direction of the LCD 303 of the operation device 3 with respect to the head-mounted display device 100, or deform an image included in the AR content. May be.

  Further, the control unit 140 determines the detection axes (X, Y, Z axes in FIG. 1) of the 9-axis sensor 66 and the LCD 303 based on the relative position and orientation with the operation device 3 specified by the operation of FIG. Correlation with the direction (inclination) may be performed. Thereby, the display image on the image display unit 20 and the display image on the LCD 303 can be changed in conjunction with both the tilt of the head-mounted display device 100 and the tilt of the LCD 303.

  In addition, as an example in which the operation device 3 and the head-mounted display device 100 operate in conjunction with each other, for example, a marker is displayed on the LCD 303 under the control of the control unit 350, and this marker is displayed on the head-mounted display device. The aspect which 100 recognizes is mentioned. The control unit 140 detects a marker from the captured image of the camera 61. When the marker displayed on the LCD 303 is a marker that can include information such as a barcode or a two-dimensional code, the control unit 140 can acquire information from the reading result. The control unit 140 can perform an operation of establishing communication with the operation device 3 by the communication unit 117 based on the acquired information. Further, when there are a plurality of operation devices 3, it is possible to identify these operation devices 3. In this case, an operation for detecting the relative position between the specific operation device 3 and the head-mounted display device 100 and an operation for establishing communication with the specific operation device 3 are possible.

  As described above, the display system 1 according to the embodiment to which the present invention is applied includes the head-mounted display device 100 and the operation device 3 configured separately from the image display unit 20. The head-mounted display device 100 includes a transmissive image display unit 20 that is mounted on the user's head and allows the user to visually recognize the outside scene. The operation device 3 includes a bezel 301, a touch panel 304, a button 305, a crown type operator 307, and a button 309 as an operation unit that receives a user's operation. The operation unit may include a 9-axis sensor 368, or the band unit 300 having the 9-axis sensor 368 may be used as the operation unit. The head-mounted display device 100 includes a control unit 140 that controls the display of the image display unit 20. Based on an operation on the operation device 3, the control unit 140 causes the image display unit 20 to perform an AR display related to an outside scene that passes through the image display unit 20. According to this configuration, when the head-mounted display device 100 performs AR display related to the outside scene, the operation of the operation device 3 separate from the head-mounted display device 100 is reflected, which is intuitive. The AR display can be controlled by an input operation.

For example, in the operation device 3, after a function is assigned to the operation unit of the band unit 300 according to the setting illustrated in FIG. 7A, an operation on the operation device 3 such as touching the operation unit or moving a finger. May be detected by the head-mounted display device 100. Specifically, the control unit 140 may detect an operation on the operation device 3 based on the captured image data of the camera 61.
In addition, the operation device 3 may include a wireless communication unit 361 that transmits data indicating the operation received by the operation unit to the head-mounted display device 100. In this case, the control unit 140 causes the image display unit 20 to perform AR display related to the outside scene that is transmitted through the image display unit 20 in a manner associated with the operation unit based on the data transmitted from the operation device 3.

  Further, the control unit 140 detects the operation of the operation unit based on the data transmitted from the operation device 3, and changes the AR display being displayed on the image display unit 20 in accordance with the detected operation. For this reason, the AR display can be changed by operating the operation device 3 while performing the AR display.

The control unit 140 may be capable of setting the allocation of AR display to the operation unit.
In this case, the control unit 140 may be able to execute processing for updating the allocation of the AR display to the operation unit.

In addition, the operation unit includes a touch panel 304 that detects a contact operation. The touch panel 304 has a plurality of operation areas, and the control unit 140 causes the image display unit 20 to display an AR display assigned to the operation area operated on the touch panel 304. For this reason, by performing a touch operation with the operation device 3, various operations for the AR display can be performed.
The control unit 350 included in the operation device 3 may detect the movement of the operation device 3 as an operation.
The head-mounted display device 100 can be operated by moving the operation device 3.

  Further, the operation device 3 may include the camera 311, and data including a captured image of the camera 311 may be transmitted to the head-mounted display device 100 by the wireless communication unit 361. In this case, the captured image captured by the operation device 3 separate from the head-mounted display device 100 can be displayed by the head-mounted display device 100. In the display system 1, the head-mounted display device 100 can be authenticated by the operation device 3.

In addition, this invention is not restricted to the structure of the said embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.
In the present embodiment, the example in which the operation device 3 used as the user interface of the head-mounted display device 100 has the shape of a wristwatch has been described. The present invention is not limited to this, and any shape that is suitable for being worn on the body of a user who uses the head-mounted display device 100 may be used. For example, it can be a ring type, brooch type, pendant or necklace type, or pen type device. Moreover, it is good also as a headphone type | mold, earphone type | mold, or headset type device with which a user's head and ear | edge are mounted | worn. In this case, the function as the operation device of the present invention may be mounted on headphones or earphones that output sound. The function as the operation device of the present invention may be implemented in a headset including a speaker that outputs audio and a microphone that collects audio uttered by the user. In this configuration, the headphones, earphones, and speakers may be bone conduction speakers, and the microphone may be a bone conduction microphone. In this case, the device preferably includes at least an operation element that detects a contact operation or a pressing operation, and includes a wireless communication unit 361, an LED 313, a camera 311, and a 9-axis sensor 368. In these configurations, the same effects as those in the above-described embodiment can be obtained except for operations related to display.

Moreover, although the head mounted display apparatus 100 of the said embodiment was set as the structure fixed to a user's body by the band part 300, the structure of the operation device of this invention is not limited to this. That is, it is not essential that the operation device is fixed by the band unit 300 or the like, as long as it can be attached to the user's body. Therefore, it may be a pendant or necklace shape as described above, or a shape imitating other accessories, may be a hat or helmet shape, or may be other clothing.
Furthermore, the operating device of the present invention is not limited to a mode in which it is directly attached to or fixed to the user's body. For example, an apparatus corresponding to an operation device may be fixed to a wristwatch, an accessory, clothes, a bag, a hat, or a helmet that is attached or fixed to the user's body. The operation device in this case can be configured, for example, by removing the band unit 300 from the operation device 3.

  The display system 1 may be configured to include a plurality of operation devices. For example, the operation device 3 may include a different type of operation device such as a wristwatch-type operation device and a pen-type operation device. In this case, the display screen of the head-mounted display device 100 may be changed based on operations on a plurality of operation devices. Examples of the method in which the head-mounted display device 100 detects operations on a plurality of operation devices include a method of transmitting data indicating operations from the plurality of operation devices.

Further, the operation device 3 transmits data indicating the detection value of the nine-axis sensor 368 that is an inertial sensor to the head-mounted display device 100, so that even if the display system 1 operates using the movement of the operation device 3 as an operation. Good.
For example, using the detection value of the 9-axis sensor 368 and the detection value of the 9-axis sensor 66 included in the head-mounted display device 100, the control unit 140 transmits an external scene that can be viewed through the image display unit 20. You may perform matching with the display area of the image display part 20. FIG. In this case, association can be performed with higher accuracy by using detection values of sensors provided in a plurality of apparatuses. This association can be used when the AR content is displayed on the image display unit 20.

  The operation device 3 may include a sensor that measures the heart rate of the user. Specifically, a sensor or the like that detects light by irradiating light on the user's arm can be used. In this case, the operation device 3 may transmit data relating to the measured heart rate to the head-mounted display device 100, and display data relating to the heart rate on the head-mounted display device 100. For example, display of the result of statistical processing of heart rate, real-time display of heart rate, heart rate and detection value of 9-axis sensor 368, and display data related to user activity and physical condition management Also good. Further, the user's excitement level and fatigue level may be estimated based on the heart rate, and the display color in the image display unit 20 and the content of the image to be displayed may be changed to bring comfort to the user.

  In the above-described embodiment, the configuration in which the operation device 3 is an electronic device that is operated by the power supply unit 360 has been described as an example. However, the present invention is not limited to this. For example, a marker for specifying the position of the bezel 301 in the rotational direction may be provided at a position adjacent to the bezel 301 on the operation device 3 and / or the planar portion 300A. In this case, the device control unit 183 can determine the rotation direction and the rotation amount of the bezel 301 based on the captured image data captured by the camera 61. Even when the operation device 3 includes the power supply unit 360, the head-mounted display device 100 may obtain the rotation direction and the rotation amount of the bezel 301 by the above operation. This operation can be realized, for example, as a power saving operation mode operation that suppresses the consumption of the power supply unit 360. When the remaining capacity of the power supply unit 360 is small, the display system 1 can be shifted to the power saving operation mode under the control of the operation device 3 or the head-mounted display device 100.

  Further, instead of the image display unit 20 of the above-described embodiment, for example, an image display unit of another method such as an image display unit worn like a hat may be adopted, corresponding to the left eye of the user. What is necessary is just to provide the display part which displays an image, and the display part which displays an image corresponding to a user's right eye. The display device of the present invention may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet. In this case, the part for positioning the position with respect to the user's body and the part positioned with respect to the part can be used as the mounting portion.

Furthermore, in the above-described embodiment, the configuration in which the image display unit 20 and the control device 10 are separated and connected via the connection unit 40 has been described as an example. The present invention is not limited to this, and the control device 10 and the image display unit 20 may be configured integrally and mounted on the user's head.
As the control device 10, a notebook computer, a tablet computer, or a desktop computer may be used. The control device 10 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like. Further, the control device 10 may be configured separately from the image display unit 20, and various signals may be transmitted and received between the control device 10 and the image display unit 20 by wireless communication.

  For example, as a configuration for generating image light in the image display unit 20, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used. Further, as a configuration for generating image light, LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like can be used.

The optical system that guides the image light to the user's eyes may include an optical member that transmits external light that is incident on the apparatus from the outside and that is incident on the user's eyes together with the image light. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field. Further, a scanning optical system that scans a laser beam or the like to obtain image light may be employed. Further, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes.
For example, the present invention can also be applied to a laser retinal projection type head mounted display. That is, the light emitting unit includes a laser light source and an optical system that guides the laser light source to the user's eyes, and the laser light is incident on the user's eyes to scan the retina and form an image on the retina. A configuration that allows the user to visually recognize an image may be employed.
Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses MEMS display technology. That is, a light emitting unit includes a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system. You may prepare. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the virtual image with the eyes, thereby recognizing the image. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. .

In addition, at least a part of the functional blocks illustrated in FIGS. 3 and 4 may be realized by hardware, or may be configured by cooperation of hardware and software. Further, the program executed by the control unit 140 may be stored in the storage unit 120 or the storage device in the control device 10, or the program stored in the external device is acquired via the communication unit 117 or the interface 125. It is good also as a structure to execute.
Moreover, only the operation part 111 among the structures formed in the control apparatus 10 may be formed as a single user interface (UI). Further, the configuration formed in the control device 10 may be formed in the image display unit 20 in an overlapping manner. For example, a control unit corresponding to the control unit 140 may be formed in both the control device 10 and the image display unit 20. Moreover, it is good also as a structure by which the function which the control part 140 formed in the control apparatus 10 and the CPU formed in the image display part 20 perform separately is divided | segmented.

  In addition, the present invention can be realized in various forms other than the head-mounted display device 100. For example, a control method of the head-mounted display device 100, an information system having the head-mounted display device 100, a control method of the head-mounted display device 100, a computer program for realizing the information system, and the computer program Can be realized in the form of a recording medium on which the computer program is recorded, a server device connected to a communication line such as the Internet for distributing the computer program, a data signal including the computer program and embodied in a carrier wave.

  DESCRIPTION OF SYMBOLS 1 ... Display system, 2 ... Frame, 3 ... Operation device, 10 ... Control apparatus, 20 ... Image display part (display part), 21 ... Right holding part, 22 ... Right display drive part, 23 ... Left holding part, 24 ... Left display drive unit, 26 ... right optical image display unit, 28 ... left optical image display unit, 61 ... camera, 66 ... 9-axis sensor, 100 ... head-mounted display device (display device), 117 ... communication unit, 120 ... Storage unit 140 ... Control unit 150 ... Operating system 160 ... Image processing unit 170 ... Display control unit 181 ... Imaging processing unit 183 ... Device control unit 185 ... AR display control unit 187 ... Audio processing unit , 300 ... Band part, 300 A ... Flat part, 301 ... Bezel, 303 ... LCD, 304 ... Touch panel (touch operation part), 305 ... Button, 307 ... Crown-type operation element, 309 ... Button, 310 ... Faculty, 311 ... Camera (imaging unit), 313 ... LED, 331-334, 341-345 ... Function display, 350 ... Control unit, 351 ... Operation detection unit, 352 ... Output control unit, 353 ... Imaging control unit, 354 ... Communication control unit, 355 ... voice processing unit, 360 ... power supply unit, 361 ... wireless communication unit, 362 ... NFC unit, 365 ... microphone, 366 ... button operation unit, 367 ... rotation detection unit, 368 ... 9-axis sensor, 369 ... Vibrator, 370 ... Speaker.

Claims (13)

A display device that is mounted on the user's head and has a transmissive display unit that allows the user to see through the outside scene, and an operation device configured separately from the display unit;
The operating device is:
An operation unit having an operation element and receiving an operation of the user with respect to the operation element;
A communication unit that transmits data indicating the operation accepted by the operation unit to the display unit,
The display device includes a control unit that performs AR display related to the outside scene that is transmitted through the display unit by the display unit based on an operation of the operation device.
The control unit executes a setting process for assigning a function to be executed by the display device in response to an operation of the operator, and a function assigned to an operation on the operator based on data transmitted by the operation device. Processing, and
In the setting process, AR display assigned to the operation element is performed at a position corresponding to the operation device in the field of view of the user that passes through the display unit ,
A display system characterized by The control unit of the display device performs an AR display related to the outside scene that is transmitted through the display unit by the display unit in a mode associated with the operation unit based on an operation of the operation device.
The display system according to claim 1. The control unit detects an operation of the operation unit based on data transmitted from the operation device, and changes the AR display being displayed on the display unit in response to the detected operation;
The display system according to claim 1, wherein: The operation device includes the operation element rotatable in a circumferential direction, and transmits operation data including a rotation direction of the operation element by the communication unit,
The control unit of the display device AR-displays function candidates assigned to the operation element in the setting process, and selects the function candidates according to the rotation of the operation element indicated by operation data transmitted from the operation device. To do,
The display system according to claim 1, wherein: The control unit of the display device enlarges the display of the display unit according to the rotation of the operation unit indicated by the operation data transmitted from the operation device as a process of executing a function assigned to the operation on the operation unit. Perform at least one of the following functions: reduction, rotation, copy, and page feed,
The display system according to claim 4. The control unit can set the allocation of the AR display to the operation unit;
Display system according to any one of claims 1 to 5, wherein. The control unit is capable of executing a process of updating the allocation of the AR display to the operation unit;
The display system according to claim 6 . The control unit of the display device obtains a relative position between the operation device and the display device, and performs AR display on the display unit based on the obtained relative position;
Display system according to any one of claims 1 to 7, characterized in. The operation unit includes a touch operation unit that detects a contact operation,
The touch operation unit has a plurality of operation areas,
The control unit displays the AR display assigned to the operation area operated by the touch operation unit by the display unit.
The display system according to claim 1 8, wherein. The operation device has a mounting portion fixed to the user's body;
Display system according to any one of claims 1 9, characterized in. The operation device includes an imaging unit, and transmits data including a captured image of the imaging unit to the display device by the communication unit;
Display system according to any one of claims 1 to 10, characterized in. A computer-executable program for controlling a display device having a display unit mounted on a user's head,
The said display unit is configured as a separate body, when the operating device that have a operator has accepted the operation of the user with respect to the operator, on the basis of the operation of the operating device, the outside scene transmitted through the display unit There line by the display unit of AR display associated with,
A setting process for assigning a function to be executed by the display device in response to an operation of the operator, and a process for executing a function assigned to an operation on the operator based on data transmitted by the operation device And
Wherein the setting process, the the position corresponding to the operation device in the user's field of view transmitted through the display unit, AR display the line Ukoto assigned to the operating element,
A program characterized by A display device having a display unit mounted on the head of a user, and the display unit is configured as a separate body, in a control method of controlling by using the operation device that having a manipulator,
Receiving an operation of the user with respect to the operator by the operation unit of the operation device,
By the display device, based on the operation of the operating device, it has rows AR display associated with the outside scene by the display unit for transmitting said display unit,
A setting process for assigning a function to be executed by the display device in response to an operation of the operator, and a process for executing a function assigned to an operation on the operator based on data transmitted by the operation device And
Wherein the setting process, the the position corresponding to the operation device in the user's field of view transmitted through the display unit, AR display the line Ukoto assigned to the operating element,
A control method of a display device characterized by the above.

Images