JP2021057747A - Display system, image display device, image display method, and program - Google Patents

Abstract

To allow an image highly useful for a user to be displayed at appropriate timing.SOLUTION: A display system 1 comprises: a head mounted display (HMD) 100 including an image display section 20, a first microphone 61 for inputting sound to be emitted by a sound source 400, and a second microphone 63 arranged at a position different from that of the first microphone 61 so as to input sound to be emitted by the sound source 400; and a smart phone 300 for determining an image to be displayed in an image display section 20 and display timing to allow the image display section 20 to display the image based on sound signals to be input from the first microphone 61 and the second microphone 63.SELECTED DRAWING: Figure 5

Description

The present invention relates to a display system, an image display device, an image display method and a program.

A method of detecting the position of a sound source or the like based on the voice input by a plurality of voice input units is known. For example, Patent Document 1 discloses a sound information display device including a plurality of microphones. This sound information display device determines the position of the sound source, the loudness and frequency of the sound output from the sound source based on the sound collection information of a plurality of microphones, and the position in the visual area corresponding to the determined position of the sound source. In addition, an image of a circle whose loudness corresponds to the determined loudness and whose frequency difference is displayed as a color difference is displayed.

Japanese Unexamined Patent Publication No. 2012-133250

However, Patent Document 1 merely displays an image corresponding to the frequency and loudness of the sound emitted by the sound source, and there is a demand for a technique for displaying an image that is highly useful to the user.

One aspect of solving the above problems is to arrange the image display unit, the first audio input unit that inputs the audio emitted from the sound source and outputs the first audio signal, and the first audio input unit at different positions. An image display device including a second audio input unit that inputs the audio and outputs a second audio signal, and the image display unit based on the first audio signal and the second audio signal. It is a display system including a control device for controlling an image to be displayed on the screen.

In the display system, the control device has the image to be displayed on the image display unit and the image to be displayed on the image display unit based on the time difference between the timing at which the first audio signal is input and the timing at which the second audio signal is input. The configuration may be such that the display timing of the image is controlled.

In the display system, when the time difference between the first audio signal and the second audio signal is equal to or less than the first threshold value, the control device causes the image display unit to display the first image, and the first image is displayed. When the time difference between the one audio signal and the second audio signal is equal to or less than the second threshold value smaller than the first threshold value, the image display unit may display the second image.

In the display system, the control device determines the display position and size of the image to be displayed on the image display unit based on the time difference between the input timing of the first audio signal and the input timing of the second audio signal. It may be configured to control at least one of them.

In the display system, the control device includes a storage unit that stores image data in association with identification information of the image data, and the control device is a watermark superimposed on an audio signal output from the sound source. The information may be extracted, the identification information may be acquired from the extracted watermark information, and an image based on the image data associated with the acquired identification information may be displayed on the image display unit.

In the display system, the control device may be configured to determine the timing of displaying an image on the image display unit based on the phase difference between the first audio signal and the second audio signal.

In the display system, the control device includes a storage unit that stores range information indicating a frequency range of audio output from the component when the component is operating normally, and includes the first audio input unit and the control device. When the frequencies of the first audio signal and the second audio signal input from the second audio input unit are not within the frequency range indicated by the range information, the component arranged in the direction of the specified sound source fails. It may be configured to display a display indicating that is occurring.

Another aspect of solving the above-mentioned problems is that the image display unit, the first audio input unit that inputs audio and outputs the first audio signal, and the first audio input unit are arranged at different positions from the audio. An image to be displayed on the image display unit based on an image display device including a second audio input unit that inputs and outputs a second audio signal, the first audio signal, and the second audio signal. It is an image display device including a control unit for controlling the above.

Another aspect of solving the above problems is a step of inputting a first audio signal output by the first audio input unit that has input the sound of the sound source, and a first position arranged at a position different from the first audio input unit. 2 The step of inputting the second audio signal that the audio input unit inputs and outputs the audio, and the image to be displayed on the image display unit based on the input first audio signal and the second audio signal. It is an image display method having a step to control.

Another aspect of solving the above-mentioned problems is a procedure of inputting a first audio signal output by the first audio input unit that inputs the audio of the sound source to the computer and arranging the computer at a position different from the first audio input unit. The second audio input unit displays the image on the image display unit based on the procedure for inputting the second audio signal for inputting and outputting the audio, and the input first audio signal and the second audio signal. It is a program to execute a procedure for controlling an image to be input.

The figure which shows an example of the structure of the display system of 1st Embodiment. The figure which shows an example of the structure of the optical system of an image display part. The perspective view which shows an example of the main part structure of an image display part. The figure which shows an example of the structure of each part which constitutes HMD. The figure which shows an example of the structure of a smartphone. The figure which shows an example of the structure of a content table. The figure which shows the case where the distance between a sound source and a plurality of microphones is different. The figure which shows the case where the distance between a sound source and a plurality of microphones is the same or substantially the same. The figure which shows the display image displayed in the display area. A diagram showing a dinosaur skeleton specimen as an example of an exhibit. The figure which shows the content image which can be seen when moving to the front of an exhibit. The flowchart which shows the operation of the 2nd control part. The figure which shows an example of the structure of the HMD of 2nd Embodiment. The figure which shows an example of the structure of the smartphone of 2nd Embodiment. The figure which shows an example of the part information table. The figure which shows an example of the display image displayed in a display area.

Hereinafter, embodiments will be described with reference to the drawings.

[Display system configuration]
FIG. 1 is a diagram showing a schematic configuration of a display system 1 according to the present embodiment.
As shown in FIG. 1, the display system 1 includes an HMD (Head Mounted Display) 100. The HMD 100 includes an image display unit 20 mounted on the head of the user U and a connecting device 10, and the image display unit 20 provides a virtual image to the user U while being mounted on the head of the user U. It is a device for visual recognition.
The HMD 100 corresponds to an example of an "image display device". In the following description, the user U refers to a user who wears and uses the HMD100.

The connecting device 10 includes a connector 11A and a connector 11D in a box-shaped case. The image display unit 20 is connected to the connector 11A via the connection cable 40. Hereinafter, when the connectors 11A and 11D are not distinguished, they may be referred to as the connector 11. The case of the connecting device 10 can also be called a housing or a main body.

The display system 1 is a system configured by connecting a smartphone 300 to the HMD 100. The connector 11D is an interface to which the smartphone 300 is connected in the HMD 100. That is, in the present embodiment, the smartphone 300 is connected to the connector 11D. The smartphone 300 corresponds to an example of a "control device". The control device may be portable to the user U and may include a display unit for displaying an image and a computer. For example, as a control device, a PDA (Personal Digital Assistant) terminal, a tablet-type personal computer, or the like can be connected to the connecting device 10.

The connector 11 is a wired interface for connecting a communication cable, and the connecting device 10 is connected to an external device by the communication cable. The connector 11A includes a terminal for connecting the connection cable 40 and an interface circuit for transmitting and receiving signals via the connector 11A.

The connector 11A is provided to connect the image display unit 20 to the connecting device 10. The connection cable 40 has a function of supplying power to the image display unit 20 from the connection device 10 and a function of transmitting and receiving data to and from the image display unit 20 and the connection device 10.

The connector 11D includes an interface circuit in which image data is input from the smartphone 300 and sensor data can be output to the smartphone 300. The smartphone 300 reproduces the content data recorded in the non-volatile storage unit. The connector 11D is, for example, a connector conforming to a known communication interface standard.
In the present embodiment, as an example, the connector 11D includes an interface circuit corresponding to input / output of image data and various data, and the smartphone 300 is connected via the USB cable 46.

As the connector 11D, for example, a USB (Universal Serial Bus) -Type C standard connector can be adopted. The interface circuit corresponding to USB-TypeC is capable of transmitting data according to the USB3.1 standard and supplying DC power within 20 volts and 5 amperes.
Further, as a function of the alternative mode of USB-TypeC, HDMI (registered trademark: High Definition Multimedia Interface) standard image data, MHL (Mobile High-definition Link) standard image data and the like can be transmitted. The smartphone 300 can supply power, send and receive data, and supply streaming data of images and sounds via the USB cable 46. An alternative mode of USB-Type C is known as Alternative mode.

The image display unit 20 has a spectacle shape in the present embodiment. The image display unit 20 includes a right display unit 22, a left display unit 24, a right light guide plate 26, and a left light guide plate 28 in a main body having a right holding unit 21, a left holding unit 23, and a front frame 27. ..

The right holding portion 21 and the left holding portion 23 extend rearward from both ends of the front frame 27 and hold the image display portion 20 on the head of the user U. Of both ends of the front frame 27, the end located on the right side of the head when the image display unit 20 is attached is referred to as the end ER, and the end located on the left side is referred to as the end EL. The right holding portion 21 is provided so as to extend from the end ER of the front frame 27 to a position corresponding to the right head of the user U when the image display portion 20 is attached. The left holding portion 23 is provided so as to extend from the end portion EL to a position corresponding to the left head of the user U in the mounted state of the image display portion 20.

The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is located in front of the right eye of the user U when the image display unit 20 is attached, and allows the right eye to visually recognize the image. The left light guide plate 28 is located in front of the left eye of the user U when the image display unit 20 is attached, and allows the left eye to visually recognize the image.

The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other, and this connection position is used when the user U attaches the image display unit 20. Corresponds to the eyebrows of person U.
The front frame 27 may be provided with a nose pad portion that comes into contact with the nose of the user U while the image display unit 20 is attached at the connection position between the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the head of the user U by the nose pad portion, the right holding portion 21, and the left holding portion 23. Further, a belt that is in contact with the back of the head of the user U when the image display unit 20 is attached may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 can be held on the head of the user U by the belt.

Each of the right display unit 22 and the left display unit 24 is a module in which an optical unit and peripheral circuits are unitized.
The right display unit 22 is a unit for displaying an image by the right light guide plate 26, is provided on the right holding unit 21, and is located near the right head of the user U in the mounted state. The left display unit 24 is a unit for displaying an image by the left light guide plate 28, is provided on the left holding unit 23, and is located in the vicinity of the left head of the user U in the mounted state. The right display unit 22 and the left display unit 24 can also be collectively referred to as a “display drive unit”.

The right light guide plate 26 and the left light guide plate 28 are optical units formed of a light-transmitting resin or the like, and guide the image light output by the right display unit 22 and the left display unit 24 to the eyes of the user U. The right light guide plate 26 and the left light guide plate 28 are, for example, prisms.
The left display unit 24 and the left light guide plate 28 correspond to an example of the “first display unit”. The right display unit 22 and the right light guide plate 26 correspond to an example of the “second display unit”.

The image light guided by the right light guide plate 26 and the external light transmitted through the right light guide plate 26 are incident on the right eye of the user U. Similarly, the image light guided by the left light guide plate 28 and the external light transmitted through the left light guide plate 28 are incident on the left eye.
The image light guided by the left light guide plate 28 corresponds to the first image P1. That is, the left display unit 24 and the left light guide plate 28 display the first image P1 on the outside view. The image light guided by the right light guide plate 26 corresponds to the second image P2. That is, the right display unit 22 and the right light guide plate 26 display the second image P2 on the outside view.
The second image P2 is an image corresponding to the first image P1. In the present embodiment, the second image P2 is, for example, the same image as the first image P1.
The first image P1 and the second image P2 will be described in detail later with reference to FIGS. 8 to 10.

An illuminance sensor 65 is arranged on the front frame 27 of the image display unit 20. The illuminance sensor 65 receives external light from the front of the user U who wears the image display unit 20.
The first camera 50 is arranged on the front frame 27 of the image display unit 20. The first camera 50 is provided at a position that does not block the external light transmitted through the right light guide plate 26 and the left light guide plate 28. In the example of FIG. 1, the first camera 50 is arranged at the connecting portion between the right light guide plate 26 and the left light guide plate 28, but may be arranged at the end ER side of the front frame 27, and the end EL It may be placed on the side.

The first camera 50 is a digital camera including an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor), an image pickup lens, or the like. The first camera 50 captures the outside view and generates the first captured image.

An LED (Light Emitting Diode) indicator 67 is arranged on the front frame 27. The LED indicator 67 is arranged at the end ER and lights up during the operation of the first camera 50 to notify that imaging is in progress.

The front frame 27 is provided with a distance sensor 64 and an infrared sensor 66. The distance sensor 64 detects the distance to the measurement object located in the preset measurement direction. The distance sensor 64 may be, for example, a light reflection type distance sensor having a light source such as an LED or a laser diode and a light receiving unit that receives the reflected light that the light emitted by the light source reflects on the object. Further, the distance sensor 64 may be an ultrasonic type distance sensor including a sound source that emits ultrasonic waves and a detection unit that receives ultrasonic waves reflected by the object to be measured. Further, the distance sensor 64 may use a laser range scanner, and in this case, the range can be measured over a wide range including the front of the image display unit 20.

Further, instead of the distance sensor 64, a plurality of cameras may be mounted on the HMD100 to measure the distance using the principle of triangular measurement, or receive radio waves such as Wi-Fi, Bluetooth, and radio beacons. The receiving unit may be provided in the HMD 100, and the distance to the object may be measured based on the intensity of the radio wave received by the receiving unit. Wi-Fi and Bluetooth are registered trademarks.
Further, as the distance sensor 64, for example, a laser radar such as LiDAR (Light Detection and Ranger, Laser Imaging Detection and Ranger) may be provided. The laser radar emits a pulsed laser and measures the distance by the reflected light or scattered light from the object.

The infrared sensor 66 detects infrared rays. Specifically, the infrared sensor 66 detects infrared rays emitted from the infrared communication unit of the smartphone 300. Further, the infrared sensor 66 may be configured as a part of the infrared communication unit. In this case, communication with the infrared communication unit of the smartphone 300 is possible. The infrared communication unit communicates according to, for example, the IrDA standard.

Each of the right display unit 22 and the left display unit 24 of the image display unit 20 is connected to the connecting device 10. In the HMD 100, the connection cable 40 is connected to the left holding unit 23, the wiring connected to the connection cable 40 is laid inside the image display unit 20, and each of the right display unit 22 and the left display unit 24 is connected to the connecting device 10. To.

The connection cable 40 includes an audio connector 36, and a headset 30 having a right earphone 32 and a left earphone 34 constituting stereo headphones and a first microphone 61 and a second microphone 63 is connected to the audio connector 36. The right earphone 32 is attached to the right ear of the user U, and the left earphone 34 is attached to the left ear of the user U. The right earphone 32 and the left earphone 34 can also be said to be audio output units.

The right earphone 32 and the left earphone 34 output audio based on the audio signal output by the connecting device 10.
The first microphone 61 and the second microphone 63 are arranged at different positions of the image display unit 20. For example, the first microphone 61 is arranged on the right display unit 22, and the second microphone 63 is displayed on the left display unit 24. The first microphone 61 and the second microphone 63 collect the sound and output the sound signal to the connecting device 10. The first microphone 61 and the second microphone 63 may be, for example, a monaural microphone, a stereo microphone, a directional microphone, or an omnidirectional microphone. The first microphone 61 corresponds to an example of the "first voice input unit" of the present invention, and the second microphone 63 corresponds to an example of the "second voice input unit" of the present invention.

The connection device 10 includes a brightness adjustment key 13, a brightness adjustment key 14, a volume adjustment key 15, and a volume adjustment key 16 as operation-operated parts operated by the user U. Each of the brightness adjustment key 13, the brightness adjustment key 14, the volume adjustment key 15, and the volume adjustment key 16 is composed of a hardware key. These operated portions are arranged on the surface of the main body of the connecting device 10, and are operated by, for example, the fingers of the user U.

The brightness adjustment keys 13 and 14 are hardware keys for adjusting the display brightness of the image displayed by the image display unit 20. The brightness adjustment key 13 instructs to increase the brightness, and the brightness adjustment key 14 instructs to decrease the brightness. The volume adjustment keys 15 and 16 are hardware keys for adjusting the volume of the sound output from the right earphone 32 and the left earphone 34. The volume adjustment key 15 instructs to increase the volume, and the volume adjustment key 16 instructs to decrease the volume.

[Structure of optical system of HMD image display unit]
FIG. 2 is a plan view of a main part showing the configuration of the optical system included in the image display unit 20. FIG. 2 illustrates the left eye LE and the right eye RE of the user U for the sake of explanation.
As shown in FIG. 2, the right display unit 22 and the left display unit 24 are symmetrically configured. The right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 that emits image light so that the right eye RE of the user U can visually recognize the image. Further, the right optical system 251 including a lens group for guiding the image light L emitted by the OLED unit 221 is provided. The image light L is guided to the right light guide plate 26 by the right optical system 251.

The OLED unit 221 has an OLED panel 223 and an OLED drive circuit 225 for driving the OLED panel 223. The OLED panel 223 is a self-luminous display composed of light emitting elements that emit light by organic electroluminescence and emit R (red), G (green), and B (blue) colored lights in a matrix. It is a panel. The OLED panel 223 includes a plurality of pixels, with one pixel as a unit including one R, G, and B element, and forms an image with pixels arranged in a matrix. The OLED drive circuit 225 selects the light emitting element included in the OLED panel 223 and energizes the light emitting element according to the control of the first control unit 120, so that the light emitting element of the OLED panel 223 emits light. The first control unit 120 will be described later with reference to FIG.
The OLED drive circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back side of the light emitting surface by bonding or the like. The OLED drive circuit 225 may be composed of, for example, a semiconductor device for driving the OLED panel 223, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 223. The temperature sensor 217 shown in FIG. 4 is mounted on this substrate.
The OLED panel 223 may have a configuration in which light emitting elements that emit white light are arranged in a matrix, and color filters corresponding to each of the R, G, and B colors are arranged in an overlapping manner. Further, in addition to the light emitting elements that emit R, G, and B colored lights, an OLED panel 223 having a WRGB configuration including a light emitting element that emits W (white) light may be used.

The right optical system 251 has a collimating lens that converts the image light L emitted from the OLED panel 223 into a light flux in a parallel state. The image light L converted into a luminous flux in a parallel state by the collimating lens is incident on the right light guide plate 26. A plurality of reflecting surfaces that reflect the image light L are formed in the optical path that guides the light inside the right light guide plate 26. The image light L is guided to the right eye RE side through a plurality of reflections inside the right light guide plate 26. A half mirror 261 (reflection surface) located in front of the right eye RE is formed on the right light guide plate 26. The image light L is reflected by the half mirror 261 and emitted from the right light guide plate 26 toward the right eye RE, and the image light L forms an image on the retina of the right eye RE, and the second image P2 is displayed to the user U. Make it visible.

Further, as a configuration for allowing the left eye LE of the user U to visually recognize the image, the left display unit 24 includes an OLED unit 241 that emits image light, a lens group that guides the image light L emitted by the OLED unit 241, and the like. It includes a system 252. The image light L is guided to the left light guide plate 28 by the left optical system 252.

The OLED unit 241 includes an OLED panel 243 and an OLED drive circuit 245 that drives the OLED panel 243. The OLED panel 243 is a self-luminous display panel configured in the same manner as the OLED panel 223. The OLED drive circuit 245 selects the light emitting element included in the OLED panel 243 and energizes the light emitting element according to the instruction of the first control unit 120, so that the light emitting element of the OLED panel 243 emits light.
The OLED drive circuit 245 is fixed to the back surface of the OLED panel 243, that is, the back side of the light emitting surface by bonding or the like. The OLED drive circuit 245 may be composed of, for example, a semiconductor device for driving the OLED panel 243, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 243. The temperature sensor 239 shown in FIG. 4 is mounted on this substrate.

The left optical system 252 has a collimating lens that converts the image light L emitted from the OLED panel 243 into a light flux in a parallel state. The image light L converted into a luminous flux in a parallel state by the collimating lens is incident on the left light guide plate 28. The left light guide plate 28 is an optical element on which a plurality of reflecting surfaces that reflect the image light L are formed, and is, for example, a prism. The image light L is guided to the LE side of the left eye through a plurality of reflections inside the left light guide plate 28. A half mirror 281 (reflection surface) located in front of the left eye LE is formed on the left light guide plate 28. The image light L is reflected by the half mirror 281 and emitted from the left light guide plate 28 toward the left eye LE, and the image light L forms an image on the retina of the left eye LE, and the first image P1 is displayed to the user U. Make it visible.

According to this configuration, the HMD 100 functions as a transmissive display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 are incident on the right eye RE of the user U. Further, the image light L reflected by the half mirror 281 and the external light OL transmitted through the half mirror 281 are incident on the left eye LE. In this way, the HMD 100 superimposes the image light L of the internally processed image and the external light OL and causes them to enter the eyes of the user U, and for the user U, the right light guide plate 26 and the left light guide plate 28 are watermarked. The outside view can be seen, and the image by the image light L is visually recognized by superimposing the outside view.
The half mirrors 261 and 281 are image extraction units that reflect the image light output by the right display unit 22 and the left display unit 24, respectively, and take out an image, and can be called a display unit.

The left optical system 252 and the left light guide plate 28 are collectively referred to as a "left light guide unit", and the right optical system 251 and the right light guide plate 26 are collectively referred to as a "right light guide unit". The configuration of the right light guide unit and the left light guide unit is not limited to the above example, and any method can be used as long as an image light is used to form a virtual image in front of the user U. For example, a diffraction grating can be used. Or a semi-transmissive reflective film may be used.

FIG. 3 is a diagram showing a configuration of a main part of the image display unit 20. FIG. 3 is a perspective view of a main part of the image display unit 20 as viewed from the head side of the user U. Note that the connection cable 40 is not shown in FIG.

FIG. 3 shows the side of the image display unit 20 in contact with the head of the user U, in other words, the side of the user U that can be seen by the right eye RE and the left eye LE. In other words, in FIG. 3, the back sides of the right light guide plate 26 and the left light guide plate 28 are visible.
In FIG. 3, the half mirror 261 that irradiates the image light on the right eye RE of the user U and the half mirror 281 that irradiates the image light on the left eye LE are seen as substantially quadrangular regions. Further, the entire right light guide plate 26 including the half mirror 261 and the entire left light guide plate 28 including the half mirror 281 transmit external light as described above. Therefore, the user U can see the outside view through the entire right light guide plate 26 and the left light guide plate 28, and can see the rectangular display image at the positions of the half mirrors 261 and 281.
Further, the first control unit 120 causes the left display unit 24 and the left light guide plate 28 to display an image, and causes the right display unit 22 and the right light guide plate 26 to display the image according to the instruction of the second control unit 350.

[Structure of each part of HMD]
FIG. 4 is a diagram showing an example of the configuration of each part constituting the HMD 100.
The right display unit 22 of the image display unit 20 has a right display unit substrate 210. The right display unit board 210 has a right I / F unit 211 connected to the connection cable 40, a receiving unit 213 that receives data input from the connecting device 10 via the right I / F unit 211, and an EEPROM (Electrically Erasable). Programmable Read Only Memory) 215 is implemented. The right I / F unit 211 connects the receiving unit 213, EEPROM 215, the temperature sensor 217, the first camera 50, the distance sensor 64, the illuminance sensor 65, the infrared sensor 66, and the LED indicator 67 to the connecting device 10. The receiving unit 213 connects the OLED unit 221 to the connecting device 10.

The left display unit 24 has a left display unit substrate 230. On the left display unit board 230, a receiving unit 233 that receives data input from the connecting device 10 via the left I / F unit 231 and the left I / F unit 231 connected to the connection cable 40 is mounted. Further, a 6-axis sensor 235 and a geomagnetic sensor 237 are mounted on the left display unit substrate 230.
The left I / F unit 231 connects the receiving unit 233, the 6-axis sensor 235, the geomagnetic sensor 237, and the temperature sensor 239 to the connecting device 10. The receiving unit 233 connects the OLED unit 241 to the connecting device 10.

I / F is an abbreviation for interface. In the present embodiment, each of the receiving unit 213 and the receiving unit 233 may be described as Rx213 and Rx233.

The EEPROM 215 non-volatilely stores various types of data. The EEPROM 215 stores, for example, data on the light emitting characteristics and display characteristics of the OLED units 221 and 241 included in the image display unit 20, data on the characteristics of the sensor included in the right display unit 22 or the left display unit 24, and the like.
Specifically, it stores parameters related to gamma correction of the OLED units 221 and 241 and data for compensating for the detected values of the temperature sensors 217 and 239. These data are generated by factory inspection of HMD100 and written to EEPROM 215. The data stored in the EEPROM 215 can be read by the first control unit 120.

The first camera 50 executes imaging according to a signal input via the right I / F unit 211, and outputs the captured image data to the right I / F unit 211.
The illuminance sensor 65 receives external light and outputs a detected value corresponding to the amount of received light or the intensity of received light. The LED indicator 67 lights up according to the control signal or drive current input via the right I / F unit 211.
The temperature sensor 217 detects the temperature of the OLED unit 221 and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value.

The distance sensor 64 executes the distance detection and outputs a signal indicating the detection result to the connecting device 10 via the right I / F unit 211. As the distance sensor 64, for example, an infrared depth sensor, an ultrasonic distance sensor, a Time Of Flight distance sensor, a distance detection unit that combines image detection and voice detection, and the like can be used. Further, the distance sensor 64 may be configured to detect the distance by processing an image obtained by stereo shooting with a stereo camera or a monocular camera.

Further, instead of the distance sensor 64, a plurality of cameras may be mounted on the HMD100 to measure the distance using the principle of triangular measurement, or receive radio waves such as Wi-Fi, Bluetooth, and radio beacons. The receiving unit may be provided in the HMD 100, and the distance to the object may be measured based on the intensity of the radio wave received by the receiving unit.
Further, as the distance sensor 64, for example, a laser radar such as LiDAR may be provided. The laser radar emits a pulsed laser and measures the distance by the reflected light or scattered light from the object.

The infrared sensor 66 executes infrared detection and outputs a signal indicating the detection result to the connecting device 10 via the right I / F unit 211.

The receiving unit 213 receives the image data for display transmitted from the connecting device 10 via the right I / F unit 211 and outputs the image data to the OLED unit 221. The OLED unit 221 displays an image based on the image data transmitted by the connecting device 10.
Further, the receiving unit 233 receives the image data for display transmitted from the connecting device 10 via the left I / F unit 231 and outputs the image data to the OLED unit 241. The OLED units 221 and 241 display an image based on the image data transmitted by the connecting device 10.

The 6-axis sensor 235 is a motion sensor including a 3-axis acceleration sensor and a 3-axis gyro sensor. As the 6-axis sensor 235, an IMU (Inertial Measurement Unit) in which the above sensor is modularized may be adopted. The geomagnetic sensor 237 is, for example, a three-axis geomagnetic sensor. The gyro sensor is also called an angular velocity sensor.

The temperature sensor 239 detects the temperature of the OLED unit 241 and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value.

Each part of the image display unit 20 operates by the electric power supplied from the connection device 10 by the connection cable 40.

The image display unit 20 includes a power supply unit 229 on the right display unit 22 and a power supply unit 249 on the left display unit 24. The power supply unit 229 distributes and supplies the electric power supplied by the connection device 10 via the connection cable 40 to each unit of the right display unit 22 including the right display unit board 210. Similarly, the power supply unit 249 distributes and supplies the electric power supplied by the connection device 10 via the connection cable 40 to each unit of the left display unit 24 including the left display unit board 230. The right display unit 22 and the left display unit 24 may include a conversion circuit or the like that converts voltage.

The connection device 10 includes an I / F unit 110, a first control unit 120, a sensor control unit 122, a first display control unit 124, a power supply control unit 126, a non-volatile storage unit 130, an operation unit 140, a connection unit 145, and voice processing. A unit 147 is provided.

The I / F unit 110 includes a connector 11D. Further, the I / F unit 110 includes an interface circuit that is connected to the connector 11D and executes a communication protocol conforming to various communication standards.

The I / F unit 110 may be, for example, an interface board on which the connector 11D and the interface circuit are mounted. Further, the first control unit 120, the sensor control unit 122, the first display control unit 124, and the power supply control unit 126 of the connection device 10 may be mounted on the connection device main board (not shown). In this case, the connector 11D of the I / F unit 110 and the interface circuit may be mounted on the main board of the connecting device.
Further, the I / F unit 110 may be provided with, for example, an interface circuit for a memory card to which an external storage device or a storage medium can be connected, or the I / F unit 110 may be configured by a wireless communication interface circuit. Good.

The first control unit 120 controls each unit of the connecting device 10. The first control unit 120 has a first processor such as a CPU (Central Processing Unit). In the first control unit 120, the first processor executes the first control program to control each unit of the HMD 100 in cooperation with software and hardware. The non-volatile storage unit 130, the operation unit 140, the connection unit 145, and the voice processing unit 147 are connected to the first control unit 120.

The sensor control unit 122 controls the first camera 50, the distance sensor 64, the illuminance sensor 65, the infrared sensor 66, the temperature sensor 217, the 6-axis sensor 235, the geomagnetic sensor 237, and the temperature sensor 239. Specifically, the sensor control unit 122 sets and initializes the sampling cycle of each sensor according to the control of the first control unit 120, and energizes each sensor and controls data according to the sampling cycle of each sensor. Send, get the detected value, etc.

The sensor control unit 122 is connected to the connector 11D of the I / F unit 110, and outputs data related to the detection value acquired from each sensor to the connector 11D at a preset timing. The smartphone 300 connected to the connector 11D can acquire the detection value of each sensor of the HMD 100 and the captured image data of the first camera 50.

The first display control unit 124 executes various processes for displaying an image based on the image data input to the I / F unit 110 by the image display unit 20. In the present embodiment, the image signal output by the smartphone 300 is input to the connector 11D. The image signal is digital image data, but may be an analog image signal.
The first display control unit 124 executes various processes such as frame cutting, resolution conversion, intermediate frame generation, and frame rate conversion. Resolution conversion involves so-called scaling. The first display control unit 124 outputs image data corresponding to each of the OLED unit 221 and the OLED unit 241 to the connection unit 145. The image data input to the connection unit 145 is transmitted from the connector 11A to the right I / F unit 211 and the left I / F unit 231 as an image signal 201. The image signal 201 is digital image data processed corresponding to each of the OLED unit 221 and the OLED unit 241.

In this embodiment, the connector 11D is composed of a USB-TypeC connector. The first display control unit 124 receives the image data transmitted in the alternative mode of USB-TypeC via the connector 11D.

The sensor control unit 122 and / or the first display control unit 124 may be realized by the cooperation of software and hardware by executing a program by the first processor. That is, the sensor control unit 122 and the first display control unit 124 are configured by the first processor, and execute the above operation by executing the program. In this example, the sensor control unit 122 and the first display control unit 124 may be realized by executing a program by the first processor constituting the first control unit 120. In other words, the first processor may execute the program to function as the first control unit 120, the first display control unit 124, and the sensor control unit 122.

Further, the first display control unit 124 and the sensor control unit 122 may be configured by programmed hardware such as a DSP (Digital Signal Processor) or an FPGA (Field Programmable Gate Array). Further, the sensor control unit 122 and the first display control unit 124 may be integrated to form a SoC (System-on-a-Chip) -FPGA.

The power control unit 126 is connected to the connector 11D. The power control unit 126 supplies power to each unit of the connecting device 10 and the image display unit 20 based on the electric power supplied from the connector 11D. Further, the power supply control unit 126 may include a voltage conversion circuit (not shown), and may be configured to convert the voltage and supply it to each unit of the connection device 10 and the image display unit 20. The power supply control unit 126 may be composed of a logic circuit or a programmed semiconductor device such as an FPGA. Further, the power supply control unit 126 may be configured by the same hardware as the sensor control unit 122 and / or the first display control unit 124.

The sensor control unit 122, the first display control unit 124, and the power supply control unit 126 may include a work memory for performing data processing, or may perform processing using the memory of the first control unit 120. ..

The operation unit 140 detects an operation on the operated unit included in the connecting device 10, and outputs data indicating the operation content or an operation signal indicating the operated operated unit to the first control unit 120.

The voice processing unit 147 generates a voice signal according to the voice data input from the first control unit 120 and outputs the voice signal to the connection unit 145. This audio signal is output from the connection unit 145 to the right earphone 32 and the left earphone 34 via the audio connector 36. Further, the voice processing unit 147 adjusts the volume of the voice signal according to the control of the first control unit 120. Further, the voice processing unit 147 generates voice data of the voice collected by the microphone 63 and outputs the voice data to the first control unit 120. This audio data may be processed by the first control unit 120 in the same manner as the detection value of the sensor included in the image display unit 20.

Further, the connecting device 10 may include a battery (not shown), and the connecting device 10 and the image display unit 20 may be supplied with electric power from the battery. The battery included in the connecting device 10 may be a rechargeable secondary battery.

[Smartphone configuration]
FIG. 5 is a diagram showing an example of the configuration of the smartphone 300.
The smartphone 300 includes a display unit 310, a second camera 321 and a 6-axis sensor 331, a geomagnetic sensor 333, an I / F unit 340, and a second control unit 350.

The display unit 310 includes a display panel 311 and a touch sensor 312, and is connected to the second control unit 350.
The display panel 311 displays various images under the control of the second control unit 350. The display panel 311 is composed of, for example, an LCD (Liquid Crystal Display).
The touch sensor 312 detects the touch operation and outputs data indicating the detected operation to the second control unit 350. The touch sensor 312 is formed integrally with the display panel 311. Specifically, the touch sensor 312 is formed on the image display surface of the display panel 311. The data output by the touch sensor 312 is coordinate data or the like indicating an operation position on the touch sensor 312.

The second camera 321 is a digital camera including an image pickup element such as a CCD or CMOS, an image pickup lens, or the like. The second camera 321 captures the outside view according to the instruction of the second control unit 350, and generates the second captured image.

The 6-axis sensor 331 is a motion sensor including a 3-axis acceleration sensor and a 3-axis gyro sensor. The 6-axis sensor 331 may employ an IMU in which the above sensor is modularized. The geomagnetic sensor 333 is, for example, a three-axis geomagnetic sensor. The gyro sensor is also called an angular velocity sensor.

The I / F section 340 includes a connector 345. Further, the I / F unit 340 includes an interface circuit connected to the connector 345 and executing a communication protocol conforming to various communication standards. The interface circuit is not shown.

The I / F unit 340 may be, for example, an interface board on which a connector 345 and an interface circuit are mounted. Further, the I / F unit 340 may include, for example, an interface circuit for a memory card to which an external storage device or a storage medium can be connected, or the I / F unit 340 may be configured by a wireless communication interface circuit. Good.

The second control unit 350 includes a second storage unit 360 and a second processor 370.
The second storage unit 360 includes a ROM (Read Only Memory) that stores data non-volatilely, and a RAM (Random Access Memory) that constitutes a work area of the second processor 370. Further, the second storage unit 360 may be configured by, for example, a magnetic recording device such as an HDD (Hard Disk Drive) or a storage device using a semiconductor storage element such as a flash memory.

The second storage unit 360 stores the control program 361 and the content table 363.
The control program 361 is a program to be executed by the second processor 370.

FIG. 6 is a diagram showing an example of the configuration of the content table 363.
In the content table 363, the content image data registered in advance, the display order of the content image data, and the identification ID assigned to the content image data are registered in association with each other.
The content image data is image data to be visually recognized by the user U who wears the image display unit 20, and for example, when the HMD100 is used in a museum or an art museum, the content image data is a display visually recognized by the user U together with the exhibit. It is the image data to be performed. For example, when the exhibit is a dinosaur bone, the content image data image is the image data of the dinosaur corresponding to this bone. Further, for example, when the exhibit is a part of the earthenware, the content image data is the image data of the finished product of the earthenware. When the exhibit is a painting or the like, the content image data is image data that reproduces the color of the painting. Further, a plurality of different image data may be registered as the content image data, and the plurality of different image data may be displayed stepwise. For example, if the exhibit is a dinosaur bone, the first stage displays environmental image data, which is an image showing the habitat of the time when the dinosaur lived, and the second stage displays the actual dinosaur image data. You may let me. If the exhibit is earthenware, in the first stage, image data showing the lifestyles of people at the time when the earthenware was used is displayed, and in the second stage, image data of the actual earthenware is displayed. You may.

[Structure of the second control unit of the smartphone]
The second control unit 350 includes a voice processing unit 371 and a display control unit 373. The second control unit 350 functions as a voice processing unit 371 and a display control unit 373 when the second processor 370 included in the second control unit 350 executes the control program 361.

The first voice signal generated by the first microphone 61 and the second voice signal generated by the second microphone 63 are input to the voice processing unit 371. The audio processing unit 371 controls an image to be displayed on the image display unit 20 and a display timing for displaying the image on the image display unit 20 based on the input first audio signal and the second audio signal.

The voice processing unit 371 first determines whether or not the frequencies of the first voice signal and the second voice signal are within a preset range. This is a process for extracting only the sound output from the sound source 400. For example, in a museum, when a dinosaur bark is output as a sound source 400, the voice processing unit 371 determines whether or not a first voice signal and a second voice signal having frequencies corresponding to the dinosaur bark have been input. When the first audio signal and the second audio signal having a frequency within the set range are input, the audio processing unit 371 specifies the direction of the sound source 400 based on the input first audio signal and the second audio signal. To do. Specifically, the audio processing unit 371 specifies the direction of the sound source 400 based on the time difference between the input timing of the first audio signal and the input timing of the second audio signal.

Further, the audio processing unit 371 determines the display timing for displaying the image on the image display unit 20 based on the time difference between the input timing of the first audio signal and the input timing of the second audio signal. For example, in the present embodiment, the audio processing unit 371 determines the timing at which the input timings of the first audio signal and the second audio signal are the same or substantially the same as the timing at which the image display unit 20 displays an image.
Further, the audio processing unit 371 causes the image display unit 20 to display an image when the time difference between the input timing of the first audio signal and the input timing of the second audio signal is within a preset range. May be decided. As a result, the image can be displayed at a position different from that of the sound source 400.

Further, the audio processing unit 371 may determine the timing for displaying the image on the image display unit 20 based on the phase difference between the first audio signal and the second audio signal. When the input timings of the first voice signal and the second voice signal are different, a phase difference occurs between the input first voice signal and the second voice signal. The audio processing unit 371 determines the timing at which the phase difference between the first audio signal and the second audio signal is determined to be zero or substantially zero as the timing for displaying the image on the image display unit 20.

The second display control unit 373 selects the image data to be displayed on the image display unit 20 based on the processing result of the voice processing unit 371, and outputs the selected image data to the connecting device 10 via the I / F unit 340. ..

FIG. 7 shows a case where the distance a1 between the sound source 400 and the first microphone 61 and the distance a2 between the sound source 400 and the second microphone 63 are different. Further, FIG. 8 shows a case where the distance a1 between the sound source 400 and the first microphone 61 and the distance a2 between the sound source 400 and the second microphone 63 are the same or substantially the same.

As shown in FIG. 7, when the distance a1 and the distance a2 are different, the timing at which the first voice signal generated by the first microphone 61 is input to the voice processing unit 371 and the second time generated by the second microphone 63. The timing at which the audio signal is input is different. The voice processing unit 371 obtains a time difference between the input timing of the first voice signal and the input timing of the second voice signal. The audio processing unit 371 specifies the direction of the sound source 400 based on the obtained time difference and information such as which of the first audio signal and the second audio signal is input to the audio processing unit 371 first. ..

FIG. 9 is a diagram showing a display image displayed in the display area VR. The display area VR is an area in which the image display unit 20 displays a display image.
When the direction of the sound source 400 is specified, the sound processing unit 371 outputs image data of an arrow image 450 indicating the direction of the specified sound source 400 and a guidance display 455 prompting the user U to move in the estimated direction of the sound source 400. Output to the connecting device 10. The image display unit 20 causes the display area VR to display the arrow image 450 and the guidance display 455 according to the control of the connecting device 10.

Further, as shown in FIG. 8, when the distance a1 and the distance a2 are the same or substantially the same, the timing at which the first audio signal is input from the first microphone 61 to the audio processing unit 371 and the second microphone 63 The timing at which the second audio signal is input is the same or substantially the same.
The voice processing unit 371 determines that the HMD 100 is located in front of the sound source 400 when the timing at which the first voice signal is input and the timing at which the second voice signal is input are the same or substantially the same. To do. The voice processing unit 371 notifies the display control unit 373 that the HMD 100 is located in front of the sound source 400.

FIG. 10 shows a dinosaur skeleton specimen as an example of the exhibit 500 exhibited in the museum. Further, FIG. 11 shows a content image visually recognized by the user U when the user U moves to the front of the exhibit 500.
For example, when viewing an exhibit along a predetermined route such as an art museum or a museum, the display control unit 373 selects content image data according to a preset display order and connects the selected content image data. Output to device 10. The display order of the content image data is set in the content table 363. Each time the display control unit 373 receives a notification from the audio processing unit 371 that the HMD 100 is located in front of the sound source 400, the display control unit 373 selects the content image data in the set display order and connects the selected content image data. Output to device 10. As a result, as shown in FIG. 11, the content image corresponding to the exhibit is displayed in the display area VR.

Further, the audio processing unit 371 determines the time difference between the input of the first audio signal and the second audio signal by the difference in angle between the image display unit 20 equipped with the first microphone 61 and the second microphone 63 and the sound source 400. It may be converted to θ. The audio processing unit 371 changes the content image data to be displayed on the image display unit 20 based on the converted angle difference θ.

The voice processing unit 371 compares the difference θ of the converted angles with a preset threshold value. In the voice processing unit 371, when the angle difference θ is θ1 or less and larger than θ2, that is, within the first range where θ2 <θ≤θ1, the range including the angle difference θ is the first. Notify the display control unit 373 that the range is one. When the voice processing unit 371 receives a notification that the range including the difference θ is the first range, the display control unit 373 outputs the arrow image 450 and the image data of the guidance display 455 to the connecting device 10. .. As a result, the arrow image 450 shown in FIG. 9 and the image of the guidance display 455 are displayed on the image display unit 20.

Further, the voice processing unit 371 includes an angle difference θ when the converted angle difference θ is θ2 or less and larger than θ3, that is, within the second range where θ3 <θ≤θ2. Notify the display control unit 373 that the range to be displayed is the second range. When the voice processing unit 371 receives a notification from the voice processing unit 371 that the range including the angle difference θ is the second range, the display control unit 373 outputs the environmental image data to the connecting device 10. The angle difference θ2 corresponds to the “first threshold” of the present invention. Further, the environmental image data corresponds to the first image of the present invention.

Further, in the voice processing unit 371, when the converted angle difference θ is θ3 or less, that is, within the third range where θ ≦ θ3, the range including the angle difference θ is the third range. This is notified to the display control unit 373. When the audio processing unit 371 receives a notification that the range including the difference θ is the third range, the display control unit 373 outputs the content image data to the connecting device 10. When the angle difference θ is θ3 or less, it can be determined that the HMD 100 is located in front of the sound source 400. The angle difference θ3 corresponds to the “second threshold” of the present invention. Further, the content image data corresponds to the second image of the present invention.

In the above description, the case where the environmental image data is displayed before displaying the content image data in the display area VR has been described, but as the user U moves to the front of the exhibit, the content image data is displayed in the display area VR. The proportion of content images may be changed.
For example, in the first range described above, the content image is displayed in the display area VR at 30%, in the second range, the content image is displayed in the display area VR at 50%, and in the third range, the content is displayed in the display area VR. The image may be displayed at 100%.

Further, the second control unit 350 may determine an image to be displayed on the image display unit 20 based on the input first audio signal and the second audio signal. The second control unit 350 converts the input time difference between the first voice signal and the second voice signal into the angle difference θ between the image display unit 20 and the sound source 400 by the voice processing unit 371. The second control unit 350 determines an image to be displayed on the image display unit 20 based on the angle difference θ converted by the voice processing unit 371. When the angle difference θ is in the first range, the arrow image 450 is determined as an image to be displayed on the image display unit 20, and when the angle difference θ is in the second range, the environment image data is displayed as an image to be displayed on the image display unit 20. When the angle difference θ is in the third range, the image based on the content image data is determined as the image to be displayed on the image display unit 20.

Further, the second control unit 350 controls at least one of the display position of the image to be displayed on the image display unit 20 and the size of the image based on the input time difference between the first audio signal and the second audio signal. May be good. The second control unit 350 converts the input time difference between the first voice signal and the second voice signal into the angle difference θ between the image display unit 20 and the sound source 400 by the voice processing unit 371. The second control unit 350 changes the display position of the image to be displayed on the image display unit 20 and the size of the image based on the difference θ of the angles converted by the voice processing unit 371.
For example, when the angle difference θ is in the first range, the second control unit 350 displays the arrow image 450 in the first size at the first position which is the end of the display area VR. Further, when the angle difference θ is in the second range, the second control unit 350 causes the image based on the environmental image data to be displayed in the second size at the second position of the display area VR. The second position is located at the center of the display area VR rather than the first position. Further, the size of the arrow image 450 displayed in the first size is larger than the size of the image based on the environmental image data displayed in the second size. Further, when the angle difference θ is in the third range, the second control unit 350 causes the image based on the environmental image data to be displayed in the third size at the third position of the display area VR. The third position is located at the center of the display area VR rather than the second position. Further, the size of the image based on the environmental image data displayed in the second size is larger than the size of the image based on the content image data displayed in the third size. On the contrary, the size of the image to be displayed in the display area VR may be increased in the order of the first position, the second position, and the third position.

[Explanation of operation of the second control unit]
FIG. 12 is a flowchart showing the operation of the second control unit 350.
The operation of this embodiment will be described with reference to the flowchart shown in FIG.
First, the second control unit 350 determines whether or not an audio signal having a frequency within a preset setting range has been input from the first microphone 61 and the second microphone 63 (step S1). When there is no input of an audio signal having a frequency within the set range (step S1 / NO), the second control unit 350 waits for the start of processing until the audio signal is input.

When an audio signal having a frequency within the set range is input (step S1 / YES), the second control unit 350 has the first audio signal input from the first microphone 61 and the second input from the second microphone 63. 2 Obtain the input time difference from the audio signal. The second control unit 350 converts the input time between the first audio signal and the second audio signal into the angle difference θ between the image display unit 20 and the sound source 400 (step S2).

Next, the second control unit 350 determines whether or not the difference θ of the converted angles is within the first range (step S3). When the angle difference θ is within the first range (step S3 / YES), the second control unit 350 includes an arrow image 450 indicating the direction of the sound source 400 and a guidance display 455 prompting the movement in the direction of the sound source 400. The image data is output to the connecting device 10. The image display unit 20 causes the display area VR to display the arrow image 450 and the guidance display 455 according to the control of the connecting device 10 (step S4).

Further, when the angle difference θ is not within the first range (step S3 / NO), the second control unit 350 determines whether or not the angle difference θ is within the second range (step S5). When the angle difference θ is within the second range (step S5 / YES), the second control unit 350 outputs the environment image data in which the dinosaur habitat is displayed to the connecting device 10. The image display unit 20 causes the display area VR to display an image in which the dinosaur habitat is displayed according to the control of the connection device 10 (step S6).

Further, when the angle difference θ is not within the second range (step S5 / NO), the second control unit 350 determines whether or not the angle difference θ is within the third range (step S7). When the angle difference θ is within the third range (step S7 / YES), the second control unit 350 outputs the content image data, which is an image of the dinosaur, to the connecting device 10. The image display unit 20 causes the display area VR to display an image of a dinosaur, which is a content image, according to the control of the connection device 10 (step S8). Further, when the angle difference θ is not within the third range (step S7 / NO), the second control unit 350 returns to step S1 and determines whether or not the first voice signal and the second voice signal have been input. To do.

As described above, the display system 1 of the present embodiment includes an HMD 100 and a smartphone 300. The HMD 100 includes an image display unit 20, a first microphone 61 into which the sound emitted by the sound source is input, a second microphone 63 in which the sound emitted by the sound source 400 is input, and a second microphone 63 arranged at a position different from the first microphone 61. To be equipped.
The smartphone 300 includes a second control unit 350 that controls an image to be displayed on the image display unit 20 based on the first audio signal and the second audio signal input from the first microphone 61 and the second microphone 63.
Therefore, the image to be displayed on the image display unit 20 can be changed based on the sound emitted by the sound source.

Further, the second control unit 350 has an image to be displayed on the image display unit 20 and an image display timing based on the time difference between the timing at which the first audio signal is input and the timing at which the second audio signal is input. And control.
For example, when the input timings of the first audio signal and the second audio signal are the same, the image display unit 20 displays an image. As a result, when the sound source is equidistant from the first microphone 61 and the second microphone 63, the image display unit 20 can display an image. Therefore, the image can be displayed at an appropriate timing. Further, when the input timings of the first audio signal and the second audio signal are different, the image display unit 20 displays an image. As a result, when the sound source is located at a different position from the first microphone 61 and the second microphone 63, the image display unit 20 can display the image. For example, by displaying the arrow image 450 indicating the direction of the sound source, the direction of the sound source can be notified to the user U.

Further, the second control unit 350 is an image display unit when the time difference between the first audio signal output by the first microphone 61 and the second audio signal output by the second microphone 63 is equal to or less than the first threshold value. 20 is made to display the first image. Also. The second control unit 350 causes the image display unit 20 to display the second image when the time difference between the first voice communication and the second voice signal is equal to or less than the second threshold value smaller than the first threshold value. Therefore, the image to be displayed on the image display unit 20 can be changed based on the difference in the distance between the sound source and the first microphone 61 and the second microphone 63.

Further, the second control unit 350 has at least one of the display positions and sizes of the images to be displayed on the image display unit 20 based on the time difference between the input timing of the first audio signal and the input timing of the second audio signal. To control.
Therefore, the display position and size of the image to be displayed on the image display unit 20 can be changed based on the difference in the distance between the sound source and the first microphone 61 and the second microphone 63.

The second control unit 350 determines the timing for displaying an image on the image display unit 20 based on the phase difference of the audio signals output from the first microphone 61 and the second microphone 63.
Therefore, the timing for displaying the image on the image display unit 20 can be determined based on the difference in the distance between the sound source and the first microphone 61 and between the sound source and the second microphone 63. That is, the timing for displaying the image on the image display unit 20 can be determined based on the positional relationship between the sound source and the image display unit 20.

In the present embodiment, the second control unit 350 of the smartphone 300 includes a voice processing unit 371 and a second display unit 373, but the embodiment of the present invention is not limited thereto. For example, the first control unit 120 of the connection device 10 of the HMD 100 may include a voice processing unit 371 and a second display unit 373. Further, the second control unit 350 of the smartphone 300 includes a part of the voice processing unit 371 and the second display unit 373, and the first control unit 120 of the connection device 10 of the HMD 100 includes the voice processing unit 371 and the second display unit 373. 2 The remaining portion with the display unit 373 may be provided.

[Modification example]
As a modification of the above-described embodiment, the sound output by the sound source 400 may include an identification ID that identifies the content image. Multiple exhibits are exhibited in museums and galleries. Therefore, by including the information of the identification ID of the content image in the sound output by the sound source 400, the content image unrelated to the exhibit actually viewed by the user is displayed by the image display unit 20. It is possible to reduce the probability of

A voice watermark can be used as a method of superimposing the identification ID information on the voice output by the sound source 400.
The voice processing unit 371 analyzes the input voice signal and determines whether or not the watermark information is superimposed. When the watermark information is superimposed on the voice signal, the voice processing unit 371 extracts the watermark information. When the extracted watermark information is the identification ID of the content image to be displayed, the voice processing unit 371 outputs the extracted identification ID to the display control unit 373.

When a notification that the HMD 100 is located in front of the sound source 400 is input from the audio processing unit 371, the display control unit 373 connects the content image data corresponding to the identification ID notified from the audio processing unit 371. Output to 10. The image display unit 20 causes the display area VR to display the content image based on the content image data according to the control of the connecting device 10.

In this modification, the smartphone 300 includes a second storage unit 360 that stores the image data and the identification information of the image data in association with each other.
The smartphone 300 extracts watermark information superimposed on an audio signal output from a sound source, acquires identification information from the extracted watermark information, and uses image data associated with the acquired identification information as a determined image. Display on the display unit 20.
Therefore, the image to be displayed on the image display unit 20 can be specified by the watermark information.

[Second Embodiment]
A second embodiment of the present invention will be described with reference to the accompanying drawings.
FIG. 13 is a diagram showing a configuration example of the second embodiment of the HMD 100.
In the second embodiment, the HMD 100 includes a beacon receiving unit 226 for receiving a beacon signal on the right display unit 22 of the image display unit 20.
Further, in a facility such as a factory, a plurality of transmitters that periodically output a beacon signal are provided. As the beacon signal output by the transmitter, for example, a signal corresponding to a communication standard of a wireless PAN (Personal Area Network) such as BLE (Bluetooth Low Energy) or ZigBee is used. ZigBee is a registered trademark.

The information contained in the beacon signal transmitted by the transmitter is arbitrary, but preferably includes information that can identify or identify the transmitter. The information included in the beacon signal includes a beacon ID (IDentification) that identifies each of a plurality of installed transmitters.

FIG. 14 is a diagram showing a configuration example of the smartphone 300. Further, FIG. 15 is a diagram showing an example of the parts information table 365.
The second storage unit 360 of the smartphone 300 stores the part information table 365 in addition to the control program 361. In the component information table 365, information regarding components that generate a driving sound during operation is registered. Specifically, in the component information table 365, the frequency range of the sound generated when the component is operating normally and the beacon ID that identifies the installation location of the component correspond to the identification ID that identifies the component. It is registered with. The frequency range corresponds to the "range information" of the present invention.

The second control unit 350 includes a position specifying unit 375 in addition to the voice processing unit 371 and the second display control unit 373.
The position specifying unit 375 performs positioning processing based on the beacon signal received by the beacon receiving unit 226 of the image display unit 20, and identifies the position of the HMD 100 in the hall. For the positioning process performed by the position identification unit 375, the current position is set to the position associated with the beacon ID of the nearest transmitter, and the distance from the transmitter whose position is obvious is estimated from the radio field strength, and three points are used. A method of estimating the position of the HMD 100 based on the principle of surveying can be mentioned.

The voice processing unit 371 acquires information on the frequency range of the sound associated with the beacon signal received by the beacon receiving unit 226 of the image display unit 20. The voice processing unit 371 determines whether or not the frequencies of the first and second voice signals input by the first microphone 61 and the second microphone 63 deviate from the range indicated by the acquired frequency range information. When the frequencies of the first and second voice signals are out of the range indicated by the frequency range information, the voice processing unit 371 outputs the corresponding identification ID to the display control unit 373.

FIG. 16 is a diagram showing an example of a display image displayed in the display area VR.
The display control unit 373 outputs the component name corresponding to the identification ID input from the voice processing unit 371 and the information indicating that the component corresponding to the component name has a failure to the connecting device 10. In the connection device 10, the image display unit 20 causes the display area VR to display a component name and a display indicating that a failure may have occurred in this component under the control of the connection device 10.

As described above, in the second embodiment, the smartphone 300 includes a second storage unit 360 that stores range information indicating the frequency range of the sound output from the component when the component is operating normally. In the smartphone 300, when the frequencies of the audio signals input from the first microphone 61 and the second microphone 63 are not within the frequency range indicated by the range information, a failure occurs in a component arranged in the direction of the specified sound source. Display a display indicating that the user is present.
Therefore, it is possible to perform a failure diagnosis of parts.

The present invention is not limited to the configuration of the above embodiment, and can be implemented in various embodiments without departing from the gist thereof.
For example, the above-described embodiment has been described by taking the display system 1 including the HDM 100 and the smartphone 300 as an example, but the first control unit 120 included in the connection device 10 functions as the second control unit 350 of the smartphone 300. Part or all may be performed. That is, the connection device 10 may control the display of the image display unit 20 instead of the smartphone 300.

Further, in the above embodiment, the "control device" is the smartphone 300, but the present invention is not limited to this. The "control device" may be configured to be portable to the user U and may include a display unit, an operation unit, and a control unit. For example, the "control device" may be a PDA terminal or a tablet-type personal computer.

Further, in the above embodiment, the configuration in which the connecting device 10 is connected to the image display unit 20 by wire is illustrated, but the present invention is not limited to this, and the image display unit 20 is wirelessly connected to the connecting device 10. You may.
Further, in the above embodiment, the configuration in which the connecting device 10 is connected to the smartphone 300 by wire is illustrated, but the present invention is not limited to this, and the smartphone 300 may be wirelessly connected to the connecting device 10.

Further, some functions included in the connecting device 10 may be provided in the image display unit 20, and the connecting device 10 may be realized by a plurality of devices. For example, instead of the connecting device 10, a wearable device that can be attached to the body, clothes, or accessories worn by the user U may be used. The wearable device in this case may be, for example, a watch-type device, a ring-type device, a laser pointer, a mouse, an air mouse, a game controller, a pen-type device, or the like.

Further, in the above embodiment, the configuration in which the image display unit 20 and the connecting device 10 are separated and connected via the connecting cable 40 will be described as an example. Not limited to this, the connecting device 10 and the image display unit 20 may be integrally configured and mounted on the head of the user U.

Further, in the above embodiment, the configuration in which the user U transmits the display unit to visually recognize the outside view is not limited to the configuration in which the right light guide plate 26 and the left light guide plate 28 transmit external light. For example, it can be applied to an image display device that displays an image in a state where the outside view cannot be visually recognized. Specifically, it is used as an image display device that displays an image captured by the first camera 50, an image or CG generated based on the captured image, an image data stored in advance, an image based on image data input from the outside, or the like. , Applicable. For example, if the image display unit 20 displays a composite image obtained by combining the external view image captured by the first camera 50 and the display image, the image display unit 20 can be used even if it does not transmit external light. The outside view and the image can be visually displayed on the person U. Of course, it can also be applied to such a so-called video see-through type image display device.

Further, for example, instead of the image display unit 20, an image display unit of another type such as an image display unit worn like a hat may be adopted, and an image corresponding to the left eye LE of the user U may be adopted. It suffices to have a display unit for displaying the image and a display unit for displaying an image corresponding to the right eye RE of the user U. Further, the image display device may be configured as an HMD mounted on a vehicle such as an automobile or an airplane, for example. Further, for example, it may be configured as an HMD built in a body protective device such as a helmet. In this case, a portion that positions the position of the user U with respect to the body and a portion that is positioned with respect to the portion can be used as the mounting portion.

Further, as an optical system for guiding the image light to the eyes of the user U, a configuration in which a virtual image is formed by the half mirrors 261 and 281 on a part of the right light guide plate 26 and the left light guide plate 28 is illustrated. The present invention is not limited to this, and the image may be displayed in a display area having an area that occupies the entire surface or most of the right light guide plate 26 and the left light guide plate 28. In this case, the operation of changing the display position of the image may include a process of reducing the image.
Further, the optical element is not limited to the right light guide plate 26 and the left light guide plate 28 having the half mirrors 261 and 281, and may be any optical component that allows the image light to enter the eye of the user U. A diffraction grating, a prism, or a holographic display unit may be used.

Further, at least a part of each functional block shown in FIGS. 4 and 5 may be realized by hardware, or may be a configuration realized by cooperation between hardware and software, as shown in the figure. It is not limited to the configuration in which independent hardware resources are arranged as described above.
Further, the second control program executed by the second control unit 350 may be stored in the non-volatile storage unit or another storage unit in the second control unit 350. Further, the second control program stored in the external device may be acquired and executed via the communication unit or the like.
Further, the configurations formed in the connecting device 10 may be duplicated and formed in the image display unit 20. For example, a processor similar to the first processor of the connecting device 10 may be arranged in the image display unit 20, or a function in which the first processor included in the connecting device 10 and the processor of the image display unit 20 are separately separated may be provided. It may be a configuration to be executed.

Further, the processing unit of the flowchart shown in FIG. 12 is divided according to the main processing contents in order to make the processing of the second control unit 350 of the smartphone 300 easy to understand. The embodiment is not limited by the method of dividing the processing unit and the name shown in the flowchart shown in FIG. Further, the processing of the second control unit 350 can be divided into more processing units according to the processing content, or can be divided so that one processing unit includes more processing. Further, the processing order of the above flowchart is not limited to the illustrated example.

Further, the image display method of the present invention can be realized by causing the second processor 370 included in the smartphone 300 to execute the control program 361 corresponding to the control method of the smartphone 300. Further, the control program 361 can also be recorded on a recording medium readable by a computer. As the recording medium, a magnetic or optical recording medium or a semiconductor memory device can be used. Specifically, it is a portable type such as a flexible disc, a CD-ROM (Compact Disk Read Only Memory), a DVD (Digital Versailles Disc), a Blu-ray (registered trademark) Disc, a magneto-optical disc, a flash memory, and a card-type recording medium. Alternatively, a fixed recording medium can be mentioned. Further, the control method of the smartphone 300 can be realized by storing the control program 361 corresponding to the control method of the smartphone 300 in the server device or the like and downloading the control program 361 from the server device to the smartphone 300.

1 ... Display system, 10 ... Connection device, 11 ... Connector, 11A ... Connector, 11D ... Connector, 13 ... Brightness adjustment key, 14 ... Brightness adjustment key, 15 ... Volume adjustment key, 16 ... Volume adjustment key, 20 ... Image display Unit, 21 ... right holding part, 22 ... right display part, 23 ... left holding part, 24 ... left display part, 26 ... right light guide plate, 27 ... front frame, 28 ... left light guide plate, 30 ... headset, 32 ... right earphone, 34 ... left earphone, 36 ... audio connector, 40 ... connection cable, 46 ... USB cable, 50 ... first camera, 61 ... first microphone, 63 ... second microphone, 64 ... distance sensor, 65 ... illuminance Sensor, 66 ... Infrared sensor, 67 ... LED indicator, 100 ... HMD, 110 ... I / F unit, 120 ... 1st control unit, 122 ... Sensor control unit, 124 ... 1st display control unit, 126 ... Power supply control unit, 130 ... Non-volatile storage unit, 140 ... Operation unit, 145 ... Connection unit, 147 ... Audio processing unit, 201 ... Image signal, 210 ... Right display unit board, 211 ... Right I / F unit, 213 ... Receiver unit, 215 ... EEPROM, 217 ... Temperature sensor, 221 ... OLED unit, 223 ... OLED panel, 225 ... OLED drive circuit, 226 ... Beacon receiver, 229 ... Power supply, 230 ... Left display board, 231 ... Left I / F, 233 ... Receiver, 235 ... 6-axis sensor, 237 ... Geomagnetic sensor, 239 ... Temperature sensor, 241 ... OLED unit, 243 ... OLED panel, 245 ... OLED drive circuit, 249 ... Power supply unit, 251 ... Right optical system, 252 ... Left Optical system, 261 ... half mirror, 281 ... half mirror, 300 ... smartphone, 311 ... display panel, 312 ... touch sensor, 321 ... second camera, 331 ... 6-axis sensor, 332 ... geomagnetic sensor, 332 ... touch sensor, 333 ... Geomagnetic sensor, 340 ... I / F unit, 345 ... Connector, 350 ... Second control unit, 360 ... Second storage unit, 361 ... Program, 363 ... Content table, 365 ... Parts information table, 370 ... Second processor , 371 ... Voice processing unit, 373 ... Second display control unit, 375 ... Position identification unit, 400 ... Sound source, 450 ... Arrow image, 455 ... Guidance display, 500 ... Exhibit.

Claims (10)

Image display and
The first audio input unit that inputs the audio emitted from the sound source and outputs the first audio signal,
An image display device including a second audio input unit that is arranged at a position different from the first audio input unit and that inputs the audio and outputs a second audio signal.
A control device that controls an image to be displayed on the image display unit based on the first audio signal and the second audio signal.
Display system with. The control device has an image to be displayed on the image display unit and a display timing of the image based on a time difference between the timing at which the first audio signal is input and the timing at which the second audio signal is input. The display system according to claim 1. When the time difference between the first audio signal and the second audio signal is equal to or less than the first threshold value, the control device causes the image display unit to display the first image, and the first audio signal and the second audio signal. The display system according to claim 2, wherein when the time difference from the second audio signal is equal to or less than the second threshold value smaller than the first threshold value, the second image is displayed on the image display unit. The control device controls at least one of the display position and the size of the image to be displayed on the image display unit based on the time difference between the input timing of the first audio signal and the input timing of the second audio signal. The display system according to any one of claims 1 to 3. The control device includes a storage unit that stores image data in association with identification information of the image data.
The control device extracts watermark information superimposed on an audio signal output from the sound source, acquires the identification information from the extracted watermark information, and is based on image data associated with the acquired identification information. The display system according to claim 1, wherein an image is displayed on the image display unit. The display system according to claim 1, wherein the control device determines a timing for displaying an image on the image display unit based on a phase difference between the first audio signal and the second audio signal. The control device includes a storage unit that stores range information indicating a frequency range of audio output from the component when the component is operating normally.
When the frequencies of the first audio signal and the second audio signal input from the first audio input unit and the second audio input unit are not within the frequency range indicated by the range information, the specified sound source The display system according to claim 1, wherein a display indicating that a failure has occurred in a component arranged in a direction is displayed. Image display and
The first audio input unit that inputs audio and outputs the first audio signal,
An image display device including a second audio input unit that is arranged at a position different from the first audio input unit and that inputs audio and outputs a second audio signal.
A control unit that controls an image to be displayed on the image display unit based on the first audio signal and the second audio signal.
An image display device comprising. The step of inputting the first audio signal output by the first audio input unit that input the audio of the sound source, and
A step in which a second voice input unit arranged at a position different from the first voice input unit inputs a second voice signal for inputting and outputting the voice,
A step of controlling an image to be displayed on the image display unit based on the input first voice signal and the second voice signal, and
Image display method having. On the computer
The procedure for inputting the first audio signal output by the first audio input unit that input the audio of the sound source, and
A procedure in which a second voice input unit arranged at a position different from the first voice input unit inputs a second voice signal for inputting and outputting the voice, and
A procedure for controlling an image to be displayed on the image display unit based on the input first voice signal and the second voice signal, and
A program that executes.

Images