JP2019114236A - Display system, electronic apparatus, and display method - Google Patents

Abstract

To control display performed by a display in accordance with an external device.SOLUTION: A display system 1 comprises: a PC 300; and an HMD 100. The PC includes an I/F unit 341 that outputs video data. The HMD includes an I/F unit 110 that acquires the video data output from the PC, an image display unit 20 that displays an image superimposed on an outdoor scene visually recognized while the HMD is attached, and a display control unit 122 that controls the image display unit to display the video data acquired by the I/F unit. The display control unit displays the video data on the image display unit in accordance with the position of the PC visually recognized as the outdoor scene.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a display system, an electronic device, and a display method.

  2. Description of the Related Art Conventionally, in a display device, one that receives and displays a display image of an external device is known (for example, see Patent Document 1). In Patent Document 1, an example in which a head mounted display (HMD) receives a display image transmitted by an external device and displays it on a display unit in order to integrate display screens of a plurality of devices. Is described.

JP, 2015-227919, A

The configuration disclosed in Patent Document 1 makes it possible to integrate the display images of the external device in the display of the HMD, on the premise that the entire display is performed by the HMD. On the other hand, there has been no conventionally proposed example of a method of controlling the display of the display device according to an external device different from the display device.
The present invention has been made in view of the above circumstances, and it is an object of the present invention to control display on a display device in accordance with an external device.

In order to solve the above-mentioned subject, a display system of the present invention is provided with electronic equipment which has the 1st indicator, and a head mount type display connected to the electronic equipment, and the electronic equipment carries out the picture The display device includes an output unit for outputting, and a second display for displaying an image superimposed on an acquisition unit for acquiring the image output from the electronic device, and an outside scene viewed in a state where the display device is mounted. And a display control unit for causing the second display unit to display the image acquired by the acquisition unit, the display control unit corresponding to the position of the electronic device visually recognized as an outside view An image is displayed on the second display unit.
According to the present invention, when an image output by the electronic device is displayed by the head-mounted display device, the image is displayed to correspond to the position of the electronic device as the outside view. For example, the image can be displayed by the display device in accordance with the position of the first display unit that can be seen as an outside scene. As described above, the display device which displays an image superimposed on the outside scene can perform display in accordance with the position of the electronic device visually recognized as the outside scene.

Further, according to the present invention, the electronic device generates an image corresponding to a virtual display area wider than the first display unit, displays a part of the generated image on the first display unit, and generates the image. The display control unit includes an electronic device control unit that causes at least a portion of the image to be output by the output unit, and the display control unit included in the display device sets at least a portion of the image output by the electronic device At the same time, it is displayed on the second display unit.
According to this configuration, it is possible to perform display corresponding to a virtual display area wider than the first display unit of the electronic device on the second display unit of the display device. Therefore, a display area larger than the first display unit can be displayed in alignment with the position of the electronic device, and the display unit can virtually enlarge the first display unit.

Further, according to the present invention, the electronic device control unit causes the output unit to output the image excluding the portion displayed on the first display unit among the images generated corresponding to the virtual display area. The display control unit included in the display device displays an image output by the electronic device on the second display unit.
According to this configuration, when the display device performs display corresponding to a virtual display area wider than the first display unit of the electronic device, a part of the image is displayed on the first display unit, and the second display unit Is a portion excluding the portion displayed on the first display unit. Therefore, it is possible to realize display combining the display of the electronic device and the display of the display device. For example, an image can be displayed by the second display unit around the electronic device by the display device, and a display mode in which the first display unit is virtually expanded can be realized.

Further, in the present invention, the display control unit included in the display device displays the image output by the electronic device around the first display unit visually recognized as an outside scene.
According to this configuration, by displaying an image around the first display unit of the electronic device by the second display unit of the display device, it is possible to realize a display mode in which the first display unit is virtually expanded.

Further, according to the present invention, the electronic device control unit may control the first part of the image generated in correspondence with the virtual display area based on a relative position of the first display unit to the second display unit. The image displayed on the display unit is output by the output unit except the portion displayed on the first display unit.
According to this configuration, the electronic device outputs the image displayed by the display device so as to correspond to the relative position of the first display unit with respect to the second display unit. Therefore, the operation of performing display corresponding to the position of the first display portion of the electronic device by the display device can be easily realized.

Further, according to the present invention, the electronic device control unit controls the first display unit among the images generated corresponding to the virtual display area based on the relative position of the first display unit with respect to the second display unit. The image output unit is configured to output an image obtained by masking the displayed portion.
According to this configuration, since the portion displayed on the first display unit of the electronic device is not displayed on the second display unit, the image displayed by the electronic device can be coordinated with the image displayed by the display device. .

Further, in the present invention, the electronic device control unit causes the output unit to output the image generated in association with the virtual display area, and the display control unit included in the display device is output by the electronic device. An image obtained by cutting out a part of the image is displayed on the second display unit.
According to this configuration, by the processing of the display device, a part of the image displayed by the electronic device can be enlarged and displayed on the second display unit.

Further, according to the present invention, the display control unit extracts a part of the image acquired by the acquiring unit based on a relative position of the first display unit with respect to the second display unit and performs the second display. Display on the department.
According to this configuration, the display device determines the relative position of the first display unit with respect to the display unit, and generates an image for display corresponding to the relative position, so that the load on the electronic device is not increased. The display corresponding to the position of the first display unit can be realized.

Further, according to the present invention, the display control unit determines a portion overlapping the first display unit in the image acquired by the acquisition unit based on the relative position of the first display unit with respect to the second display unit. The masked image is displayed on the second display unit.
According to this configuration, in the image displayed on the second display unit by the processing of the display device, the portion displayed on the first display unit is masked, so the electronic device can be operated without increasing the load on the electronic device. It is possible to coordinate the image displayed by the image with the image displayed by the display device.

Further, according to the present invention, the electronic device control unit determines the position of the virtual display area based on the position of the electronic device in the real space, and based on the relative position between the virtual display area and the second display unit. Adjusting a display mode of the second display unit.
According to this configuration, the display device can display the image in accordance with the position of the electronic device in the real space. With this image, it is possible to obtain an effect of complementing or expanding the image displayed by the first display unit of the electronic device.

Further, in the present invention, when the electronic device control unit detects that the first display unit is included in a range viewed through the second display unit, the position of the first display unit is determined. As a reference, the display mode of the image is initialized.
According to this configuration, it is possible to adjust the display mode in accordance with whether or not the first display unit of the electronic device can be viewed as an outside scene in the display device.

Further, according to the present invention, the second display unit is a display unit for the left eye that emits image light toward the left eye of the user wearing the display device, and an image light directed to the right eye of the user And a display position by the left-eye display unit and a display position for the right-eye display, corresponding to the position of the first display unit viewed as an outside scene by the second display unit. The display position is controlled, and the convergence angle of the image displayed on the second display unit is adjusted.
According to this configuration, by adjusting the convergence angle of the image displayed by the display device, the distance in which the display image of the display device is viewed can be made to correspond to the position of the first display portion of the electronic device. Thus, the display by the display device and the display of the electronic device can be coordinated more appropriately.

Further, according to the present invention, the second display unit includes an optical system capable of adjusting a visible distance at which the user perceives an image displayed by the second display unit, and the second display unit is displayed on the second display unit. The optical system is controlled according to the convergence angle of the image.
According to this configuration, since the distance in which the display image of the display device is viewed corresponds to the position of the first display unit of the electronic device, the display by the display device and the display of the electronic device can be coordinated more appropriately. Can.

Further, in order to solve the above problems, an electronic device according to the present invention is an electronic device connected to a head-mounted display device that displays an image superimposed on an outside scene, comprising: a first display unit; Corresponding to the position of the first display portion that is viewed as an outside scene by the display device based on the relative position of the first display portion to the second display portion of the display device and an output unit that outputs an image to the display device. And a control unit that causes the output unit to output an image for displaying the image on the second display unit.
According to this configuration, when the electronic device connected to the display device outputs an image to the display device, the display device performs display such that it corresponds to the position of the first display portion of the electronic device as the outside scene. be able to.

Further, in order to solve the above problems, the display method of the present invention includes an electronic device having a first display portion and a head-mounted display device having a second display portion that displays an image superimposed on an outside scene. A display method, comprising: outputting an image by the electronic device; acquiring the image output by the electronic device by the display device; displaying the acquired image by the second display unit; The image is displayed on the second display unit in correspondence with the position of the electronic device that is visually recognized as the outside scene in the image.
According to the present invention, when the image output by the electronic device is displayed by the display device, the image is displayed to correspond to the position of the electronic device as the outside scene. For example, since the image can be displayed by the display device in accordance with the position of the electronic device viewed as the outside scene, the display device can perform display in accordance with the display of the electronic device.

  The present invention can also be realized in various forms other than the above-described display system, electronic device, and display method. For example, a program for executing the above display method by a computer, a recording medium recording the above program, a server device for distributing the above program, a transmission medium for transmitting the above program, a data signal embodying the above program in a carrier wave It can be realized in the form of

FIG. 2 is an external view of an HMD and a PC that constitute a display system. The figure which shows the structure of the optical system of HMD. The principal part perspective view which looked at the image display part from the user's head side. Explanatory drawing which shows a response | compatibility with the display part and imaging range of HMD. FIG. 2 is a block diagram of each part of the display system. Explanatory drawing which shows the process which calculates | requires the relative position of PC with respect to an image display part. 6 is a flowchart showing the operation of the display system. The figure which shows the example of the display mode of a display system. The figure which shows the example of the display mode of a display system. The figure which shows the example of the display mode of a display system. 6 is a flowchart showing the operation of the display system. The figure which shows the example of the display mode of the display system in 2nd Embodiment. The flowchart which shows operation of the display system in a 3rd embodiment. The flowchart which shows operation of the display system in a 3rd embodiment. The block diagram of each part which comprises the display system in 4th Embodiment.

[1. First embodiment]
[1-1. Display system configuration]
FIG. 1 is a view showing a configuration of a display system 1 according to a first embodiment to which the present invention is applied.
The display system 1 includes an HMD (Head Mounted Display: head mounted display) 100 and a PC (Personal Computer) 300 as an external device of the HMD 100.

  The HMD 100 includes an image display unit 20 (second display unit) that allows the user to visually recognize a virtual image in a state of being attached to the head of a user (user), and the connection device 10 that controls the image display unit 20. It is a display device. The connection device 10 includes a plurality of connectors 11 in a box-shaped case (also referred to as a housing or a main body). The image display unit 20 and the connection device 10 are connected by a connection cable 40.

  In the example of FIG. 1, the connection device 10 comprises three connectors 11A, 11B, 11C. In the following description, when the connectors 11A, 11B, and 11C are not distinguished, they are collectively referred to as the connector 11. The connector 11 is a wired interface for connecting a communication cable, and the connection device 10 is connected to an external device by the communication cable. The connectors 11A, 11B, 11C are, for example, connectors conforming to known communication interface standards, and may be connectors of the same shape or may be different types of connectors. In the present embodiment, as an example, the connector 11A is a connector conforming to the HDMI (High Definition Multimedia Interface) standard. Further, the connector 11B is a USB (Universal Serial Bus) -Type C connector. Further, the connector 11C is a Micro USB connector.

  In the example of FIG. 1, the connection device 10 and the PC 300 are connected by the cable 2. The cable 2 includes an HDMI cable 2A connecting the PC 300 and the connector 11A, and a USB cable 2B connecting the PC 300 and the connector 11B. In this example, the PC 300 is connected to the connection device 10 by an HDMI interface for video transmission and a USB interface for data communication.

The PC 300 is a computer including a display unit 330 that displays an image, and corresponds to the electronic device of the present invention. The PC 300 is preferably a portable computer, and includes a tablet computer, a notebook computer, a smartphone, and the like. The PC 300 of FIG. 1 includes a display unit 330 as a first display unit on the surface of a flat plate-like main body. The display unit 330 includes a display panel 331 (FIG. 5) such as a liquid crystal display panel or an organic EL (Electro Luminescent) display panel, and detects the touch operation of the user on the surface of the display panel 331 (FIG. 5) Is provided.
The PC 300 functions as an external device for the HMD 100. The external device may be an electronic device having a display screen and having a function of displaying an image on the display screen, and in the present embodiment, the PC 300 is shown as an example only.

  The image display unit 20 is a mounted body mounted on the head of the user, and is a so-called head mounted display (HMD). That is, the HMD 100 has a configuration in which the connection device 10 for connecting an external device such as the PC 300 is connected to the image display unit 20 as the HMD main body. The image display unit 20 has an eyeglass shape in the present embodiment. The image display unit 20 includes a right display unit 22 (right-eye display unit), a left display unit 24 (left-eye display unit), and a main body including a right holding unit 21, a left holding unit 23, and a front frame 27. , The right light guide plate 26 and the left light guide plate 28.

  The right holding unit 21 and the left holding unit 23 respectively extend rearward from the both ends of the front frame 27 and hold the image display unit 20 on the head of the user like temples of glasses. Here, among the both ends of the front frame 27, the end located on the right side of the user in the mounted state of the image display unit 20 is the end ER, and the end located on the left of the user is the end EL Do. The right holding portion 21 extends from the end portion ER of the front frame 27 to a position corresponding to the user's right head in the mounted state of the image display portion 20. The left holding portion 23 is extended from the end portion EL to a position corresponding to the left head of the user in the mounted state of the image display portion 20.

  The right light guide plate 26 and the left light guide plate 28 are provided on the front frame 27. The right light guide plate 26 is located in front of the user's right eye in the mounted state of the image display unit 20 and makes the right eye visually recognize the image. The left light guide plate 28 is positioned in front of the user's left eye in the mounted state of the image display unit 20, and makes the left eye visually recognize the image.

  The front frame 27 has a shape in which one end of the right light guide plate 26 and one end of the left light guide plate 28 are connected to each other, and this connection position is used by the user while the user wears the image display unit 20. Corresponds to the gap between The front frame 27 may be provided with a nose pad portion that contacts the nose of the user in a mounted state of the image display unit 20 at the connection position of the right light guide plate 26 and the left light guide plate 28. In this case, the image display unit 20 can be held on the head of the user by the nose pad, the right holding unit 21 and the left holding unit 23. Further, a belt (not shown) in contact with the back of the user in the mounted state of the image display unit 20 may be connected to the right holding unit 21 and the left holding unit 23. In this case, the image display unit 20 is used by the belt. Can be held on the head of the person.

The right display unit 22 and the left display unit 24 are modules obtained by unitizing the optical unit and the peripheral circuit, respectively.
The right display unit 22 is a unit related to the display of an image by the right light guide plate 26. The right display unit 22 is provided in the right holding unit 21 and is positioned near the user's right head in the mounted state. The left display unit 24 is a unit related to the display of an image by the left light guide plate 28, is provided in the left holding unit 23, and is located near the left head of the user in the mounted state. The right display unit 22 and the left display unit 24 are collectively referred to simply as "display drive unit".

  The right light guide plate 26 and the left light guide plate 28 are optical parts formed of a light transmissive resin or the like, and guide the image light output from the right display unit 22 and the left display unit 24 to the eyes of the user. The right light guide plate 26 and the left light guide plate 28 are, for example, prisms.

  A light control plate (not shown) may be provided on the surfaces of the right light guide plate 26 and the left light guide plate 28. The light control plate is an optical element on a thin plate whose transmittance is different depending on the wavelength range of light, and functions as a so-called wavelength filter. The light control plate is disposed, for example, to cover the front side of the front frame 27 which is the side opposite to the side of the user's eye. By appropriately selecting the optical characteristics of this light control plate, it is possible to adjust the transmittance of light of any wavelength range such as visible light, infrared light and ultraviolet light, and the right light guide plate 26 and the left It is possible to adjust the amount of external light that is incident on the light plate 28 and transmitted through the right light guide plate 26 and the left light guide plate 28.

  The image display unit 20 guides the image light generated by the right display unit 22 and the left display unit 24 to the right light guide plate 26 and the left light guide plate 28, and causes the user to visually recognize the virtual image by the image light. Display When external light is incident on the user's eyes through the right light guide plate 26 and the left light guide plate 28 from the front of the user, the image light and the external light constituting the virtual image are incident on the user's eyes Therefore, the visibility of the virtual image is affected by the intensity of the external light. Therefore, for example, by mounting a light control plate on the front frame 27 and appropriately selecting or adjusting the optical characteristics of the light control plate, the visibility of the virtual image can be adjusted. In a typical example, it is possible to use a light control plate having a light transmittance that allows a user wearing the HMD 100 to at least visually recognize the outside scenery. Further, when the light control plate is used, the effect of protecting the right light guide plate 26 and the left light guide plate 28 and suppressing the damage of the right light guide plate 26 and the left light guide plate 28 or the adhesion of dirt can be expected. The light control plate may be detachably attached to each of the front frame 27 or the right light guide plate 26 and the left light guide plate 28. Alternatively, a plurality of types of light control plates may be replaced and mounted. May be omitted.

  The right display unit 22 and the left display unit 24 of the image display unit 20 are connected to the connection device 10, respectively. In the HMD 100, the connection cable 40 is connected to the left holding unit 23, the wiring connected to the connection cable 40 is laid inside the image display unit 20, and the right display unit 22 and the left display unit 24 are connected to the connection device 10. Ru.

  The camera 61 is disposed on the front frame 27 of the image display unit 20. It is desirable that the camera 61 picks up an image of the outside view direction viewed by the user while wearing the image display unit 20, and in the front of the front frame 27, outside light passing through the right light guide plate 26 and the left light guide plate 28 is It is provided at an unobstructed position. In the example of FIG. 1, the camera 61 is disposed on the end ER side of the front frame 27. The camera 61 may be disposed on the end EL side, or may be disposed at a connecting portion between the right light guide plate 26 and the left light guide plate 28.

  The camera 61 is a digital camera provided with an imaging element such as a CCD or CMOS, an imaging lens, etc. The camera 61 in this embodiment is a monocular camera, but may be a stereo camera. The camera 61 captures an outside view of at least a part of the direction of the front side of the HMD 100, in other words, the direction of the user's view in the state where the HMD 100 is mounted. The outside view can be reworded as real space.

  In another expression, the camera 61 images an area or direction overlapping the user's field of view, and images the direction in which the user gazes. Although the width of the angle of view of the camera 61 can be set as appropriate, in the present embodiment, as described later, the user includes the external world visually recognized through the right light guide plate 26 and the left light guide plate 28. More preferably, the imaging range of the camera 61 is set so as to be able to image the entire view of the user visible through the right light guide plate 26 and the left light guide plate 28.

The HMD 100 includes a distance sensor 64. The distance sensor 64 is disposed at the boundary between the right optical image display unit 26 and the left optical image display unit 28. When the user wears the image display unit 20, the position of the distance sensor 64 is approximately halfway between the user's eyes in the horizontal direction and above the user's eyes in the vertical direction.
The distance sensor 64 detects the distance to the measurement object located in the preset measurement direction. The measurement direction of the distance sensor 64 of the present embodiment is the front side direction of the HMD 100, and overlaps the imaging direction of the camera 61.

FIG. 2 is a plan view of relevant parts showing the configuration of the optical system of the HMD 100. FIG. The left eye LE and the right eye RE of the user are illustrated in FIG. 2 for the sake of explanation.
As shown in FIG. 2, the right display unit 22 and the left display unit 24 are configured to be laterally symmetrical. The right display unit 22 includes an OLED (Organic Light Emitting Diode) unit 221 that emits image light, as a configuration for causing the user's right eye RE to visually recognize the image. The right optical system 251 is provided with a lens group or the like for guiding the image light L emitted by the OLED unit 221. The image light L is guided to the right light guide plate 26 by the right optical system 251.

  The OLED unit 221 includes an OLED panel 223 and an OLED driving circuit 225 that drives the OLED panel 223. The OLED panel 223 is a self-emission type display configured by arranging light emitting elements emitting light by organic electroluminescence to emit R (red), G (green) and B (blue) color light in a matrix, respectively. It is a panel. The OLED panel 223 includes a plurality of pixels with a unit including one each of R, G, and B elements as one pixel, and forms an image by pixels arranged in a matrix.

The OLED drive circuit 225 executes selection of the light emitting element included in the OLED panel 223 and energization of the light emitting element based on the image data input from the connection device 10 to cause the light emitting element of the OLED panel 223 to emit light. The OLED driving circuit 225 is fixed to the back surface of the OLED panel 223, that is, the back side of the light emitting surface by bonding or the like. The OLED drive circuit 225 is configured of, for example, a semiconductor device that drives the OLED panel 223, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 223. A temperature sensor 217 is mounted on this substrate.
The OLED panel 223 may have a configuration in which light emitting elements emitting white light are arranged in a matrix, and color filters corresponding to R, G, and B colors are overlapped and arranged. In addition to light emitting elements that respectively emit R, G, and B color light, an OLED panel 223 having a WRGB configuration may be used that includes light emitting elements that emit W (white) light.

  The right optical system 251 has a collimating lens that converts the image light L emitted from the OLED panel 223 into parallel light beams. The image light L converted into parallel light beams by the collimator lens is incident on the right light guide plate 26. A plurality of reflecting surfaces for reflecting the image light L are formed in the light path for guiding the light inside the right light guide plate 26. The image light L is guided to the right eye RE side through multiple reflections inside the right light guide plate 26. The right light guide plate 26 is formed with a half mirror 261 (reflection surface) located in front of the right eye RE. The image light L is reflected by the half mirror 261 and emitted from the right light guide plate 26 toward the right eye RE, and this image light L forms an image on the retina of the right eye RE and causes the user to view the image.

  In addition, as a configuration for causing the left eye LE of the user to visually recognize an image, the left display unit 24 includes an OLED unit 241 that emits image light and a lens group that guides the image light L emitted by the OLED unit 241. And 252. The image light L is guided to the left light guide plate 28 by the left optical system 252.

  The OLED unit 241 includes an OLED panel 243 and an OLED driving circuit 245 that drives the OLED panel 243. The OLED panel 243 is a self-emission display panel configured similarly to the OLED panel 223. The OLED drive circuit 245 executes selection of the light emitting element included in the OLED panel 243 and energization of the light emitting element based on the image data input from the connection device 10 to cause the light emitting element of the OLED panel 243 to emit light. The OLED driving circuit 245 is fixed to the back surface of the OLED panel 243, that is, the back side of the light emitting surface by bonding or the like. The OLED drive circuit 245 is configured of, for example, a semiconductor device that drives the OLED panel 243, and may be mounted on a substrate (not shown) fixed to the back surface of the OLED panel 243. A temperature sensor 239 is mounted on this substrate.

  The left optical system 252 has a collimating lens that converts the image light L emitted from the OLED panel 243 into parallel light beams. The image light L converted into parallel light beams by the collimator lens is incident on the left light guide plate 28. The left light guide plate 28 is an optical element in which a plurality of reflecting surfaces for reflecting the image light L are formed, and is, for example, a prism. The image light L is guided to the left eye LE side through multiple reflections inside the left light guide plate 28. The left light guide plate 28 is formed with a half mirror 281 (reflection surface) positioned in front of the left eye LE. The image light L is reflected by the half mirror 281 and emitted from the left light guide plate 28 toward the left eye LE, and this image light L forms an image on the retina of the left eye LE to make the user visually recognize the image.

  The HMD 100 functions as a see-through display device. That is, the image light L reflected by the half mirror 261 and the external light OL transmitted through the right light guide plate 26 enter the right eye RE of the user. Further, the image light L reflected by the half mirror 281 and the external light OL transmitted through the half mirror 281 enter the left eye LE. As described above, the HMD 100 superimposes the image light L of the image processed internally and the external light OL and makes the light incident on the user's eyes, and for the user, the right light guide plate 26 and the left light guide plate 28 Is superimposed on this outside scene, and an image by the image light L is viewed. The half mirrors 261 and 281 are image extracting units that reflect the image light output from the right display unit 22 and the left display unit 24 to extract an image, and can also be called a display unit.

  The left optical system 252 and the left light guide plate 28 are collectively referred to as "left light guide portion", and the right optical system 251 and the right light guide plate 26 are collectively referred to as "right light guide portion". The configuration of the right light guide and the left light guide is not limited to the above example, and any method can be used as long as a virtual image is formed in front of the user's eye using image light, for example, a diffraction grating It may be used or a semi-transmissive reflective film may be used.

FIG. 3 is a perspective view of the main part when the image display unit 20 is viewed from the head side of the user, the side of the image display unit 20 in contact with the user's head, in other words, the right eye RE and left eye of the user It is the side seen by LE. In other words, the back side of the right light guide plate 26 and the left light guide plate 28 can be seen.
In FIG. 3, the half mirror 261 that irradiates the image light to the right eye RE of the user and the half mirror 281 that irradiates the image light to the left eye LE can be seen as a substantially rectangular area. Further, the entire right light guide plate 26 and the left light guide plate 28 including the half mirrors 261 and 281 transmit external light as described above. Therefore, the user can visually recognize the outside scene through the entire right light guide plate 26 and the left light guide plate 28, and a rectangular display image is visually recognized at the positions of the half mirrors 261 and 281.

FIG. 4 is an explanatory view showing the correspondence between the image display unit 20 of the HMD 100 and the imaging range.
The camera 61 is disposed at the right end of the image display unit 20 as described above, and images the direction in which the user's eyes are directed, that is, the front of the user.

  FIG. 4 is a view schematically showing the position of the camera 61 together with the right eye RE and the left eye LE of the user in a plan view. The angle of view (imaging range) of the camera 61 is indicated by C. Although the angle of view C in the horizontal direction is shown in FIG. 4, the actual angle of view of the camera 61 extends in the vertical direction as in a general digital camera.

  The optical axis of the camera 61 is a direction including the viewing direction RD of the right eye RE and the viewing direction LD of the left eye LE. The view that can be viewed by the user while wearing the HMD 100 is not limited to infinity. For example, as shown in FIG. 4, when the user gazes at the object OB with both eyes, the user's gaze RD, LD is directed to the object OB. In this case, the distance from the user to the object OB is often about 30 cm to 10 m, and more often about 1 m to 4 m. Therefore, for the HMD 100, an indication of the upper limit and the lower limit of the distance from the user to the object OB during normal use may be determined. This standard may be obtained by survey or experiment, or may be set by the user. The optical axis of the camera 61, and the angle of view, when the distance to the object OB in normal use corresponds to a standard of the set upper limit, and when it corresponds to a standard of the lower limit, It is preferably set to be included in the angle of view.

  Generally, the viewing angle of human beings is approximately 200 degrees in the horizontal direction and approximately 125 degrees in the vertical direction, and of these, the effective field of view excellent in information acceptance ability is approximately 30 degrees in the horizontal direction and approximately 20 degrees in the vertical direction. Furthermore, a stable fixation field in which a fixation point at which a person gazes is seen quickly and stably appears to be 60 to 90 degrees in the horizontal direction and 45 degrees to 70 degrees in the vertical direction. In this case, when the fixation point is the object OB of FIG. 4, the effective visual field is approximately 30 degrees in the horizontal direction and approximately 20 degrees in the vertical direction around the sight lines RD and LD. Further, a stable fixation field of about 60 to 90 degrees in the horizontal direction and 45 to 70 degrees in the vertical direction is a stable fixation field, and a viewing angle is about 200 degrees in the horizontal direction and about 125 degrees in the vertical direction. Furthermore, an actual visual field that a user passes through the image display unit 20 and passes through the right light guide plate 26 and the left light guide plate 28 for visual recognition can be called a field of view (FOV). In the configuration of the present embodiment shown in FIG. 1 and FIG. 2, the actual field of view corresponds to the actual field of view seen by the user through the right light guide plate 26 and the left light guide plate 28. The real view is narrower than the view angle and the stable fixation, but wider than the effective view.

  The angle of view C of the camera 61 is preferably capable of imaging a wider range than the field of view of the user. Specifically, the angle of view C is preferably at least wider than the effective field of view of the user. Furthermore, it is more preferable that the angle of view C be wider than the user's actual field of view. More preferably, the angle of view C is wider than the user's stable fixation field, and most preferably, the angle of view C is wider than the viewing angle of both eyes of the user.

  The camera 61 may have a so-called wide-angle lens as an imaging lens, and may be configured to be capable of imaging a wide angle of view. The wide-angle lens may include a lens called an ultra-wide-angle lens or a quasi-wide-angle lens, or may be a single-focus lens or a zoom lens, and the camera 61 includes a lens group including a plurality of lenses. It may be.

  Further, as described above, the camera 61 of the present embodiment is disposed on the end ER side in the front frame 27 of the image display unit 20, but may be disposed on the end EL side. And the left light guide plate 28 may be disposed. In this case, the position of the camera 61 in the left-right direction is different from the position in FIG. 4, and the angle of view C is appropriately set according to the position of the camera 61. Specifically, when the camera 61 is on the end EL side, the angle of view C faces the front right in FIG. If, for example, the camera 61 is disposed at the connection portion between the right light guide plate 26 and the left light guide plate 28, the angle of view C faces the front of the image display unit 20.

When the user looks at an object with the right eye RE and the left eye LE, the distance to the object is perceived and recognized by the angle formed by the direction of the right eye RE and the direction of the left eye LE. This angle is called the convergence angle, and the convergence angle is PA, for example, when looking at the object OB shown in FIG.
The convergence angle when the user looks at the image displayed on the half mirror 261, 281 is the viewing direction when the image of the half mirror 261 is viewed with the right eye RE and the image when the half mirror 281 is viewed with the left eye LE. Is the angle with the direction of the eyes of the The size of the convergence angle in this case is determined by the display position of the image on the half mirrors 261 and 281. Therefore, by adjusting the display position at which the right display unit 22 and the left display unit 24 display the image, the convergence angle can be controlled to control the sense of distance that the user visually recognizes. For example, with regard to the images displayed by the right display unit 22 and the left display unit 24, it is possible to adjust the sense of distance (visible distance) recognized by the user.

  Further, the distance sensor 64 is disposed to face forward at the center of the right optical image display unit 26 and the left optical image display unit 28.

[1-2. Control system of display system]
FIG. 5 is a block diagram showing the configurations of the HMD 100 and the PC 300 that constitute the display system 1.
As described above, the HMD 100 is configured by connecting the connection device 10 and the image display unit 20 by the connection cable 40.

The image display unit 20 includes the right display unit 22 and the left display unit 24 as described above. The right display unit 22 has a display unit substrate 210. The display unit substrate 210 includes a connection unit 211 connected to the connection cable 40, a reception unit (Rx) 213 that receives data input from the connection device 10 via the connection unit 211, and an EEPROM 215 (storage unit). Implemented.
The connection unit 211 connects the reception unit 213, the EEPROM 215, the temperature sensor 217, the camera 61, the distance sensor 64, the illuminance sensor 65, and the LED indicator 67 to the connection device 10.

  An EEPROM (Electrically Erasable Programmable Read-Only Memory) 215 stores various data in a nonvolatile manner. The EEPROM 215 stores, for example, data related to light emission characteristics and display characteristics of the OLED units 221 and 241 included in the image display unit 20, data related to characteristics of sensors included in the right display unit 22 or the left display unit 24, and the like. Specifically, parameters relating to the gamma correction of the OLED units 221 and 241, data for compensating the detection values of the temperature sensors 217 and 239, and the like are stored. These data are generated by factory inspection of the HMD 100 and written to the EEPROM 215. The data stored in the EEPROM 215 can be read by the control unit 120.

The camera 61 performs imaging in accordance with a signal input via the connection unit 211, and outputs captured image data to the connection unit 211.
As shown in FIG. 1, the illuminance sensor 65 is provided at the end ER of the front frame 27 and is arranged to receive external light from the front of the user wearing the image display unit 20. The illuminance sensor 65 outputs a detection value corresponding to the light receiving amount (light receiving intensity).
The LED indicator 67 is disposed near the camera 61 at the end ER of the front frame 27, as shown in FIG. The LED indicator 67 lights up while imaging by the camera 61 is performed to notify that imaging is in progress.

  The temperature sensor 217 detects a temperature, and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 217 is mounted on the back side of the OLED panel 223 (FIG. 3). The temperature sensor 217 may be mounted on the same substrate as the OLED drive circuit 225, for example. With this configuration, the temperature sensor 217 mainly detects the temperature of the OLED panel 223.

The distance sensor 64 performs distance detection, and outputs a signal indicating the detection result to the connection device 10 via the connection unit 211. The distance sensor 64 may be, for example, an infrared depth sensor, an ultrasonic distance sensor, a TOF (Time Of Flight) distance sensor, or a distance detection unit in which image detection and voice detection are combined. Alternatively, the distance may be detected by processing an image obtained by stereo imaging with a stereo camera or a single-eye camera.
Although one distance sensor 64 is illustrated in FIG. 5, a pair of distance sensors 64, 64 shown in FIG. 4 may operate simultaneously. In addition, each of the pair of distance sensors 64 may be connected to the connection unit 211 and operate independently.

  The receiving unit 213 receives the image data for display transmitted from the connection apparatus 10 via the connection unit 211 and outputs the image data to the OLED unit 221.

The left display unit 24 has a display unit substrate 210. On the display unit substrate 210, a connection unit 231 connected to the connection cable 40 and a reception unit (Rx) 233 for receiving data input from the connection device 10 through the connection unit 231 are mounted. Further, on the display unit substrate 210, a six-axis sensor 235 and a magnetic sensor 237 are mounted.
The connection unit 231 connects the reception unit 233, the six-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239 to the connection device 10.

  The six-axis sensor 235 is a motion sensor (inertial sensor) including a three-axis acceleration sensor and a three-axis gyro (angular velocity) sensor. The six-axis sensor 235 may employ an IMU (Inertial Measurement Unit) in which the above-described sensor is modularized. The magnetic sensor 237 is, for example, a triaxial geomagnetic sensor.

The temperature sensor 239 detects a temperature, and outputs a voltage value or a resistance value corresponding to the detected temperature as a detected value. The temperature sensor 239 is mounted on the back side of the OLED panel 243 (FIG. 3). The temperature sensor 239 may be mounted on the same substrate as the OLED drive circuit 245, for example. With this configuration, the temperature sensor 239 mainly detects the temperature of the OLED panel 243.
Also, the temperature sensor 239 may be incorporated in the OLED panel 243 or the OLED drive circuit 245. The substrate may be a semiconductor substrate. Specifically, when the OLED panel 243 is implemented as an Si-OLED, as an integrated circuit on an integrated semiconductor chip together with the OLED drive circuit 245 etc, the temperature sensor 239 may be implemented on this semiconductor chip.

  Each part of the image display unit 20 operates with the power supplied from the connection device 10 by the connection cable 40. The image display unit 20 may include a power supply circuit (not shown) for performing voltage conversion and distribution of the power supplied by the connection cable 40.

  The connection device 10 includes an I / F (interface) unit 110, a control unit 120, a display control unit 122, a sensor control unit 124, a power control unit 126, a non-volatile storage unit 130, an operation unit 140, and a connection unit 145. . The I / F unit 110 as an acquisition unit includes connectors 11A, 11B, and 11C. The I / F unit 110 may also include an interface circuit (not shown) connected to the connectors 11A, 11B, and 11C to execute communication protocols conforming to various communication standards. The I / F unit 110 may be configured to receive power supply via the connectors 11A, 11B, and 11C.

  The I / F unit 110 may include, for example, a memory card interface or the like to which an external storage device or storage medium can be connected, or the I / F unit 110 may be configured by a wireless communication interface. The I / F unit 110 may be, for example, an interface board on which the connectors 11A, 11B, 11C and the interface circuit are mounted. Further, the control unit 120, the display control unit 122, the sensor control unit 124, and the power control unit 126 of the connection device 10 may be mounted on the connection device main board (not shown). In this case, the connectors 11A, 11B, 11C and the interface circuit of the I / F unit 110 may be mounted on the connection device main board.

The control unit 120 controls each unit of the connection device 10. The control unit 120 includes a processor (not shown) such as a CPU (Central Processing Unit) or a microcomputer. The control unit 120 controls each unit of the HMD 100 by cooperation of software and hardware by executing a program by the processor. Also, the control unit 120 may be configured by programmed hardware. The control unit 120 may include, in addition to the processor, a random access memory (RAM) forming a work area, and a read only memory (ROM) storing a control program. In addition, the semiconductor device may be an integrated processor, RAM, and ROM.
The non-volatile storage unit 130, the operation unit 140, and the connection unit 145 are connected to the control unit 120.

The display control unit 122 executes various processes for displaying an image based on image data and video data input to the I / F unit 110 by the image display unit 20. For example, the display control unit 122 executes various processes such as frame extraction, resolution conversion (scaling), intermediate frame generation, and frame rate conversion. The display control unit 122 outputs the image data corresponding to each of the OLED unit 221 of the right display unit 22 and the OLED unit 241 of the left display unit 24 to the connection unit 145. The image data input to the connection unit 145 is transmitted to the connection units 211 and 231 via the connection cable 40.
In a specific example of “cutout” of a frame, the display control unit 122 develops in the work area the entire image larger than the size of the display area where the image display unit 20 displays the image. The display area is, for example, an area where an image is formed in the OLED units 221 and 241. Also, the size of the image is indicated by the number of pixels or resolution. The display control unit 122 transfers, to the right display unit 22 and the left display unit 24, only the data of the region to be cut out of the developed image.

  In addition, the display control unit 122 acquires image data larger than the display area of the image display unit 20, and generates image data in which only image data of a size to be displayed by the image display unit 20 is extracted from the acquired image data. It may be transferred to the right display unit 22 and the left display unit 24.

  The display control unit 122 executes 3D video decoding, for example, when the video data input to the I / F unit 110 is 3D (stereoscopic) video data. In the 3D video decoding process, the display control unit 122 generates a frame for the right eye and a frame for the left eye from the 3D video data. Examples of the format of 3D video data input to the I / F unit 110 include a side-by-side format, a top-and-bottom format, and a frame packing format, but may be 3D model data.

  The display control unit 122 is connected to the connector 11A included in the I / F unit 110 and the connector 11B. The display control unit 122 executes processing on video data input to the connector 11A and video data input to the connector 11B. In addition, the display control unit 122 may have a function of transmitting and receiving various control data related to transmission of video data with the device connected to the connector 11A or the connector 11B.

  The sensor control unit 124 controls the camera 61, the distance sensor 64, the illuminance sensor 65, the temperature sensor 217, the six-axis sensor 235, the magnetic sensor 237, and the temperature sensor 239. Specifically, the sensor control unit 124 performs setting and initialization of the sampling cycle of each sensor according to the control of the control unit 120, powers on each sensor, and transmits control data according to the sampling cycle of each sensor, Execute acquisition of detected value etc.

  Further, the sensor control unit 124 is connected to the connector 11B of the I / F unit 110, and outputs, to the connector 11B, data on detection values acquired from each sensor at a preset timing. As a result, the device connected to the connector 11B can acquire the detection value of each sensor of the HMD 100 and the captured image data of the camera 61. The data output by the sensor control unit 124 may be digital data including a detection value. Further, the sensor control unit 124 may output data as a result of arithmetic processing based on the detection value of each sensor. For example, the sensor control unit 124 integrally processes detection values of a plurality of sensors and functions as a so-called sensor fusion processing unit. By executing the sensor fusion, the sensor control unit 124 outputs data obtained from the detection value of the sensor, for example, movement trajectory data of the image display unit 20, relative coordinate data of the image display unit 20, and the like. The sensor control unit 124 may have a function of transmitting and receiving various control data related to data transmission with a device connected to the connector 11B.

  The display control unit 122 and / or the sensor control unit 124 may be realized by cooperation of software and hardware when a processor such as a CPU executes a program. That is, the display control unit 122 and the sensor control unit 124 are configured by a processor, and execute the above-described operation by executing a program. In this example, the display control unit 122 and the sensor control unit 124 may be realized as a processor configuring the control unit 120 executes a program. In other words, the processor may function as the control unit 120, the display control unit 122, and the sensor control unit 124 by executing a program. Here, the processor can be reworded as a computer.

  In addition, the display control unit 122 and the sensor control unit 124 may be configured by programmed hardware such as a digital signal processor (DSP) or a field programmable gate array (FPGA). Further, the display control unit 122 and the sensor control unit 124 may be integrated to be configured as a SoC (System-on-a-Chip) -FPGA.

The power control unit 126 is connected to the connector 11B and the connector 11C included in the I / F unit 110. The power control unit 126 supplies power to each unit of the connection device 10 and the image display unit 20 based on the power supplied from the connectors 11B and 11C. The power supply control unit 126 may be configured to include a voltage conversion circuit (not shown) and supply different voltages to the connection device 10 and each unit of the image display unit 20. The power control unit 126 may be configured by a logic circuit or a programmed semiconductor device such as an FPGA. Further, the power control unit 126 may be configured by hardware (including a processor) common to the display control unit 122 and / or the sensor control unit 124.
The display control unit 122, the sensor control unit 124, and the power supply control unit 126 may have a work memory for performing data processing, and processing is performed using a work area of a RAM (not shown) provided in the control unit 120. You may go.

  The control unit 120 reads data from the EEPROM 215 of the right display unit 22 and sets the operations of the display control unit 122 and the sensor control unit 124 based on the read data. Further, the control unit 120 operates the respective units including the display control unit 122, the sensor control unit 124, and the power control unit 126 according to the operation in the operation unit 140. In addition, the control unit 120 causes the display control unit 122, the sensor control unit 124, and the power supply control unit 126 to identify devices connected via the I / F unit 110, and to execute an operation suitable for each device. The display control unit 122, the sensor control unit 124, and the power control unit 126 are controlled.

  Further, the control unit 120 controls the start and stop of energization of the LED indicator 67. For example, the LED indicator 67 is lighted or blinked at the timing when the camera 61 starts and ends imaging.

  The non-volatile storage unit 130 is a storage device that stores data and the like processed by the control unit 120 in a non-volatile manner. The nonvolatile storage unit 130 is, for example, a magnetic storage device such as a hard disk drive (HDD) or a storage device using a semiconductor storage element such as a flash memory.

  Further, the connection device 10 may be provided with a rechargeable battery (not shown), and power may be supplied from the battery to the connection device 10 and each part of the image display unit 20.

  The PC 300 includes a control unit 310, a non-volatile storage unit 320, a display unit 330, an I / F (interface) unit 341, and a communication unit 345. The control unit 310 (electronic device control unit) includes a processor (not shown) such as a CPU or a microcomputer, and controls each unit of the PC 300 by executing a program using this processor. The control unit 310 may include a ROM which non-volatilely stores a control program executed by a processor (so-called computer), and a RAM which constitutes a work area of the processor.

  The non-volatile storage unit 320 stores the program executed by the control unit 310 and the data processed by the control unit 310 in a non-volatile manner. The non-volatile storage unit 130 is, for example, a storage device using a magnetic recording device such as an HDD or a semiconductor storage element such as a flash memory.

The non-volatile storage unit 320 stores, for example, content data 321 of content including video. The content data 321 is a file of a format that can be processed by the control unit 310, includes video data, and may include audio data.
Further, the non-volatile storage unit 320 stores an operating system (OS) as a basic control program executed by the control unit 310, an application program operating with the OS as a platform, and the like. In addition, the non-volatile storage unit 320 stores data processed at the time of execution of the application program and data of the processing result.

  The display panel 331 included in the display unit 330 and the touch sensor 332 are connected to the control unit 310. The display panel 331 displays various images based on the control of the control unit 310. The touch sensor 332 detects a touch operation, and outputs data indicating the detected operation to the control unit 310. Data output from the touch sensor 332 is coordinate data or the like indicating an operation position on the touch sensor 332.

  The I / F unit 341 is an interface connected to an external device, and corresponds to the output unit of the present invention. The I / F unit 341 executes, for example, communication conforming to a standard such as an HDMI interface or a USB interface. The I / F unit 341 includes a connector (not shown) that connects a cable (for example, the cable 2), and an interface circuit (not shown) that processes signals transmitted through the connector. The I / F unit 341 is an interface board having a connector and an interface circuit, and is connected to a main board on which the processor or the like of the control unit 310 is mounted. Alternatively, the connector and interface circuit constituting the I / F unit 341 are mounted on the main substrate of the PC 300. In the present embodiment, the I / F unit 341 includes an HDMI interface and a USB interface, and is connected to the connectors 11A and 11B by the HDMI cable 2A and the USB cable 2B. For example, the control unit 310 outputs video data by the HDMI cable 2A, and receives data and the like regarding the output value of the sensor from the connection device 10 by the USB cable 2B. The I / F unit 341 can independently execute communication by the HDMI cable 2A and communication by the USB cable 2B. The I / F unit 341 may also be a wireless communication interface. In this case, the I / F unit 341 can be an interface substrate on which a communication circuit including an RF unit is mounted, or a circuit mounted on a main substrate.

  The communication unit 345 is a communication interface that performs data communication with an external device. The communication unit 345 may be a wired communication interface capable of connecting a cable or may be a wireless communication interface. For example, a wired LAN interface compatible with Ethernet (registered trademark) or a wireless LAN interface compliant with the IEEE 802.11 standard may be used.

The control unit 310 functions as the input / output control unit 311, the detection value acquisition unit 312, the position detection unit 313, and the image adjustment unit 315 by executing the program as described above.
The input / output control unit 311 detects an input by the user based on data input from the touch sensor 332. Further, the input / output control unit 311 controls input / output of data by the I / F unit 341 and the communication unit 345.

  The detection value acquisition unit 312 acquires, from the connection device 10 connected by the I / F unit 341, data on detection values of the respective sensors included in the HMD 100. The detection value acquisition unit 312 acquires, from the connection device 10, data regarding detection values of the captured image data of the camera 61, the distance sensor 64, the illuminance sensor 65, the EEPROM 215, the six-axis sensor 235, the magnetic sensor 237, the temperature sensor 239, and the like. Do. The data acquired by the detection value acquisition unit 312 is data processed by the sensor control unit 124, and may be data including detection values of each sensor or data statistically processed by the sensor control unit 124. .

The position detection unit 313 detects the position of the HMD 100 based on the data acquired by the detection value acquisition unit 312. More specifically, the position detection unit 313 detects the relative position between the PC 300 and the image display unit 20. Here, the relative position detected by the position detection unit 313 may be a position in the space in which the image display unit 20 and the PC 300 exist, or may include the relative orientation of the image display unit 20 and the display unit 330. . For example, it may be information indicating the position and / or the direction of the display unit 330 based on the image display unit 20. Further, for example, information indicating the position and / or the direction of the image display unit 20 with reference to the PC 300 may be used. Also, for example, coordinates in a three-dimensional coordinate system set in the space in which the image display unit 20 and the PC 300 exist may be included.
The position detection by the position detection unit 313 will be described later.

  The video data output from the PC 300 to the connection device 10 can be video data of a screen displayed by the display unit 330 as well as video data obtained by reproducing the content data 321. In this case, the connection adapter 10 displays the same screen as the display unit 330 and performs so-called mirroring display.

  Further, the image adjustment unit 315 may be configured to display a larger image including the image displayed on the connection device 10 and the image not displayed on the PC 300 on the display unit 330. Also, for example, the connection device 10 may be configured to enlarge and display a part of the image displayed by the PC 300. The display mode of the PC 300 will be described later.

[1-3. Display System Behavior]
FIG. 6 is an explanatory view showing a process of obtaining the relative position of the PC 300 with respect to the image display unit 20, and particularly shows a calibration process. FIG. 7 is a flowchart showing the operation of the display system 1, and particularly shows the operation of the PC 300 in the calibration process.

  In FIG. 6, a visual field which the user visually recognizes by passing through the image display unit 20 is indicated by a symbol VR, and an imaging range of the camera 61 is indicated by a symbol VC. In this example, the imaging range VC overlaps with the visual field VR but does not match. The imaging range VC and the visual field VR may coincide with each other, or the imaging range VC may be included in the visual field VR.

Using the display system 1, as shown in FIG. 6, calibration is performed in a state where the PC 300 exists in the imaging range VC of the camera 61.
In the present embodiment, the reference point P1 is set in the display unit 330, and the reference position P2 is set in the image display unit 20. The reference point P1 is a display that can be visually or recognized by the user, and is a so-called marker. The PC 300 can detect the reference point P1 from the captured image data of the camera 61 by image processing. The reference position P2 is set as a position reference in the image display unit 20, may be a virtual position, and may not have a display or an object indicating the reference position P2.
The reference point P1 may be an object or a position explicitly recognized, and there is no need to newly provide a display or an object to be the reference point P1. For example, the specific position of the pattern or shape of the display unit 330 may be set as the reference point P1, such as setting one of four corners of the display unit 330 as the reference point P1.

FIG. 7 is a flowchart showing the operation of the display system 1 and shows the operation of the PC 300 that performs calibration.
In the calibration, the user moves the head or the image display unit 20 so that the reference point P1 is positioned at a position preset in the user's visual field VR (step S11). When the set position in the visual field VR and the reference point P1 overlap, the user performs an operation for determining the position on the image display unit 20 or the PC 300 (step S12). For example, the user performs an operation of hitting the front frame 27 of the image display unit 20. This operation can be detected as acceleration by the six-axis sensor 235. When the image display unit 20 is equipped with a knock sensor (not shown), the operation can be detected based on the detection value of the knock sensor. The operation of step S12 may be an operation detected by the touch sensor 332. In step S12, the PC 300 may detect an operation by voice. That is, in a configuration in which the PC 300 has a microphone and the control unit 310 can process voice collected by the microphone, the control unit 310 performs voice recognition processing for converting voice into text, or a pattern registered in advance. Voice command detection processing for detecting voice can be executed. In this configuration, the control unit 310 may detect the voice emitted by the user as an input operation in step S12.

  The control unit 310 obtains captured image data of the camera 61 when the operation in step S12 is detected, and obtains the position of the reference point P1 in the captured image data of the camera 61 (step S13). Thereby, the position in the captured image data of the camera 61 and the position in the field of view VR of the user wearing the image display unit 20 are associated with each other. Therefore, when the PC 300 appears in the captured image data of the camera 61, the relative position of the PC 300 with respect to the visual field VR can be specified from the captured image data.

  The control unit 310 acquires data on output values of the six-axis sensor 235 and the magnetic sensor 237 (step S14). The output value of the sensor acquired in step S14 indicates the reference position of the image display unit 20 in a state in which the reference point P1 overlaps the set position of the visual field VR. For example, by integrating the output values of the six-axis sensor 235, the direction and the amount of movement of the image display unit 20 from the reference position can be obtained. Further, based on the output value of the magnetic sensor 237, the relative direction of the image display unit 20 with respect to the reference position of the image display unit 20 can be obtained. Therefore, the relative position between the image display unit 20 and the PC 300 can be obtained.

The control unit 310 generates calibration data including the output value acquired in step S14, stores the calibration data in the non-volatile storage unit 320 (step S15), and ends the present process.
For example, the position detection unit 313 executes steps S13 and S15, and the detection value acquisition unit 312 executes step S14.

  In the calibration process shown in FIG. 7, the reference point P1 of the PC 300 and the set position in the visual field VR are matched to associate the relative positional relationship of the image display unit with the reference position of the image display unit. The calibration data generated in this process is, for example, data indicating the reference of the relative position between the reference point P1 and the visual field VR. In this calibration, in addition to overlapping the predetermined position in the visual field VR with the reference point P1, a method may be employed in which the PC 300 is arranged at a predetermined distance from the image display unit 20 in step S11. For example, in step S11, the image display unit 20 may display an image for guiding the PC 300 to be away from the image display unit 20 by a predetermined distance. This image can be a marker such as a straight line, a rectangle, or a point indicating the size at which the PC 300 at a position separated by a predetermined distance is viewed. The user may adjust the relative position of the PC 300 to the marker of the image display unit 20 so that the PC 300 appears to overlap with the marker of the image display unit 20, and then adjust the reference point P1 to a predetermined position to perform the operation of step S12.

  FIG. 8, FIG. 9, and FIG. 10 are diagrams showing examples of display modes of the display system 1 in the present embodiment. In FIG. 8, FIG. 9 and FIG. 10, a visual field VR shows a visual field in which the user wearing the image display unit 20 can visually recognize the light through the image display unit 20. Moreover, in the example shown to these figures, the display area which the image display part 20 displays an image so that a user may visually recognize is the same magnitude | size and position as the visual field VR, and overlaps. The present invention is also applicable to a configuration in which the display area of the image display unit 20 is smaller than the visual field VR.

FIG. 8 shows a virtual image VD generated by the PC 300. The virtual image VD includes an area VD1 including the same image as the image displayed on the display unit 330 by the PC 300, and areas VD2, VD3, and VD4 adjacent to the area VD1. That is, the virtual image VD is an image larger in size than the image displayed on the display unit 330.
The PC 300 sets a virtual display area larger than the display area of the display panel 331, and generates a virtual image VD as an image displayed in the display area. The virtual display area and the virtual image VD may be different in resolution and size, but in the present embodiment, they will be described as being coincident.

  In the following description, the sizes of the image displayed by the display unit 330, the virtual image VD, and the areas VD1, VD2, VD3, and VD4 are indicated by display resolution. For example, assuming that the display resolution of the display unit 330 is 1920 dots wide × 1080 dots high and the resolution of the virtual image VD and the virtual display area is 3840 dots × 2160 dots, the size of the virtual image VD is the display image of the display unit 330 Four times

  The aspect ratio of the image displayed by the display unit 330 and the virtual image VD may be different, but if the aspect ratios match, the process for generating the virtual image VD including the display image of the display unit 330 is It has the advantage of being able to be implemented quickly. In addition, when the user visually recognizes both the display image of the display unit 330 and the virtual image VD, there is an advantage that there is no sense of incongruity, which is preferable. For the same reason, the aspect ratios and the resolutions of the regions VD1 to VD4 constituting the virtual image VD may not be identical to those of the display unit 330, but are preferably identical to each other. The resolution of the virtual image VD may not be an integral multiple of the display resolution of the display unit 330, but is preferably an integral multiple.

  The size of the virtual image VD is preferably the same as or larger than the image that can be displayed by the image display unit 20. That is, the resolution of the virtual image VD is preferably equal to or higher than the display resolution of the display area of the image display unit 20. As an example, the display resolution of the display unit 330 is 1920 dots wide × 1080 dots high, the display resolution of the image display unit 20 is 3840 dots × 1080 dots, and the resolution of the virtual image VD is 3840 dots × 2160 dots Can be mentioned. Also, the aspect ratio of the display area of the virtual image VD and the display area of the image display unit 20 may be different, and the aspect ratio and resolution of each of the areas VD1 to VD4 do not match the display area of the image display unit 20. It is also good. In addition, the resolution of the virtual image VD may not be an integral multiple of the resolution of the display area of the image display unit 20.

  The virtual image VD can be referred to as an expanded display area in which the image displayed on the display unit 330 by the PC 300 is expanded. For example, when a screen for operation is displayed on the display unit 330 as a user interface of an operating system or application program executed by the PC 300, the control unit 310 configures the screen for operation in accordance with the display resolution of the display unit 330. . When using the virtual image VD, the control unit 310 configures a screen as a user interface in accordance with the resolution of the virtual image VD, and causes the display unit 330 to display a part of the screen. The same applies to the case where the PC 300 reproduces the content data 321 or the case where another image or the like is displayed. As described above, the virtual image VD includes the image displayed on the display unit 330. In other words, the display unit 330 displays a part of the virtual image VD.

The size of the virtual image VD, the display image of the display unit 330, and the image displayed by the image display unit 20 may be defined as a size visually recognized by the user wearing the image display unit 20.
The size of the image is determined by the size of the pixel and the resolution. It is appropriate to consider the size of the pixel of the display unit 330 as a pixel when the user visually recognizes the display unit 330 through the image display unit 20. Further, since the virtual image VD is not viewed unless the image display unit 20 displays it, the size of the pixel of the virtual image VD is the size when the user visually recognizes the pixel of the image displayed by the image display unit 20. It is appropriate to think. Then, the virtual image VD may be larger than the display unit 330 based on the size visually recognized by the user wearing the image display unit 20. In this case, the resolution of the virtual image VD may be equal to or less than the display resolution of the display unit 330. Furthermore, the resolution of the video data cut out from the virtual image VD and output to the image display unit 20 may be a resolution equal to or less than the display resolution of the display unit 330. The control unit 310 is a process of generating a virtual image VD including the display image of the display unit 330, performs resolution conversion of converting the display image of the display unit 330 into a lower resolution image, and around the converted image. Another image is added to generate a virtual image VD.

FIG. 11 is a flowchart showing an operation related to the display of the display system 1, and particularly shows the operation of the PC 300. In the present embodiment, the HMD 100 displays an image based on video data output by the PC 300 via the HDMI cable 2A, and the PC 300 performs display control based on the relative position between the image display unit 20 and the PC 300.
Hereinafter, the operation of the display system 1 will be described with reference to FIGS. 8 to 10 and FIG.

  The operation illustrated in FIG. 11 is an operation in which the control unit 310 performs display using the virtual image VD. The control unit 310 executes the operation illustrated in FIG. 11 when the start of the video output to the HMD 100 is instructed by the operation detected by the touch sensor 332 and the display mode using the virtual image VD is designated.

  In the following description, when the display unit 330 is included in the field of view of the image display unit 20, the user wearing the HMD 100 can see the display unit 330 of the PC 300 as an outside scene through the image display unit 20 Point to a case. The field of view of the right eye RE and the left eye LE passing through the image display unit 20, that is, the range of the outside view seen by the user can be determined from the direction of the image display unit 20. In addition, as illustrated in FIG. 4, when the relationship between the angle of view of the camera 61 and the field of view of the right eye RE and the left eye LE when viewing the outside through the image display unit 20 is known, the control unit In 310, it can be determined from the captured image data of the camera 61 whether or not the display unit 330 is in the field of view.

The control unit 310 acquires the captured image data of the camera 61 and the output value of each sensor from the connection device 10 via the USB cable 2B (step S21). In step S21, the control unit 310 acquires at least captured image data and an output value of the distance sensor 64. The detection value acquisition unit 312, for example, executes step S21.
The control unit 310 determines whether the display unit 330 is included in the captured image data of the camera 61 (step S22). In step S22, the control unit 310 detects an image of the display unit 330 from the captured image data by pattern matching, for example.
When it is determined that the display unit 330 is not included in the captured image data (step S22; NO), the control unit 310 returns to step S21.

  When it is determined that the display unit 330 is included in the captured image data (step S22; YES), the control unit 310 detects the position and orientation of the display unit 330 based on the output value of the sensor acquired in step S21 ( Step S23). In step S23, the control unit 310 obtains a relative position between the image display unit 20 and the display unit 330 based on, for example, execution of SLAM (Simultaneous Localization and Mapping) or a detection value of the distance sensor 64. Further, the control unit 310 may detect the reference point P1 of the display unit 330 in the captured image data, and obtain the relative position of the display unit 330 with respect to the image display unit 20 from the position of the detected reference point P1.

  The control unit 310 maps the position and orientation of the display unit 330 in the three-dimensional space according to the position and orientation determined in step S23 (step S24). The position and orientation of the display unit 330 are represented by three-dimensional coordinates with the reference position P2 as the origin or reference point.

  The control unit 310 arranges the virtual image VD in the three-dimensional space in accordance with the position and the direction of the display unit 330 (step S25). The size of the virtual image VD is preset. For example, in the non-volatile storage unit 320, data defining the resolution of the virtual image VD and the position of the virtual image VD to be set in the display unit 330 is stored as setting data (not shown).

  The control unit 310 arranges the virtual image VD in parallel with the display unit 330 and in a plane overlapping the display unit 330. In other words, the virtual image VD is a virtual plane that includes the display unit 330 and is larger than the display unit 330. The control unit 310 obtains three-dimensional coordinates with the reference position P2 as an origin or a reference point, with respect to a reference position that can specify the position of the virtual image VD, such as positions of four corners and centers of the virtual image VD.

  Control unit 310 generates image data of virtual image VD (step S26). The virtual image VD is a screen obtained by virtually enlarging the display unit 330 as described above, and is, for example, a screen forming a user interface of the operating system.

  Control unit 310 determines the range to be displayed by image display unit 20 in virtual image VD (step S27). As shown in FIG. 8, the image display unit 20 displays a part of the virtual image VD. The control unit 310 obtains the range displayed by the image display unit 20 in the virtual image VD by the relative position between the direction of the image display unit 20 and the virtual image VD.

  Control unit 310 specifies the position of display unit 330 in the range in which image display unit 20 displays an image (step S28). The image display unit 20 displays an image on the half mirrors 261 and 281, and the range in which the image is visually recognized overlaps with the visual field VR in the example of FIGS. The control unit 310 specifies the position where the display unit 330 can be seen in the visual field VR when the display unit 330 is viewed through the image display unit 20 in the visual field VR of the user wearing the image display unit 20. This position can be obtained from the position and posture of the PC 300 identified in step S23.

  The control unit 310 cuts out or extracts an image in a range to be displayed on the image display unit 20 from the virtual image VD (step S29), and performs mask processing on the cut out image (step S30). The range in which the mask process is performed in step S30 is a range in which the display unit 330 overlaps the image displayed by the image display unit 20 or a range including a portion overlapping the display unit 330.

  In the example illustrated in FIG. 8, the regions VD1 and VD2 of the virtual image VD are cut out as the range displayed by the image display unit 20. The display unit 330 is viewed at the position where the area VD1 is displayed. In this case, the control unit 310 may mask the entire area VD1 or may perform mask processing only on the area overlapping the display section 330 in the area VD1.

  The user can visually recognize the outside scene passing through the image display unit 20 while the image display unit 20 is displaying the virtual image VD. Here, when the display unit 330 is included in the visual field VR in which the image display unit 20 is visually recognized as an outside view, the user sees the image displayed by the display unit 330 and the image displayed by the image display unit 20 overlapping. . In such a state, in order to provide the user with a better visual experience, in the present embodiment, the display mode of the image display unit 20 is adjusted to improve the visibility of the display unit 330. This process is called mask process.

FIG. 9 shows a state in which the control unit 310 has performed mask processing. In the example of FIG. 9, the mask processing is performed on the entire region VD1 which is a range overlapping the display unit 330. The mask process is, for example, a process of making color data (so-called pixel data, pixel value) of a pixel black or dark. The image display unit 20 displays an image by causing light emitted from the OLED units 221 and 241 to be incident on the right eye RE and the left eye LE. Therefore, when the pixel value is black or dark, the image display unit 20 The amount of incident light is reduced. When the light amount emitted from the image display unit 20 decreases, the light amount of the external light OL transmitted through the image display unit 20 relatively increases, and the visibility of the outside scene is enhanced. Therefore, the user can view the outside scene well in the range in which the mask processing is performed, and the visibility of the display unit 330 as the outside scene is enhanced by the mask processing.
As described above, in the mask processing performed by the control unit 310, the pixel value of the video data to be output to the connection device 10 to be displayed on the image display unit 20 is black (R, G, B = 0) so that the light amount is reduced. , 0, 0) or a preset dark color.

Since the pixel value of the range including the image displayed by the display unit 330 is changed to black or dark in the video data output to the connection device 10 by the mask process, the image displayed by the display unit 330 is substantially displayed. Is removed. As described above, when the range in which the image display unit 20 displays an image overlaps the display unit 330, the control unit 310 outputs video data excluding the image displayed by the display unit 330 to the connection device 10.
In the mask-processed video data, the image of region VD1 which is the region overlapping display unit 330 is removed by the mask, and the image of the region (region VD2 in FIG. 10) located around it is displayed by image display unit 20. Ru.

  The control unit 310 sets the convergence angle of the image subjected to the mask processing (step S31). In step S31, the control unit 310 sets the convergence angle of the image so that a part of the virtual image VD displayed by the image display unit 20 is visually recognized as a plane including the display unit 330. The PC 300 outputs video data including image data of an image displayed by the right display unit 22 of the image display unit 20 and image data of an image displayed by the left display unit 24. The control unit 310 adjusts the position of the virtual image VD in the image for the right display unit 22 and the position of the virtual image VD in the image for the left display unit 24 to allow the left eye LE and the right eye RE of the user. Adjust the convergence angle of

The control unit 310 starts output of the video data to the connection device 10 (step S32). The connection device 10 decodes the video data input to the I / F unit 110 by the display control unit 122, and starts the process of displaying the video data by the right display unit 22 and the left display unit 24.
For example, the image adjusting unit 315 executes steps S24 to S32.

  After the display start, the control unit 310 determines whether the visual field VR has moved (step S33). Here, the movement of the field of view VR means the movement of the relative position of the field of view VR with respect to the virtual image VD. In step S33, the image adjustment unit 315 makes a determination based on the detection value of the position detection unit 313. In addition, the image adjusting unit 315 executes the following steps S34 to S39.

  FIG. 10 shows an example in which the visual field VR has moved. In the example of FIG. 10, the visual field VR is moved upward as compared with the state of FIG. By this movement, the visual fields VR overlap so as to straddle the regions VD1, VD2, VD3, and VD4.

  If the control unit 310 determines that the visual field VR has moved (step S33; YES), the position of the PC 300, more specifically, the position of the display unit 330 is out of the range where the image display unit 20 displays an image. It is determined whether or not (step S34). In step S34, the control unit 310 makes a determination based on the relative positions and directions of the image display unit 20 and the display unit 330.

When it is determined that the position of the display unit 330 is out of the display range of the image display unit 20 (step S34; YES), the control unit 310 cancels the mask processing performed in step S30 (step S35). The control unit 310 changes the range cut out from the virtual image VD according to the moved visual field VR (step S36), and returns to step S33.
In the example illustrated in FIG. 10, the control unit 310 cuts out a range spanning the areas VD1, VD2, VD3, and VD4 from the virtual image VD, generates video data corresponding to the cut image, and outputs the video data to the connection device 10. . Thereby, the image display unit 20 displays the virtual image VD in the range in which the visual fields VR overlap.

When control unit 310 determines that the position of display unit 330 is not outside the display range of image display unit 20 (step S 34; NO), display unit 330 is within the display range of image display unit 20. In this case, control unit 310 determines whether display unit 330 has moved from the outside of the display range of image display unit 20 into the range (step S37).
When the display unit 330 enters the display range from the state where the display unit 330 is out of the display range of the image display unit 20 due to the movement of the visual field VR (step S37; YES), the control unit 310 Is initialized (step S38). In the initialization, the control unit 310 returns the presence / absence of the mask processing in the video data output to the connection device 10, the range cut out from the virtual image VD, and the like to an initial state. The control unit 310 returns to step S23, and executes again the process for starting the output of the video data.

  Specifically, the initialization process in step S38 includes the following example. As a first example, when it is determined in step S37 that the entire display unit 330 has entered the field of view VR, a process of generating a display image using the virtual image VD arranged and processed in step S25 executed previously Can be mentioned. Further, as the second process, there is a process of executing the processes after step S23 according to the position of the display unit 330 and newly setting the position (including the depth) of the virtual image VD. Other than these, the specific example of the process which the control part 310 performs is arbitrary.

  Further, when display unit 330 has not moved from the outside of the display range of image display unit 20 (step S37; NO), that is, the state where display unit 330 is within the display range of image display unit 20 continues. If yes, the controller 310 moves to step S36.

  Further, when it is determined that the visual field VR has not moved (step S33; NO), the control unit 310 determines whether to end the display (step S39). For example, when display end is instructed by an operation or the like on the display unit 330 (step S39; YES), the control unit 310 stops the output of the video data to the connection device 10, and the present process ends. On the other hand, when the display is continued (step S39; NO), the control unit 310 returns to step S33.

  In step S33, when the image display unit 20 moves, for example, when the user wearing the image display unit 20 moves the head, the control unit 310 determines that the visual field VR has moved. Also, even when the relative position between the image display unit 20 and the PC 300 changes due to the movement of the PC 300 with the image display unit 20 not moving, it may similarly be determined that the visual field VR has moved. That is, the movement of the visual field VR detected in step S33 is a change in the relative position between the image display unit 20 and the PC 300, and when the image display unit 20 moves, the PC 300 moves, and And the case where both PC 300 move.

  Therefore, depending on whether the position of the PC 300 is within the display range of the image display unit 20, when the user moves the head or when the user or another person moves the PC 300. The processes after step S34 are performed.

  In the processing illustrated in FIGS. 7 and 11, the display system 1 may detect the distance using captured image data of the camera 61 instead of the distance sensor 64. For example, the display system 1 is data in which the distance from the image display unit 20 to the subject is associated in advance with the size of the subject image captured in the captured image data of the camera 61 in the non-volatile storage unit 130 or the non-volatile storage unit 320. Are stored as reference data. In this case, the control unit 310 calculates the distance from the image display unit 20 to the PC 300 or the display unit 330 based on the size of the image of the PC 300 or the display unit 330 in the captured image data of the camera 61 and the reference data. can do.

  As described above, the display system 1 according to the first embodiment includes the PC 300 having the display unit 330, and the HMD 100 connected to the PC 300. The PC 300 includes an I / F unit 341 that outputs an image. The HMD 100 includes an acquisition unit I / F unit 110 that acquires video data output from the PC 300. Further, the HMD 100 is configured to be able to visually recognize the outside scene, and the image display unit 20 that displays an image superimposed on the outside scene to be viewed in the mounted state of the HMD 100 and the image acquired by the acquiring unit And a display control unit 122 for displaying. The display control unit 122 displays an image on the image display unit 20 in correspondence with the position of the PC 300 visually recognized as the outside scene.

  The PC 300 corresponds to the electronic device of the present invention connected to the HMD 100. The PC 300 includes a display unit 330 and an I / F unit 341 that outputs video data to the HMD 100. The PC 300 further includes a control unit 310 that obtains the relative position of the display unit 330 with respect to the image display unit 20 of the HMD 100. Based on the relative position of the display unit 330 to the image display unit 20, the control unit 310 generates, on the image display unit 20, video data for displaying an image corresponding to the position of the display unit 330 viewed as an outside scene. Output from the I / F unit 341.

  According to the display system 1 to which the present invention is applied and the PC 300, the image output from the PC 300 can be displayed by the HMD 100 so as to correspond to the position of the display unit 330 of the PC 300 as an outside scene. For example, the image can be displayed by the HMD 100 in accordance with the position of the display unit 330 of the PC 300 viewed as an outside scene. As described above, the HMD 100 displaying an image superimposed on the outside scene can perform display in accordance with the display unit 330 of the PC 300 visually recognized as the outside scene.

  Further, control unit 310 generates virtual image VD corresponding to a virtual display area wider than display unit 330. Control unit 310 causes display unit 330 to display a portion of virtual image VD generated corresponding to the virtual display area, and causes I / F unit 341 to output a display image including at least a portion of virtual image VD. . For example, the control unit 310 causes the display unit 330 to display a part of the virtual image VD, and causes the I / F unit 341 to output an image excluding the portion displayed by the display unit 330 in the virtual image VD. Further, the display system 1 causes the image display unit 20 included in the HMD 100 to display an image based on the video data output by the control of the control unit 310 in alignment with the position of the display unit 330 visually recognized as an outside scene. Specifically, video data obtained by masking an area corresponding to a position overlapping the display unit 330 in the image display unit 20 is output from the PC 300 to the connection device 10. Therefore, display corresponding to the virtual image VD wider than the display unit 330 of the PC 300 can be performed on the image display unit 20 of the HMD 100. Therefore, the display unit 330 of the PC 300 can be virtually enlarged by the HMD 100.

  Further, the control unit 310 generates video data excluding the area displayed on the display unit 330 from the virtual image VD and causes the I / F unit 341 to output the generated video data. The control unit 310 causes the image display unit 20 included in the HMD 100 to display the display image output by the control of the control unit 310 around the display unit 330 visually recognized as an outside scene. That is, when the virtual image VD corresponding to the virtual display area wider than the display unit 330 of the PC 300 is displayed by the HMD 100, the display is performed excluding the area displayed on the display unit 330. Therefore, the display of the display unit 330 and the display by the HMD 100 can be combined without overlapping. For example, by displaying an image around the display unit 330 of the PC 300 by the HMD 100, it is possible to realize a display mode in which the display unit 330 is virtually expanded.

  Further, the control unit 310 obtains the relative position of the display unit 330 with respect to the image display unit 20 of the HMD 100, performs mask processing based on the obtained relative position, generates video data, and causes the I / F unit 341 to output. As described above, since the video data is generated to correspond to the relative position of the display unit 330 with respect to the image display unit 20, the display corresponding to the position of the display unit 330 of the PC 300 can be easily realized by the HMD 100.

  Further, the control unit 310 generates a virtual image VD, and determines the position of the virtual image VD based on the position of the PC 300 in the real space. Control unit 310 adjusts the display mode of virtual image VD displayed by image display unit 20 so as to correspond to the relative position between virtual image VD and image display unit 20. Thus, the HMD 100 can display the virtual image VD in accordance with the position in the real space of the display unit 330 of the PC 300. By this virtual image VD, an effect of complementing or expanding the display unit 330 can be obtained.

  In addition, when the control unit 310 detects that the display unit 330 is included in the range viewed as an outside scene in the image display unit 20, the display mode of the virtual image VD is set based on the detected position of the display unit 330. initialize. Thus, the display mode can be adjusted according to whether or not the display unit 330 of the PC 300 can be viewed as an outside scene in the HMD 100. In addition, while the HMD 100 is displaying an image, even if the viewing state of the display unit 330 of the PC 300 changes, it is possible to appropriately cope with it.

  Further, the image display unit 20 is mounted on the head of the user and emits image light toward the left eye LE of the user and emits image light toward the user's right eye RE. A right display unit 22 is provided. The control unit 310 controls the display position by the left display unit 24 and the display position by the right display unit 22 corresponding to the position of the display unit 330 visually recognized as the outside view of the image display unit 20. Adjust the convergence angle of the image displayed on 20. In this case, by adjusting the convergence angle of the image displayed by the HMD 100, the distance at which the display image of the HMD 100 is visually recognized can be made to correspond to the position of the display unit 330 of the PC 300. Thereby, the display by HMD100 and the display of the display part 330 of PC300 can be coordinated more appropriately.

  Note that, in the process of FIG. 11, the display unit 330 may approach the image display unit 20 extremely enough to block the view passing through the image display unit 20. For example, the case where the position of the display unit 330 detected in step S23 is closer to the image display unit 20 than the range set in advance may be mentioned. In this case, the control unit 310 may stop the video output to the connection device 10, and may return to step S21.

In the above embodiment, the control unit 310 adjusts the display position of the image in the image for the right eye RE and the image for the left eye LE to control the convergence angle, and the display image of the image display unit 20 is displayed. On the other hand, the configuration is such that the viewing distance perceived by the user is controlled.
When the image display unit 20 can adjust the visual distance perceived by the user about the display image of the image display unit 20 by the optical system included in the right display unit 22 and the left display unit 24, the adjustment by control of the optical system May be performed instead of control of the convergence angle. That is, the optical system may be controlled such that the user perceives the display image of the image display unit 20 which is a part of the virtual image VD as an image at the same position as the display unit 330. In this case, the distance in which the display image of the image display unit 20 is viewed can be made to correspond to the position of the display unit 330 of the PC 300, and the display by the HMD 100 and the display of the PC 300 can be coordinated more appropriately.

  An example of the above optical system is a configuration provided with a lens movable in the left-right direction connecting the right eye RE and the left eye LE. In this configuration, a lens movable in the left-right direction is disposed in each of the right display unit 22 and the left display unit 24. The lens may be disposed, for example, between the right optical system 251 and the half mirror 261 and between the left optical system 252 and the half mirror 281. In addition, lenses may be disposed inside the right optical system 251 and the left optical system 252. The convergence angle can be changed by moving the position of the image light incident on each of the right eye RE and the left eye LE in the left-right direction in accordance with the position of the lens. In this example, the control unit 120 controls the movement of the lens, and the control unit 310 calculates the control unit 310 by outputting the video data and the control data for specifying the lens position to the connection device 10. A convergence angle can be realized.

  In addition, a holographic display device may be provided as an optical system in which the image display unit 20 outputs image light. In this case, the image display unit 20 includes, for example, a light source and a spatial light modulator (SLM: Spatial Light Modulator) that modulates light from the light source. For example, a reflective spatial light phase modulator (LCOS (Liquid Crystal On Silicon) -SLM) using liquid crystal can be used as the SLM. In this configuration, the connection device 10 outputs display data to the SLM, whereby the reference light emitted from the light source is modulated by the SLM to form a stereoscopic image. In this case, by adjusting the display data generated by the display control unit 122 from the video data output from the PC 300 to the connection device 10 or the video data output from the PC 300, the three-dimensional image formed by the image display unit 20 The focal length can be adjusted. Therefore, instead of the control of adjusting the convergence angle in the present embodiment, the image display unit 20 configured as a holographic display device uses the image displayed by the image display unit 20 by controlling the focal length of the stereoscopic image. It is possible to adjust the distance viewed by the person.

Further, the light control shade may be provided outside the image display unit 20. The light control shade is a plate-like electronic device that includes a liquid crystal panel or the like and whose light transmittance is changed by the control of the control unit 310. When the light control shade is provided in the image display unit 20, the light control shade is transmitted to the right eye RE and the left eye LE of the user wearing the image display unit 20, and the right light guide plate 26 and the left light guide plate 28 are further provided. Outside light OL that has passed through is incident. The user visually recognizes the outside scene by the external light transmitted through the light control shade and the image display unit 20. Therefore, the visibility with which a user visually recognizes an outside scene changes with the transmittance | permeability (light transmittance) of a light control shade.
When performing the mask process, the control unit 310 may perform control to increase the transmittance of the area in the light control shade in order to enhance the visibility of the area to be subjected to the mask process.

  In the first embodiment, the configuration in which the position of the display unit 330 in the display range of the image display unit 20 is specified and the image is displayed on the image display unit 20 in accordance with the position of the display unit 330 has been described. That is, the image is displayed in accordance with the relative position between the image display unit 20 and the display unit 330. The present invention is not limited to this. For example, the position of the PC 300 in the display range of the image display unit 20 may be detected or specified based on part or all of the appearance of the PC 300.

  That is, in step S22, the control unit 310 determines whether at least part of the appearance of the PC 300 is included in the captured image data of the camera 61. In step S22, for example, the control unit 310 detects an image of the PC 300 from the captured image data by pattern matching. Here, the control unit 310 performs pattern matching on the shape of the corner of the case of the PC 300, the shape of the boundary portion between the display unit 330 and the case in the PC 300, the shape on the back side of the PC 300, the paint color on the back side of the PC 300, etc. It may be detected from the image captured by the camera 61. Further, the control unit 310 may detect a marker attached or fixed to the housing of the PC 300 from the captured image of the camera 61. As the marker, for example, an image code such as a barcode and a two-dimensional code, characters, and other images can be used. Moreover, the pattern or shape of the exterior of the PC 300 may be a pattern or shape that functions as a marker. For example, the PC 300 is not limited to the flat plate shape illustrated in FIG. 1, and may have a shape having a protrusion or a recess, and the control unit 310 may detect the protrusion or the recess. In this case, the control unit 310 may specify the position and orientation of the PC 300 from the captured image data, and obtain the relative position of the PC 300 with respect to the image display unit 20. Further, the control unit 310 may perform processing for obtaining the relative position of the PC 300 with respect to the image display unit 20 and obtaining the relative position between the image display unit 20 and the display unit 330 based on the obtained relative position.

[2. Second embodiment]
FIG. 12 is a view showing an example of a display mode in the second embodiment to which the present invention is applied.
The second embodiment is an example in which a virtual image VD different from the first embodiment is formed by the display system 1 having the same configuration as the first embodiment.

  Similar to the virtual image VD illustrated in FIG. 8, the virtual image VD illustrated in FIG. 12 is an image that is larger in size than the image displayed on the display unit 330. In the second embodiment, the control unit 310 arranges the entire image displayed on the display unit 330 in a virtual display area to generate a virtual image VD.

  As shown in FIG. 12, the image displayed on the entire virtual image VD has the same content as the entire image displayed on the display unit 330. If the resolution of the virtual display area does not match the resolution of the image displayed by the display unit 330 or the display resolution of the display unit 330, the control unit 310 performs resolution conversion to generate a virtual image VD.

  The virtual image VD is divided into areas VD1, VD2, VD3, and VD4. A part obtained by dividing the display image of the display unit 330 is allocated to each of the areas VD1 to VD4. The virtual image VD of FIG. 12 divides the display image of the display unit 330 into four, and the divided regions are displayed in the regions VD1, VD2, VD3, and VD4.

  Control unit 310 cuts out a region to be displayed by image display unit 20 from virtual image VD (step S29 in FIG. 11 and step S36 in FIG. 11), and cuts out any of regions VD1 to VD4 as a unit. . That is, video data of one of the regions VD1 to VD4 is output to the connection device 10.

  Then, in the process of determining whether the field of view VR has moved (step S33), the control unit 310 determines that the field of view VR has moved when the field of view VR has exceeded the boundary of the areas VD1 to VD4. In this determination, the control unit 310 makes an affirmative determination if the center of the visual field VR moves beyond the boundaries of the regions VD1 to VD4, ie, if it moves from one of the regions VD1 to VD4 to an adjacent region. Do.

  In the second embodiment, the PC 300 causes the display unit 330 to display an image corresponding to the display unit 330, generates a virtual image VD corresponding to the display image on the display unit 330, and includes at least a portion of the virtual image VD. Output the video data of The display system 1 causes the image display unit 20 included in the HMD 100 to display an image based on the video data output by the control of the control unit 310 in alignment with the position of the display unit 330 visually recognized as the outside scene. Thereby, at least a part of the image displayed on the display unit 330 of the PC 300 can be displayed on the image display unit 20 in alignment with the position of the display unit 330 visually recognized as the outside scene in the image display unit 20. Therefore, the display image of the display unit 330 and the image displayed by the HMD 100 can be coordinated.

  Further, since the control unit 310 causes the I / F unit 341 to output an image obtained by cutting out part of the virtual image VD, a part of the display image of the display unit 330 can be enlarged and displayed on the image display unit 20.

  Further, the control unit 310 obtains the relative position of the display unit 330 with respect to the image display unit 20 of the HMD 100, and generates video data based on the obtained relative position. Thereby, the display corresponding to the position of the display unit 330 of the PC 300 can be easily realized by the HMD 100 including the image display unit 20.

[3. Third embodiment]
FIG. 13 and FIG. 14 are flowcharts showing the operation of the display system 1 in the third embodiment to which the present invention is applied. FIG. 13 shows the operation of the PC 300, and FIG. 14 shows the operation of the HMD 100.
The third embodiment is an example in which an operation different from that of the first embodiment is performed by the display system 1 having the same configuration as the first embodiment.

  In the third embodiment, while the PC 300 executes the process of detecting the relative position of the PC 300 with respect to the image display unit 20, the HMD 100 executes the process of cutting out part of the virtual image VD generated by the PC 300.

In the flowchart shown in FIG. 13, the same processing as that in FIG.
Control unit 310 executes the processing of steps S21 to S28. At step S28, control unit 310 specifies the position of display unit 330 in the range in which image display unit 20 displays an image (step S28). Here, the control unit 310 starts processing of outputting the position data indicating the position of the display unit 330 identified in step S28 to the HMD 100 together with the video data of the virtual image VD (step S51). The position data output by the control unit 310 in step S51 is data for cutting out an image in a range to be displayed on the image display unit 20 from the virtual image VD.

  The control unit 310 determines whether the visual field VR has moved, as in step S32 (FIG. 11) (step S52). In step S52, the image adjustment unit 315 makes a determination based on the detection value of the position detection unit 313. Further, the image adjusting unit 315 executes steps S53 to S57 described below.

  If the control unit 310 determines that the visual field VR has moved (step S52; YES), the control unit 310 determines whether the position of the PC 300 is out of the range where the image display unit 20 displays an image (step S53). In step S53, the control unit 310 may determine whether the position of the display unit 330 is out of the range in which the image display unit 20 displays an image. In step S53, the control unit 310 performs determination based on the relative position and direction of the image display unit 20 and the display unit 330.

  If it is determined that the position of the PC 300 is out of the display range of the image display unit 20 (step S53; YES), the position data to be output to the HMD 100 is changed according to the moved visual field VR (step S54). Return to

When control unit 310 determines that the position of PC 300 is not outside the display range of image display unit 20 (step S53; NO), control unit 310 moves PC 300 from the outside of the display range of image display unit 20 into the range. It is determined whether or not (step S55).
When the display unit 330 enters the display range from the state where the display unit 330 is out of the display range of the image display unit 20 due to the movement of the visual field VR (step S55; YES), the control unit 310 needs to initialize the display of the HMD 100. It is determined that there is (step S56). The control unit 310 returns to step S23, and executes again the process for starting the output of the video data. The process of this initialization is the same as the process described in step S38, and in the operation example shown in FIG. 13 and FIG. 14, the initialization is performed by the image display unit 20.

  If it is determined that the PC 300 has not moved from the outside of the display range of the image display unit 20 (step S55; NO), the control unit 310 proceeds to step S54 and changes the position data.

  In step S52, as in step S33, whether the relative position between the image display unit 20 and the PC 300 has changed, whether the image display unit 20 has moved or the PC 300 has moved, It is determined that the visual field VR has moved.

  If the control unit 310 determines that the visual field VR has not moved (step S52; NO), the control unit 310 determines whether to end the display (step S57). For example, when display end is instructed by an operation on the display unit 330 or the like (step S57; YES), the control unit 310 stops the output of the video data to the connection device 10, and the present process ends. On the other hand, when the display is continued (step S57; NO), control unit 310 returns to step S52.

  In the operation illustrated in FIG. 14, the control unit 120 of the HMD 100 starts the process of acquiring the data of the virtual image VD output by the PC 300 and the position data (step S61). The control unit 120 specifies the position of the display unit 330 in the range in which the image display unit 20 displays an image based on the acquired position data (step S62).

  The control unit 120 executes a process of cutting out an image of a range to be displayed on the image display unit 20 from the virtual image VD (step S63). The control unit 120 performs mask processing on the clipped image (step S64). The range to which the mask process is performed in step S64 is the same as step S30 (FIG. 11), and is a range in which the display unit 330 overlaps the image displayed by the image display unit 20 or a range including a portion overlapping the display unit 330. .

  The control unit 120 sets the convergence angle of the image subjected to the mask processing (step S65). In step S65, the control unit 120 visually recognizes, as a plane including the display unit 330, a part of the virtual image VD displayed by the image display unit 20, similarly to the process performed by the control unit 310 in step S31 (FIG. 11). Set the convergence angle of the image as Control unit 120 generates video data including image data of an image displayed by right display unit 22 of image display unit 20 and image data of an image displayed by left display unit 24 and starts display (Step S66). Here, the control unit 120 adjusts the position of the virtual image VD in the image for the right display unit 22 and the position of the virtual image VD in the image for the left display unit 24 to obtain the left eye LE of the user. The convergence angle of the right eye RE can be adjusted.

  The control unit 120 determines whether the position data output from the PC 300 has been changed (step S67). If the control unit 120 determines that the position data has been changed (step S67; YES), the control unit 120 determines whether the position of the PC 300 is out of the range where the image display unit 20 displays an image (step S68). The process of step S68 is, for example, the same as the process performed by the control unit 310 in step S34 (FIG. 11).

  If it is determined that the position of the PC 300 is out of the display range of the image display unit 20 (step S68; YES), the control unit 120 cancels the mask processing performed in step S64 (step S69). The control unit 120 changes the range to be cut out from the virtual image VD in accordance with the changed position data (step S70), and returns to step S67.

  If control unit 120 determines that the position of PC 300 is not outside the display range of image display unit 20 (step S 68; NO), whether or not PC 300 has moved from outside the display range of image display unit 20 into the range It determines (step S71). When the PC 300 enters the display range from the state where it is out of the display range of the image display unit 20 (step S71; YES), the control unit 120 initializes the display of the image display unit 20 (step S72). The initialization in step S72 is similar to, for example, step S38 (FIG. 11), and the control unit 120 returns the presence / absence of mask processing, the range to be cut out from the virtual image VD, and the like to the initial state. The control unit 120 returns to step S61, and executes again the process for starting the display.

  If the PC 300 has not moved from the outside of the display range of the image display unit 20 into the range (step S71; NO), the control unit 120 moves to step S70.

  When it is determined that the position data is not changed (step S67; NO), the control unit 120 determines whether to end the display (step S73). For example, when display end is instructed by a command input from the PC 300, an operation on the operation unit 140, or the like (step S73; YES), the control unit 120 stops the display by the image display unit 20 and ends this processing. . On the other hand, when the display is continued (step S73; NO), the control unit 120 returns to step S67.

  According to the third embodiment, the PC 300 specifies the relative position of the PC 300 with respect to the image display unit 20, performs processing for obtaining the position of the PC 300 in the display range of the image display unit 20, and the HMD 100 generates an image from the virtual image VD. It is cut out and displayed on the image display unit 20. As described above, by configuring the PC 300 and the HMD 100 to share processing with each other to execute the processing, the weight and size of the HMD 100 can be reduced by simplifying the configuration of the HMD 100. In addition, the image display unit 20 can execute processing such as setting of a convergence angle at high speed by executing processing of cutting out an image from the virtual image VD.

  In the operation described in the third embodiment, the same processing as steps S33 to S38 (FIG. 11) may be performed. In this case, it is preferable that the control unit 310 execute the processes of steps S33, S34, and S37. Moreover, it is preferable that the control part 120 acquires the process result of step S33, S34, S37 which the control part 310 performed, and performs the process of step S35, S36, S38.

[4. Fourth embodiment]
FIG. 15 is a block diagram of each part of the display system 1A according to the fourth embodiment.
In the display system 1A described in the first embodiment, the display system 1A is configured such that the HMD 100 includes a control device 10A instead of the connection device 10. The control device 10A includes a control unit 121, and the control unit 121 processes video data input from the PC 300. The display system 1A includes components similar to those of the display system 1 except for the configuration of the control unit 121 and the related units. The same components as those of the display system 1 are denoted by the same reference numerals, and the description thereof is omitted.

  Similar to the control unit 120 (FIG. 5), the control unit 121 includes a processor (not shown) such as a CPU or a microcomputer, and by executing a program using this processor, the HMD 100 can be implemented by cooperation of software and hardware. Control each part. In addition, the control unit 121 may be configured by programmed hardware.

  The non-volatile storage unit 130, the operation unit 140, and the connection unit 145 are connected to the control unit 121. The control unit 121 also includes a position detection unit 123 and an image adjustment unit 125. The position detection unit 123 detects the position of the HMD 100 in the same manner as the position detection unit 313. More specifically, the position detection unit 123 detects the relative position between the PC 300 and the image display unit 20 based on the output value of the sensor included in the image display unit 20. Also, for example, as described in the first embodiment, the control unit 121 can execute the processing of obtaining the distance from the captured image data of the camera 61 to the subject.

  The relative position detected by the position detection unit 123 may be a position in the space where the image display unit 20 and the PC 300 exist. Also, the relative orientation between the image display unit 20 and the display unit 330 may be included. For example, it may be information indicating the position and / or the direction of the display unit 330 based on the image display unit 20. Further, for example, information indicating the position and / or the direction of the image display unit 20 with reference to the PC 300 may be used. Also, for example, coordinates in a three-dimensional coordinate system set in the space in which the image display unit 20 and the PC 300 exist may be included.

The image adjustment unit 125 performs image processing of video data input to the I / F unit 110. The image processing performed by the image adjustment unit 125 is the same as the processing of the image adjustment unit 315 described in the first embodiment and the second embodiment.
For example, the image adjustment unit 125 is processing for performing resolution conversion (scaling), frame rate conversion, color tone correction, data format change, etc. according to the specification of the HMD 100, for the image data input to the I / F unit 110. .

Similar to the display system 1, the display system 1A executes the calibration process shown in FIG. 7 and the operation shown in FIG.
When the display system 1A performs the operation of FIG. 7, the position detection unit 123 executes steps S13 to S15. When the display system 1A executes the operation of FIG. 11, the position detection unit 123 executes steps S21 and S23, and the image adjustment unit 125 executes steps S22 and S24 to S39.

  In the fourth embodiment, the image adjustment unit 125 sets a virtual display area corresponding to the virtual image VD, arranges video data output from the PC 300 in the virtual display area, and generates the virtual image VD. The image adjustment unit 125 cuts out a range to be displayed by the image display unit 20 from the virtual image VD, performs mask processing as necessary, and causes the image display unit 20 to display an image.

  That is, the HMD 100A determines the relative position of the display unit 330 with respect to the image display unit 20, processes the video data based on the determined relative position, and displays an image corresponding to the position of the display unit 330 of the outside on the image display unit 20. The control unit 121 is provided.

  Thus, the HMD 100A determines the relative position of the display unit 330 with respect to the image display unit 20, and generates a display image to correspond to the relative position, so that the display unit 330 is not increased. The display corresponding to the position of can be realized.

  Further, in a configuration in which the PC 300 mirrors and outputs the same image as the image displayed by the display unit 330 with the HDMI cable 2A, the HMD 100A adjusts the display image of the image display unit 20. That is, the HMD 100A changes the display mode of the image displayed by the image display unit 20 in accordance with the direction of the image display unit 20 and the relative position between the image display unit 20 and the PC 300.

  Therefore, the display system 1A can realize the effects of the display system 1 described in the first embodiment. Furthermore, since video data can be processed in the HMD 100A, when a general-purpose device that outputs video data is connected instead of the PC 300, the display mode can be changed according to the position and direction of the image display unit 20. There is an advantage.

The present invention is not limited to the configurations of the above-described embodiments, and can be implemented in various modes without departing from the scope of the invention.
For example, in the above embodiment, the configuration in which the user passes the display unit to visually recognize the outside view is not limited to the configuration in which the right light guide plate 261 and the left light guide plate 281 transmit outside light. For example, the present invention is also applicable to a display device that displays an image in a state where the outside scene can not be viewed. Specifically, on a display device for displaying a photographed image of the outside scene photographing camera 61, an image or CG generated based on the photographed image, video data stored in advance, video based on video data input from the outside, or video data input from the outside. The present invention can be applied. The display device of this type can include a so-called close display device in which the outside scene can not be viewed. Further, as described in the above embodiment, AR display may be used to display an image superimposed on real space, or MR (Mixed Reality) display may be used in which a photographed real space image and a virtual image are combined. Alternatively, the present invention can be applied to a display device that does not perform processing such as VR (Virtual Reality) display for displaying a virtual image. For example, a display device that displays video data or an analog video signal input from the outside is, of course, included as an application target of the present invention.

  For example, instead of the image display unit 20, an image display unit of another type such as an image display unit worn like a hat may be adopted, for example, and an image is displayed corresponding to the user's left eye And a display unit for displaying an image corresponding to the user's right eye. Further, the display device of the present invention may be configured as, for example, a head mounted display mounted on a vehicle such as an automobile or an airplane. Also, for example, it may be configured as a head mounted display incorporated in a body protector such as a helmet. In this case, the mounting portion may be a portion for positioning the position of the user relative to the body, and a portion positioned relative to the portion.

  Furthermore, in the HMD 100A of the fourth embodiment, the control device 10A and the image display unit 20 may be integrally configured, and may be mounted on the head of the user. Further, as the control device 10A, a notebook computer, a tablet computer, a game machine, a portable telephone, a portable electronic device including a smartphone, a portable media player, or other dedicated device may be used. In the above embodiment, the connection device 10 and the image display unit 20 may be connected by a wireless communication line.

  Moreover, the apparatus connected to HMD100, 100A in the display system 1 is not limited to PC300. For example, a stationary television receiver or a monitor of a stationary personal computer may be used. Moreover, it may replace with PC300 and the projector which projects an image on a display surface may be used, and in this case, the projection surface which a projector projects an image corresponds to a 1st display part. In addition to these devices, portable or stationary electronic devices can be used instead of the PC 300. Various electronic devices used in place of the PC 300 or PC 300 may be wirelessly connected to the HMD 100, 100A. For example, instead of the connector 11A, a wireless video communication interface such as Miracast (registered trademark) or WirelessHD (registered trademark) may be used. Also, instead of the connector 11B, a wireless LAN (including WiFi (registered trademark)) may be used, or Bluetooth (registered trademark) may be used.

  Further, at least a part of each functional block shown in the block diagram may be realized by hardware, or may be realized as a combination of hardware and software, and as shown in the figure, it is independent. The configuration is not limited to the configuration of allocating the hardware resources.

  Reference Signs List 2 cable 2A HDMI cable 2B USB cable 10 connection device 10A control device 11 11A 11B 11C connector 20 image display unit (second display unit) 22 right display Unit (display unit for right eye), 24: left display unit (display unit for left eye), 61: camera, 64: distance sensor, 65: illuminance sensor, 67: LED indicator, 100, 100A: HMD (display device) 110: I / F unit (acquisition unit) 120: control unit 121: control unit 123: position detection unit 124: sensor control unit 125: image adjustment unit 126: power supply control unit 130: non-volatile Storage unit, 140: operation unit, 145: connection unit, 217: temperature sensor, 235: 6-axis sensor, 237: magnetic sensor, 239: temperature sensor, 261: half Reference numeral 281: half mirror 300: PC (electronic device) 310: control unit (electronic device control unit) 311: input / output control unit 312: detection value acquisition unit 313: position detection unit 315: image adjustment Unit 320 Nonvolatile storage unit 321 Content data 330 Display unit (first display unit) 331 Display panel 332 Touch sensor 341 I / F unit (output unit) 345 Communication unit , VD: virtual image, VD1 to VD4: area, VR: field of view.

Claims (15)

An electronic device having a first display unit, and a head-mounted display device connected to the electronic device;
The electronic device is
It has an output unit that outputs an image,
The display device is
An acquisition unit configured to acquire the image output from the electronic device;
A second display unit configured to display an image superimposed on an outside scene viewed in a state in which the display device is mounted;
A display control unit that causes the second display unit to display the image acquired by the acquisition unit;
The display control unit displays the image on the second display unit in association with the position of the electronic device viewed as an outside scene. The electronic device generates an image corresponding to a virtual display area wider than the first display unit, displays a part of the generated image on the first display unit, and generates at least a part of the generated image And an electronic device control unit that causes the output unit to output the information.
The display system according to claim 1, wherein the display control unit included in the display device displays at least a part of the image output by the electronic device on the second display unit in alignment with the position of the electronic device. The electronic device control unit causes the output unit to output the image excluding the portion displayed on the first display unit among the images generated corresponding to the virtual display area.
The display system according to claim 2, wherein the display control unit included in the display device displays an image output by the electronic device on the second display unit.   The display system according to claim 3, wherein the display control unit included in the display device displays the image output by the electronic device around the first display unit visually recognized as an outside scene.   The electronic device control unit displays, on the first display unit, a part of the image generated corresponding to the virtual display area based on the relative position of the first display unit to the second display unit; The display system according to claim 3, wherein the output unit outputs the image excluding a portion displayed on the first display unit.   The electronic device control unit is configured to display a portion of the image generated corresponding to the virtual display area based on the relative position of the first display unit to the second display unit. The display system according to claim 5, wherein the output unit outputs the masked image. The electronic device control unit causes the output unit to output the image generated in association with the virtual display area.
The display system according to claim 2, wherein the display control unit included in the display device displays an image obtained by cutting out a part of the image output by the electronic device on the second display unit.   The display control unit extracts a part of the image acquired by the acquisition unit based on the relative position of the first display unit to the second display unit, and causes the second display unit to display the image. Item 7. A display system according to Item 7.   The display control unit is configured to, based on a relative position of the first display unit with respect to the second display unit, set an image obtained by masking a portion overlapping the first display unit in the image acquired by the acquisition unit. The display system according to claim 7 or 8, wherein the second display unit displays the second display unit.   The electronic device control unit determines the position of the virtual display area based on the position of the electronic device in the real space, and the second display unit based on the relative position of the virtual display area and the second display unit. The display system according to any one of claims 2 to 6, wherein the display mode of is adjusted.   When the electronic device control unit detects that the first display unit is included in a range viewed through the second display unit, the electronic device control unit is configured to use the position of the first display unit as a reference. The display system according to any one of claims 2 to 10, wherein the display mode is initialized. The second display unit is a display unit for the left eye that emits image light toward the left eye of the user wearing the display device, and a display for the right eye that emits image light toward the right eye of the user Equipped with
The display position by the left-eye display unit and the display position by the right-eye display unit are controlled according to the position of the first display unit visually recognized as an outside scene by the second display unit, and the second The display system according to any one of claims 1 to 11, wherein the convergence angle of the image displayed on the display unit is adjusted. The second display unit includes an optical system capable of adjusting a visual distance at which the user perceives an image displayed by the second display unit.
The display system according to claim 12, wherein the optical system is controlled according to a convergence angle of an image displayed on the second display unit. An electronic device connected to a head-mounted display device for displaying an image superimposed on an outside scene,
A first display unit,
An output unit that outputs an image to the display device;
Based on the relative position of the first display unit to the second display unit included in the display device, an image corresponding to the position of the first display unit viewed by the display device as the outside view is displayed on the second display unit An electronic apparatus comprising: a control unit that causes the output unit to output an image to be output. A display method comprising: an electronic device having a first display portion; and a head-mounted display device having a second display portion for displaying an image superimposed on an outside scene,
Output an image by the electronic device;
The display device
Acquiring the image output by the electronic device;
Displaying the acquired image by the second display unit;
A display method, wherein the image is displayed on the second display unit in correspondence with the position of the electronic device viewed as an outside scene in the second display unit.

Images