JP6394174B2 - Head-mounted display device, image display system, method for controlling head-mounted display device, and computer program - Google Patents

Description

  The present invention relates to an image display system.

  There is known a technique called augmented reality (AR) in which information for additional presentation (for example, a virtual object) is additionally presented to a real environment using a computer. Patent Document 1 describes that in an image processing apparatus, a virtual object associated with a real object in a real environment is projected onto the real object.

JP 2013-235374 A JP 2013-141049 A Special table 2012-035768

  The augmented reality described above is realized, for example, in a transmissive head-mounted display device (hereinafter referred to as “HMD”). The transmissive HMD is a type of HMD in which the user's field of vision is not blocked when the HMD is worn. In this case, for example, the HMD captures an outside scene with a camera, recognizes an image obtained by the imaging, and generates a virtual object. The HMD displays only the generated virtual object as a virtual image in front of the user's eyes. The user can experience augmented reality by visually recognizing both the virtual object displayed as a virtual image and the actual outside scene that can be seen through the immediate image display unit.

  By using such augmented reality technology, in a transmissive HMD, an actual outside scene that a user sees through the immediate image display section, which is expressed in a two-dimensional manner (for example, There has been a demand to display objects included in a two-dimensional image on a screen or a display or a two-dimensional representation such as a painting or a poster in a three-dimensional manner. In the techniques described in Patent Documents 1 to 3, no consideration is given to three-dimensional display of objects included in an outside scene expressed two-dimensionally.

  For this reason, in a transmissive HMD (head-mounted display device), an external scene that a user sees through the immediate image display unit, and objects included in the two-dimensionally expressed external scene are tertiary. A technique that can be displayed originally is desired. In addition, HMDs are desired to be downsized, low cost, power saving, easy manufacturing, and improved usability.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to one aspect of the present invention, there is provided a transmissive head-mounted display device that allows a user to visually recognize a virtual image and an outside scene. The head-mounted display device includes: an image display unit that allows the user to visually recognize the virtual image; and an object included in the outside scene that is visible through the image display unit and viewed by the user. And an image adjusting unit that displays at least the virtual image representing the object three-dimensionally on the image display unit. According to the head-mounted display device of this aspect, the image adjustment unit is an object included in the outside scene, and reduces the visibility of the object that the user sees through the image display unit. A three-dimensional virtual image is displayed on the image display unit. In this way, in the transmissive head-mounted display device, it is possible to virtually and three-dimensionally display objects included in the outside scene that the user sees through the image display unit in front of him / her. .

(2) In the head-mounted display device according to the above aspect; the image adjusting unit may reduce the saturation of the object to a position that is transmitted through the image display unit and overlaps the object visually recognized by the user. An adjustment image in which possible colors or colors capable of increasing the brightness of the object are arranged is generated; the virtual image representing the adjustment image is displayed on the image display unit, thereby improving the visibility of the object. It may be lowered. According to this form of the head-mounted display device, the image adjustment unit is a color that can reduce the saturation of the object to a position that overlaps the object that the user sees through the image display unit, or An adjustment image in which a color capable of increasing the brightness of the object is arranged is generated, and a virtual image representing the adjustment image is displayed. Therefore, the user superimposes the object that has passed through the image display unit and the adjustment image in which the color that can reduce the saturation of the object or the color that can increase the brightness of the object is arranged. It will be visible at the same time. As a result, the image adjustment unit can reduce the visibility of an object that is included in the outside scene and that is visible to the user through the image display unit.

(3) In the head-mounted display device according to the above aspect; the image adjustment unit; generates an adjustment image in which a color similar to the object that is visually recognized by the user is transmitted through the image display unit. The visibility of the object may be lowered by displaying the virtual image representing the adjusted image on the image display unit. According to this form of the head-mounted display device, the image adjustment unit generates an adjustment image in which the same color as the object that is visible to the user through the image display unit is arranged, and represents the adjustment image. Display a virtual image. For this reason, the user superimposes the object transmitted through the image display unit and the color similar to the object spreading over the entire field of view at the same time. As a result, the image adjustment unit can reduce the visibility of an object that is included in the outside scene and that is visible to the user through the image display unit.

(4) In the head-mounted display device according to the above aspect; the image adjusting unit is an image representing the outside scene that is transmitted through the image display unit and visually recognized by the user, and the object included in the outside scene Generating an adjustment image in which a part of the image from which the image is excluded is arranged at a position overlapping with the object that is visible to the user through the image display unit; and the virtual image representing the adjustment image is displayed in the image display unit The visibility of the object may be reduced by causing the object to be displayed. According to this form of the head-mounted display device, the image adjustment unit transmits a part of the image representing the outside scene and excluding the object included in the outside scene through the image display unit by the user. An adjustment image arranged at a position overlapping with the object to be visually recognized is generated, and a virtual image representing the adjustment image is displayed. For this reason, the user superimposes the object that has passed through the image display unit and the adjustment image in which the image from which the object included in the outside scene is excluded is superimposed and simultaneously viewed. As a result, the image adjustment unit can reduce the visibility of an object that is included in the outside scene and that is visible to the user through the image display unit.

(5) The head-mounted display device according to the above aspect further includes: a three-dimensional image acquisition unit that acquires a three-dimensional image for representing the object three-dimensionally; and the image adjustment unit includes the three-dimensional image The color may be determined based on: According to the head-mounted display device of this aspect, the image adjusting unit can easily determine the color for the adjusted image based on the acquired three-dimensional image.

(6) In the head-mounted display device according to the above aspect, further; a three-dimensional image acquisition unit that acquires a three-dimensional image for representing the object three-dimensionally; and a two-dimensional image that represents the outside scene two-dimensionally A two-dimensional image acquiring unit that acquires the image; and the image adjusting unit may identify the overlapping position in the adjusted image by comparing the two-dimensional image and the three-dimensional image. According to the head-mounted display device of this aspect, the image adjustment unit easily compares the three-dimensional image acquired from the outside with the two-dimensional image acquired from the outside, thereby easily “overlapping position” in the adjustment image. Can be specified.

(7) In the head-mounted display device according to the above aspect, further: a three-dimensional image acquisition unit that acquires a three-dimensional image for representing the object three-dimensionally; a state in which the head-mounted display device is mounted A two-dimensional image acquisition unit that acquires a two-dimensional image obtained by imaging the user's visual field direction; and the image adjustment unit compares the two-dimensional image with the three-dimensional image to thereby adjust the adjustment. The overlapping position in the image may be specified. According to this form of the head-mounted display device, the image adjustment unit compares the three-dimensional image acquired from the outside with the two-dimensional image obtained by capturing the user's visual field direction, thereby Based on the information on the viewing direction, the “overlapping position” in the adjusted image can be accurately specified.

(8) The image display system including the head-mounted display device of the above aspect further includes: an information processing device; an image display device; the information processing device; A transmission unit that transmits a two-dimensional image including the image and at least transmits a three-dimensional image for representing the object three-dimensionally to the head-mounted display device; the image display device; You may provide the display part which displays the said two-dimensional image transmitted from the apparatus. According to the image display system of this aspect, the object included in the outside scene that the user sees through the image display unit in front of the virtual object is included in the two-dimensional image displayed on the image display device. Therefore, it can be displayed three-dimensionally.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with a new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including some or all of the two elements of the image display unit and the image adjustment unit. That is, this apparatus may or may not have an image display unit. In addition, this apparatus may or may not have an image adjustment unit. Such a device can be realized, for example, as a head-mounted display device, but can also be realized as a device other than the head-mounted display device. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device.

  The present invention can be realized in various modes. For example, a head-mounted display device, a method for controlling the head-mounted display device, an image display system including the head-mounted display device, The present invention can be realized in the form of a method for controlling an image display system, a computer program for realizing the functions of these apparatuses, methods, and systems, a storage medium storing the computer program, and the like.

It is explanatory drawing which shows schematic structure of the image display system in one Embodiment of this invention. 2 is a block diagram functionally showing the configuration of the HMD 100. FIG. It is explanatory drawing which shows an example of the virtual image visually recognized by the user. 3 is a block diagram functionally showing the configuration of a server 300. FIG. It is a sequence diagram which shows the procedure of a display process. It is explanatory drawing for demonstrating a cancellation area | region. It is explanatory drawing about the 1st production | generation method of an adjustment image. It is explanatory drawing about the 2nd generation method of an adjustment image. It is explanatory drawing about the 3rd generation method of an adjustment image. It is explanatory drawing which shows the result of a display process. It is another explanatory drawing which shows the result of a display process. It is explanatory drawing which shows the result of a display process (deformation). It is a block diagram which shows functionally the structure of HMD100a in 2nd Embodiment. It is a flowchart which shows the procedure of the display process in 2nd Embodiment. It is explanatory drawing which shows the result of the display process in 2nd Embodiment. It is explanatory drawing which shows the structure of the external appearance of HMD in a modification.

A. First embodiment:
A-1. Image display system configuration:
FIG. 1 is an explanatory diagram showing a schematic configuration of an image display system according to an embodiment of the present invention. The image display system 1 includes a head-mounted display device 100, a projector 200, a screen 250, and a server 300.

  The head-mounted display device 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The HMD 100 of the present embodiment is an optically transmissive head mounted display that allows a user to visually recognize a virtual image and at the same time directly view an outside scene.

  The projector 200 projects the supplied image on the screen 250. The screen 250 functions as a “display unit”, and the screen 250 and the projector 200 cooperate with each other to function as an “image display device”.

  The server 300 supplies an image to the HMD 100 and the projector 200. Specifically, the server 300 supplies image data of a two-dimensional image to the projector 200. Further, the server 300 supplies the HMD 100 with image data of a three-dimensional image for three-dimensionally representing an arbitrary object included in the two-dimensional image.

  Here, the “object” refers to each object included in the image displayed as a two-dimensional image, or each object present in the outside scene (real world) that the user views through the HMD 100. Means one of them. The “two-dimensional image” means an image that does not make the user feel a stereoscopic effect by displaying the same image (virtual image) in both eyes of the user of the HMD 100, in other words, an image without parallax. The “three-dimensional image” means an image that allows the user to feel a stereoscopic effect by displaying an image with parallax on both eyes of the user of the HMD 100. Note that the three-dimensional image may be realized by a method other than adding parallax. For example, it may be an image that causes the user to feel a stereoscopic effect by tilting the image, or may be an image that causes the user to feel a stereoscopic effect by using a perspective method. Moreover, the image which makes a user feel a three-dimensional effect by the movement stereoscopic vision accompanying the movement of a user's viewpoint may be sufficient. Further, it may be an image that gives a user a stereoscopic effect by adding light or shadow to an object in the image, or an image that gives the user a stereoscopic effect by adjusting the contrast of the image. Also good.

  The HMD 100 is connected to the Internet INT by wireless communication via the communication carrier BS. The projector 200 is connected to the Internet INT by wireless communication via the communication carrier BS. The server 300 is connected to the Internet INT by wired communication. As a result, the server 300 and the head mounted display 100, and the server 300 and the projector 200 are connected to each other via the Internet INT. The communication carrier BS includes a transmission / reception antenna, a radio base station, and an exchange station.

  The HMD 100 of the present embodiment reduces the visibility of an arbitrary object that is included in an outside scene and is viewed by the user through the HMD 100 by executing display processing described later. A virtual image representing the object three-dimensionally can be visually recognized by the user.

A-2. Configuration of head mounted display (HMD):
As shown in FIG. 1, the HMD 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when mounted on the user's head, and a control unit (controller) 10 that controls the image display unit 20. ing. In the following description, a virtual image visually recognized by the user with the HMD 100 is also referred to as a “display image” for convenience. In addition, the HMD 100 emitting the image light generated based on the image data is also referred to as “displaying an image”.

A-2-1. Configuration of image display:
FIG. 2 is a block diagram functionally showing the configuration of the HMD 100. As shown in FIG. 1, the image display unit 20 is a wearing body that is worn on the user's head, and in the present embodiment, has an eyeglass shape. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. And a 9-axis sensor 66. Hereinafter, the positional relationship and function of each unit of the image display unit 20 in a state where the user wears the image display unit 20 will be described.

  As shown in FIG. 1, the right optical image display unit 26 and the left optical image display unit 28 are disposed so as to be positioned in front of the user's right eye and in front of the left eye, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected at a position corresponding to the eyebrow of the user. As shown in FIG. 2, the right optical image display unit 26 includes a right light guide plate 261 and a light control plate (not shown). The right light guide plate 261 is formed of a light transmissive resin material or the like, and guides the image light output from the right display driving unit 22 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The light control plate is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 (the side opposite to the user's eye side). The light control plate protects the light guide plate 261 and suppresses damage to the light guide plate 261 and adhesion of dirt. Further, by adjusting the light transmittance of the light control plate, it is possible to adjust the external light quantity entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The left optical image display unit 28 includes a left light guide plate 262 and a light control plate (not shown). These details are the same as those of the right optical image display unit 26. Note that the right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”. The optical image display unit can use any method as long as it forms a virtual image in front of the user's eyes using image light. For example, the optical image display unit may be realized using a diffraction grating, or a transflective film may be used. It may be realized by using.

  As shown in FIG. 1, the right holding unit 21 extends from the other end ER of the right optical image display unit 26 to a position corresponding to the user's temporal region. The left holding unit 23 is provided to extend from the other end EL of the left optical image display unit 28 to a position corresponding to the user's temporal region. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses. The right holding unit 21 and the left holding unit 23 are also collectively referred to simply as “holding unit”.

  As shown in FIG. 1, the right display driving unit 22 is disposed inside the right holding unit 21 (the side facing the user's head). The left display driving unit 24 is disposed inside the left holding unit 23. As shown in FIG. 2, the right display driving unit 22 functions as a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, and a display element. A right LCD (Liquid Crystal Display) control unit 211, a right LCD 241, and a right projection optical system 251 are provided. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are also collectively referred to as “image light generation unit”. The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED (Light Emitting Diode) or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data1 input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix. The right projection optical system 251 is a collimating lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state.

  The left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, a left LCD control unit 212 and a left that function as a display element. An LCD 242 and a left projection optical system 252 are provided. These details are the same as those of the right display drive unit 22. Note that the right display drive unit 22 and the left display drive unit 24 are collectively referred to simply as a “display drive unit”.

  The camera 61 is disposed at a position corresponding to the right eye corner of the user. The camera 61 captures an outside scene image in the front side direction of the image display unit 20, in other words, an outside scene (external scenery) in the user's viewing direction with the head mounted display 100 attached. The camera 61 is a so-called visible light camera, and the outside scene image acquired by the camera 61 is an image representing the shape of the object from the visible light emitted from the object. The camera 61 in the present embodiment is a monocular camera, but may be a stereo camera. The camera 61 functions as an “imaging unit”.

  The 9-axis sensor 66 is disposed at a position corresponding to the temple on the right side of the user. The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the head, it functions as a “motion detection unit” that detects the movement of the head. Here, the movement of the head includes changes in the speed, acceleration, angular velocity, direction, and direction of the head.

  As shown in FIG. 1, the image display unit 20 includes a connection unit 40 for connecting the image display unit 20 and the control unit 10. The connection unit 40 includes a main body cord 48 connected to the control unit 10, a right cord 42 and a left cord 44 branched from the main body cord 48, and a connecting member 46 provided at the branch point. The right cord 42 is connected to the right display driving unit 22, and the left cord 44 is connected to the left display driving unit 24. The connecting member 46 is provided with a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30. A connector (not shown) is provided at the end of the main body cord 48 opposite to the connecting member 46. This connector realizes connection / disconnection between the control unit 10 and the image display unit 20 by fitting / releasing with a connector (not shown) provided in the control unit 10. The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

A-2-2. Configuration of control unit:
As shown in FIG. 1, the control unit 10 is a device for controlling the HMD 100. The control unit 10 includes a determination key 11, a lighting unit 12, a display switching key 13, a track pad 14, a luminance switching key 15, a direction key 16, a menu key 17, and a power switch 18. Yes. The determination key 11 detects a pressing operation and outputs a signal for determining the content operated in the control unit 10. The lighting unit 12 is realized by, for example, an LED, and notifies the operation state of the HMD 100 (for example, ON / OFF of the power source) by the light emission state. The display switching key 13 detects a pressing operation, and outputs a signal for switching the display mode of the content moving image between 3D and 2D, for example. The track pad 14 detects the operation of the user's finger on the operation surface of the track pad 14 and outputs a signal corresponding to the detected content. As the track pad 14, various methods such as an electrostatic method, a pressure detection method, and an optical method can be adopted. The luminance switching key 15 detects a pressing operation and outputs a signal for increasing or decreasing the luminance of the image display unit 20. The direction key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the HMD 100 by detecting a slide operation of the switch.

  2, the control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a wireless communication unit 132, a GPS module 134, a CPU 140, an interface 180, and a transmission unit (Tx ) 51 and 52, and the respective parts are connected to each other by a bus (not shown).

  The input information acquisition unit 110 acquires signals corresponding to operation inputs to the enter key 11, the display switching key 13, the track pad 14, the luminance switching key 15, the direction key 16, the menu key 17, and the power switch 18. The input information acquisition unit 110 can acquire operation inputs using various methods other than those described above. For example, an operation input using a foot switch (a switch operated by a user's foot) may be acquired. For example, an operation input based on a user's line of sight detected by a line-of-sight detection unit (not shown) or a command associated with eye movement may be acquired. This command may be set to be addable by the user. For example, a user's gesture may be detected using a camera that captures an image in the user's field of view, and an operation input by a command associated with the gesture may be acquired. When detecting a gesture, a user's fingertip, a ring attached to the user's hand, a medical instrument held by the user's hand, or the like can be used as a mark for motion detection. If it is possible to acquire an operation input using a foot switch or a line of sight, the input information acquisition unit 110 can acquire an operation input from the user even in a task that is difficult for the user to release.

  The storage unit 120 includes a ROM, a RAM, a DRAM, a hard disk, and the like. The storage unit 120 stores various computer programs including an operating system (OS).

  The power supply 130 supplies power to each part of the HMD 100. As the power supply 130, for example, a secondary battery can be used.

  The wireless communication unit 132 performs wireless communication with an external device in accordance with a predetermined wireless communication standard. The predetermined wireless communication standard is, for example, infrared, short-range wireless communication exemplified by Bluetooth (registered trademark), wireless LAN exemplified by IEEE 802.11, or the like.

  The GPS module 134 detects the current position of the user of the HMD 100 by receiving a signal from a GPS satellite, and generates current position information representing the current position information of the user. The current position information can be realized by coordinates representing latitude and longitude, for example.

  The CPU 140 reads and executes a computer program stored in the storage unit 120, thereby executing a 2D image acquisition unit 142, a 3D image acquisition unit 144, an image adjustment unit 146, an OS 150, an image processing unit 160, and an audio processing unit. 170 functions as the display control unit 190.

  The two-dimensional image acquisition unit 142 acquires a two-dimensional image received by the HMD 100 from the server 300 in a display process described later. The three-dimensional image acquisition unit 144 acquires a three-dimensional image received by the HMD 100 from the server 300 in the display process described later.

  The image adjustment unit 146 performs display processing described later. The display process is an arbitrary object included in the outside scene, and reduces the visibility of an arbitrary object that the user views through the image display unit 20 of the HMD 100, and represents the object three-dimensionally. This is a process for forming a virtual image. Details will be described later.

  The image processing unit 160 generates a signal based on content (video) input via the interface 180 or the wireless communication unit 132. For example, when the content is in a digital format, the image processing unit 160 generates a clock signal PCLK and image data Data. In the case of the digital format, since the clock signal PCLK is output in synchronization with the image signal, generation of the vertical synchronization signal VSync and the horizontal synchronization signal HSync and A / D conversion of the analog image signal are unnecessary. The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 to the image via the transmission units 51 and 52. Transmit to the display unit 20. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data Data1”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data Data2”. Note that the image processing unit 160 performs image processing such as resolution conversion processing, various tone correction processing such as adjustment of luminance and saturation, and keystone correction processing on the image data Data stored in the storage unit 120. May be.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 turns on and off the left and right LCDs 241 and 242 by the left and right LCD control units 211 and 212 and the left and right backlights by the left and right backlight control units 201 and 202 according to the control signal. The generation and emission of image light by the right display drive unit 22 and the left display drive unit 24 are controlled by individually controlling the driving ON / OFF of the 221 and 222, respectively. The display control unit 190 transmits these control signals to the image display unit 20 via the transmission units 51 and 52.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and supplies the acquired audio signal to a speaker (not shown) of the right earphone 32 and a speaker (not shown) of the left earphone 34.

  The interface 180 communicates with the external device OA in accordance with a predetermined wired communication standard. The predetermined wired communication standards include, for example, MicroUSB (Universal Serial Bus), USB, HDMI (High Definition Multimedia Interface, HDMI is a registered trademark), DVI (Digital Visual Interface), VGA (Video Graphics Array), Composite, RS- The wired LAN is exemplified by 232C (Recommended Standard 232) and IEEE802.3. As the external device OA, for example, a personal computer PC, a mobile phone terminal, a game terminal, or the like can be used.

  FIG. 3 is an explanatory diagram illustrating an example of a virtual image visually recognized by the user. As described above, the image light guided to both eyes of the user of the HMD 100 forms an image on the retina of the user, so that the user visually recognizes the virtual image VI. In the illustrated example, the virtual image VI is a standby screen of the OS 150 of the HMD 100. Further, the user views the outside scene SC through the right optical image display unit 26 and the left optical image display unit 28. Thus, the user of the HMD 100 of the present embodiment can see the virtual image VI and the outside scene SC behind the virtual image VI for the portion of the visual field VR where the virtual image VI is displayed. In addition, the portion of the visual field VR where the virtual image VI is not displayed can be seen through the optical scene display portion and directly through the outside scene SC.

A-3. Server configuration:
FIG. 4 is a block diagram functionally showing the configuration of the server 300. The server 300 includes a CPU 310, a ROM 320, a RAM 330, a communication interface (I / F) 340, and a storage unit 350, and the respective units are connected to each other via a bus (not shown). The server 300 functions as an “information processing apparatus”.

  The CPU 310 controls each unit of the server 300 by developing computer programs stored in the ROM 320 and the storage unit 350 in the RAM 330 and executing them. In addition, the CPU 310 also functions as the transmission unit 312.

  The communication interface 340 includes a wireless communication interface and a wired communication interface. The wireless communication interface includes a transmission / reception circuit (not shown), and performs demodulation and generation of data received via the antenna, and generation and modulation of radio wave transmitted via the antenna. The wired communication interface is connected to another device via a wired cable.

  The storage unit 350 includes a ROM, RAM, DRAM, hard disk, and the like. The storage unit 350 includes an image storage unit 352. The image storage unit 352 stores “2D image data” for a two-dimensional image and “partial 3D image data” for a three-dimensional image in association with each other. The two-dimensional image stored in the same row (entry) and the three-dimensional image have a correspondence relationship. Specifically, the three-dimensional image is an image for three-dimensionally representing an arbitrary object included in the two-dimensional image. For example, when 2Ddata1 that is a two-dimensional image is “an image representing a landscape where an apple is falling in a forest”, 3Ddata1 that is a three-dimensional image is an “image for representing an apple three-dimensionally”. . In this example, “apple” corresponds to an arbitrary object.

  The transmission unit 312 of the server 300 transmits 2D image data and partial 3D image data corresponding to the 2D image data to the HMD 100 in a display process described later. In addition, the transmission unit 312 transmits the same 2D image data transmitted to the HMD 100 to the projector 200.

A-4. Display processing:
FIG. 5 is a sequence diagram showing the procedure of the display process. The display process of the present embodiment is executed by the cooperation of the HMD 100, the projector 200, and the server 300. The start trigger of the display process can be arbitrarily determined. For example, the start trigger may be that the HMD 100 is activated, in other words, that the power is turned on. Further, for example, a process start request from the OS 150 of the HMD 100 or a specific application may be used as a start trigger.

  In step S <b> 102, the transmission unit 312 of the server 300 transmits 2D image data to the projector 200. In step S <b> 104, the projector 200 projects an image based on the received image data on the screen 250.

  In step S <b> 106, the transmission unit 312 of the server 300 transmits 2D image data and partial 3D image data corresponding to the 2D image data to the HMD 100. 4, the 2D image data transmitted from the server 300 to the HMD 100 in step S106 is the same image data as the 2D image data transmitted from the server 300 to the projector 200 in step S102. In step S106, the 2D image acquisition unit 142 of the HMD 100 acquires 2D image data, and the 3D image acquisition unit 144 acquires partial 3D image data.

  FIG. 6 is an explanatory diagram for explaining the cancel region. In step S108 in FIG. 5, the image adjustment unit 146 of the HMD 100 identifies the cancel area A2 (FIG. 6). Here, the cancel area A2 is an arbitrary area in the outside scene SC (FIG. 3) that the user transmits through the image display section 20 in the entire area where the virtual image VI (FIG. 3) is formed in the image display section 20. This is the position where the object is visually recognized. The cancel area A2 functions as an “overlapping position”.

  The entire area where the virtual image VI is formed in the image display section 20 is the maximum area (hereinafter also referred to as “maximum emission area A1”) from which the display drive section (FIG. 2) can emit image light. When the display drive unit is configured by a backlight and an LCD as in the present embodiment, the maximum emission area A1 is the entire area where the liquid crystal is arranged in the right LCD 241 and the left LCD 242, and the right backlight 221 and The left backlight 222 is defined as an area where illumination light can be irradiated. In the example of FIG. 6, the maximum emission area A1 starts from the coordinates (0, 0) and ends at the coordinates (n, m).

As specific procedures for specifying the cancel area A2 (FIG. 6), for example, the following procedures a1 to a3 can be adopted.
(A1) The image adjustment unit 146 temporarily arranges the 2D image data acquired by the two-dimensional image acquisition unit 142 in the maximum emission area A1. At the time of arrangement, the image adjustment unit 146 expands and contracts the size of the 2D image data according to the size of the maximum emission area A1. Depending on the size of the screen 250, the user of the HMD 100 may permeate the image display unit 20 and view only an image of a partial area of the image displayed on the screen 250. . In such a case, the image adjustment unit 146 obtains an area visually recognized by the user of the HMD 100 based on an outside scene image obtained by the camera 61, for example, and obtains the area obtained from the acquired 2D image data. The image data may be cut out.

(A2) The image adjustment unit 146 processes the partial 3D image data acquired by the 3D image acquisition unit 144 to generate comparative 3D image data. For example, when the 3D implementation method employed in the HMD 100 is the side-by-side method, there are two pieces of partial 3D image data for the right eye and for the left eye, in which the position of the object is shifted left and right (horizontal direction). Therefore, the image adjustment unit 146 generates one piece of comparison 3D image data in which an object is arranged at a position that the user will visually recognize from the two pieces of partial 3D image data. Further, when the 2D image data is expanded and contracted in the procedure a1, the image adjusting unit 146 expands and contracts the comparison 3D image data at the same rate. Note that the image adjustment unit 146 may use the partial 3D image data acquired by the three-dimensional image acquisition unit 144 as it is (without processing) as comparison 3D image data if not necessary.

(A3) The image adjustment unit 146 performs image recognition (comparison) using a two-dimensional image (an image based on 2D image data) arranged in the maximum emission area A1 and a three-dimensional image based on comparison 3D image data. ) And a region where the two-dimensional image and the three-dimensional image coincide with each other is set as a “cancel region A2”.

  In the example of FIG. 6, the cancel area A2 starts from the coordinates (x1, y1) and ends at the coordinates (x2, y2). In FIG. 6, the cancel area A2 is indicated by a broken line. In this way, the image adjustment unit 146 performs a three-dimensional image (partial 3D image data, comparison 3D image data) acquired from the server 300 (external) and a two-dimensional image (2D acquired from the server 300 (external)). Image data), the cancel area A2 (overlapping position) in the adjusted image AD can be easily specified.

  In FIG. 6, the cancel area A2 has the same rectangular shape as the partial 3D image data acquired by the three-dimensional image acquisition unit 144, but the cancel area A2 is an object (example in the figure) included in the partial 3D image data. Then, it may be in the shape of an apple).

  In step S110 of FIG. 5, the image adjustment unit 146 of the HMD 100 generates an “adjusted image” for reducing the visibility of the two-dimensional transmission image in the specified cancellation area A2. Here, the two-dimensional transmission image means an outside scene SC (FIG. 3) that the user of the HMD 100 sees through the optical image display unit. That is, the adjustment image is an image used for making it difficult for the user to see the outside scene SC in the area of the cancellation area A2. The image adjustment unit 146 can generate an adjusted image using any one of the following first to third generation methods.

  FIG. 7 is an explanatory diagram of a first adjustment image generation method. The image adjustment unit 146 can lower the saturation of the object included in the two-dimensional transmission image (outside scene SC) with respect to the cancel area A2 that starts from the coordinates (x1, y1) and ends at the coordinates (x2, y2). An adjustment image AD is generated by arranging a color or a color that can increase the brightness of an object and arranging black data for other regions. The image adjustment unit 146 is a three-dimensional image (partial) for representing the object three-dimensionally by “color that can lower the saturation of the object” or “color that can increase the brightness of the object”. 3D image data). Note that the image adjustment unit 146 may set the Enable signal for switching between valid / invalid of image light generation in the display drive unit of the HMD 100 as a Lo value (a value indicating invalidity) instead of arranging black data. .

  According to the second method of generating the adjusted image, the image adjusting unit 146 can reduce the saturation of the object to the cancel area A2 (a position that overlaps with the object that can be seen by the user through the image display unit 20). An adjustment image AD in which possible colors or colors capable of increasing the brightness of an object are arranged is generated. Then, the image adjustment unit 146 displays a virtual image representing the adjustment image AD in step S112 described later. For this reason, the user superimposes the object that has passed through the image display unit 20 and the adjustment image AD in which a color that can reduce the saturation of the object or a color that can increase the brightness of the object is arranged. And will be visible at the same time. As a result, the image adjustment unit 146 can reduce the visibility of an object that is included in the outside scene and that is visible to the user through the image display unit 20.

  FIG. 8 is an explanatory diagram of a second adjustment image generation method. The image adjustment unit 146 arranges a color similar to the object included in the two-dimensional transmission image (outside scene SC) with respect to the maximum emission area A1 that starts from the coordinates (0, 0) and ends at the coordinates (n, m). Thus, the adjusted image AD is generated. When an object is composed of a plurality of colors, the “similar color to the object” may be, for example, a color that is an average value of RGB values of the colors constituting the object, or the RGB values of the colors constituting the object It is good also as a color which is a representative value of. The image adjustment unit 146 can determine “color similar to the object” based on a three-dimensional image (partial 3D image data) for representing the object three-dimensionally. Note that the image adjustment unit 146 may determine “a color similar to the object” based on the outside scene image obtained by the camera 61. Then, the color can be determined in consideration of ambient light.

  According to the first method for generating an adjusted image, the image adjusting unit 146 arranges a color similar to the object that is transmitted through the image display unit 20 and visually recognized by the user in the maximum emission area A1 (entire). Is generated. Then, the image adjustment unit 146 displays a virtual image representing the adjustment image AD in step S112 described later. For this reason, the user superimposes the object transmitted through the image display unit 20 and the color similar to the object spreading over the entire field of view at the same time. As a result, the image adjustment unit 146 can reduce the visibility of an object that is included in the outside scene and that is visible to the user through the image display unit 20.

  FIG. 9 is an explanatory diagram of a third adjustment image generation method. The two-dimensional image acquisition unit 142 further acquires a background two-dimensional image BD from the server 300. The background two-dimensional image BD is an image obtained by excluding the object (apple in the example in the figure) included in the corresponding partial 3D image data from the image represented by the 2D image data transmitted by the server 300 in step S106 of FIG. It is. The image adjustment unit 146 starts from a part of the image corresponding to the cancel area A2 from the background two-dimensional image BD, that is, the coordinates (x1, y1) specified in step S108 in FIG. Extract the image of the area that ends with. The image adjusting unit 146 places the extracted image in the cancel area A2 starting from the coordinates (x1, y1) and ending at the coordinates (x2, y2), and arranging black data in the other areas. Then, the adjusted image AD is generated. Note that the image adjustment unit 146 may set the Enable signal for switching between valid / invalid of image light generation in the display drive unit of the HMD 100 as a Lo value (a value indicating invalidity) instead of arranging black data. .

  Note that the two-dimensional image acquisition unit 142 may generate the background two-dimensional image BD by itself instead of acquiring the background two-dimensional image BD from the server 300. For example, the 2D image acquisition unit 142 performs 2D image data for the background by performing image processing for masking an object included in the corresponding partial 3D image data on the 2D image data acquired from the server 300. An image BD can be generated. In the image processing for masking, for example, an object in the image may be painted black or white, or the object may be erased using an image around the object.

  According to the third adjustment image generation method, the image adjustment unit 146 cancels a part of the background two-dimensional image BD (an image representing an outside scene and excluding objects included in the outside scene). An adjustment image AD arranged in the area A2 (a position that overlaps with an object that can be seen by the user through the image display unit 20) is generated. Then, the image adjustment unit 146 displays a virtual image representing the adjustment image AD in step S112 described later. For this reason, the user superimposes the object that has been transmitted through the image display unit 20 and the adjustment image AD on which the image from which the object included in the outside scene is excluded is superimposed and viewed simultaneously. As a result, the image adjustment unit 146 can reduce the visibility of an object that is included in the outside scene and that is visible to the user through the image display unit 20.

  In step S112 of FIG. 5, the image adjustment unit 146 of the HMD 100 displays both the adjustment image AD generated in step S110 and the partial 3D image acquired by the three-dimensional image acquisition unit 144. Specifically, the image adjustment unit 146 transmits the image data of the adjustment image AD and the partial 3D image data to the image processing unit 160. The image processing unit 160 that has received these image data performs the display process of FIG. 2 on the image data obtained by combining the image data of the adjustment image AD and the partial 3D image data.

  In addition, when corresponding to a moving image, the HMD 100, the projector 200, and the server 300 may repeatedly execute the processes in steps S102 to S112 described above.

  FIG. 10 is an explanatory diagram showing the result of the display process. As a result of step S112 of the display process, the user U1 of the HMD 100 transmits the two-dimensional image D2 represented by the 2D image data displayed on the screen 250 through the optical image display unit of the HMD 100 in the visual field VR1 as the outside scene SC. Visually check. Furthermore, the user U1 of the HMD 100 visually recognizes the virtual image VI representing the adjustment image AD that is generated and displayed based on any one of the first to third generation methods. As described above, the user U1 of the HMD 100 visually recognizes the adjustment image AD, thereby reducing the visibility of the two-dimensional object (apple in the example illustrated) displayed on the screen 250. For this reason, the user U1 of the HMD 100 can more clearly recognize the virtual image VI of the object (apple) displayed three-dimensionally by the partial 3D image data. On the other hand, the user U2 who is not a user of the HMD 100 views the two-dimensional image D2 represented by the 2D image data displayed on the screen 250 in the visual field VR2.

  FIG. 11 is another explanatory diagram showing the result of the display process. In the display process described above, the cancel area A2 surrounded by the broken line and the object OBJ displayed three-dimensionally by the partial 3D image data are at the same position. This is because the image adjustment unit 146 determines the cancel area A2 based on the 2D image data acquired from the server 300 in step S108 of the display process.

A-5. Transformation of display processing:
Hereinafter, a configuration for changing the position of the cancel area A2 and the object OBJ displayed three-dimensionally by the partial 3D image data will be described. Hereinafter, only portions different from the display processing of FIG. 5 will be described.

  In step S <b> 106, the two-dimensional image acquisition unit 142 of the HMD 100 instructs the camera 61 to perform imaging, and acquires an outside scene image in the viewing direction of the user of the HMD 100. In this modification, the outside scene image captured by the camera 61 functions as a “two-dimensional image”. In this modification, transmission of 2D image data from the server 300 to the HMD 100 can be omitted.

  Steps S108 to S112 are the same as the steps in FIG. However, the description “2D image data” in the description is read as “the outside scene image acquired in step S106”.

  According to step S108 of the display process (deformation), the image adjustment unit 146 captures the three-dimensional image (partial 3D image data, comparison 3D image data) acquired from the server 300 (external) and the viewing direction of the user. By comparing the two-dimensional image, that is, the outside scene image, the cancel area A2 (overlapping position) in the adjustment image AD can be accurately specified based on the information on the visual field direction of the actual user.

  FIG. 12 is an explanatory diagram showing the result of display processing (deformation). In the modification of the display process described above, the cancel area A2 surrounded by the broken line and the object OBJ displayed three-dimensionally by the partial 3D image data are at different positions. In step S108 of the display process (deformation), the image adjustment unit 146 cancels based on the outside scene image in the viewing direction of the user U1 (that is, the image of the outside scene that the user U1 actually sees). This is because the area A2 is determined.

  As described above, according to the display processing of the first embodiment, the image adjustment unit 146 of the HMD 100 (head-mounted display device) is an arbitrary object included in the outside scene SC and is transmitted through the image display unit 20. As a result, the visibility of the object visually recognized by the user U1 is reduced, and a virtual image VI that represents the object three-dimensionally is displayed on the image display unit 20. In this way, in the transmissive HMD 100, an arbitrary object included in the outside scene SC viewed through the image display unit (optical image display unit) in front of the user U1 is virtually and three-dimensionally displayed. Can be displayed.

B. Second embodiment:
In the second embodiment of the present invention, a configuration in which the same processing as that of the first embodiment is performed by a head-mounted display device (HMD) alone will be described. Below, only the part which has a different structure and operation | movement from 1st Embodiment is demonstrated. In the figure, the same components as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment described above, and detailed description thereof is omitted.

B-1. Image display system configuration:
In the second embodiment, an HMD 100a is provided instead of the HMD 100 shown in FIG. Furthermore, in 2nd Embodiment, the projector 200, the screen 250, and the server 300 are not provided.

B-2. Configuration of head mounted display (HMD):
FIG. 13 is a block diagram functionally showing the configuration of the HMD 100a in the second embodiment. The difference from the first embodiment shown in FIG. 2 is that a control unit 10 a is provided instead of the control unit 10.

B-2-1. Configuration of image display:
The configuration of the image display unit 20 is the same as that of the first embodiment shown in FIGS.

B-2-2. Configuration of control unit:
The control unit 10a includes a 2D image acquisition unit 142a instead of the 2D image acquisition unit 142, includes a 3D image acquisition unit 144a instead of the 3D image acquisition unit 144, and performs image adjustment instead of the image adjustment unit 146. Part 146a. The two-dimensional image acquisition unit 142a, the three-dimensional image acquisition unit 144a, and the image adjustment unit 146a are different from the first embodiment in the processing content in the display process. Details will be described later.

  The control unit 10 a further includes a partial 3D image data storage unit 122 in the storage unit 120. The partial 3D image data storage unit 122 is a set of three-dimensional image data for three-dimensionally expressing an object included in a two-dimensionally expressed outside scene (for example, a two-dimensional representation such as a painting or a poster). It is. Each data in the partial 3D image data storage unit 122 uses the 3D model space, the movement of the user's head obtained by the 9-axis sensor 66, and the current position information obtained by the GPS module 134. It may be generated dynamically.

B-3. Display processing:
FIG. 14 is a flowchart illustrating a procedure of display processing according to the second embodiment. The display process of the second embodiment is executed by the HMD 100a. Similar to the first embodiment described with reference to FIG. 5, the display processing start trigger can be arbitrarily determined.

  In step S202, the two-dimensional image acquisition unit 142a instructs the camera 61 to capture an image, and acquires an outside scene image in the visual field direction of the user of the HMD 100a. In the second embodiment, an outside scene image captured by the camera 61 functions as a “two-dimensional image”.

  In step S204, the 3D image acquisition unit 144a searches the partial 3D image data storage unit 122 for partial 3D image data that matches the outside scene image acquired in step S202, and acquires the partial 3D image data. Specifically, the three-dimensional image acquisition unit 144a performs image analysis on the outside scene image, searches the partial 3D image data storage unit 122 for partial 3D image data for representing the object included in the outside scene image three-dimensionally, get.

  In step S206, the image adjustment unit 146a identifies the cancel area A2 (FIG. 6). The details are the same as step S108 in FIG. However, the description “2D image data” in the description is read as “the outside scene image acquired in step S202”.

  In step S208, the image adjustment unit 146a generates an adjustment image AD (FIGS. 7 to 9). Details are the same as step S110 in FIG. However, the description “2D image data” in the description is read as “the outside scene image acquired in step S202”.

  In step S210, the image adjustment unit 146a displays both the adjustment image AD generated in step S208 and the partial 3D image data acquired by the three-dimensional image acquisition unit 144a. Details are the same as step S112 of FIG. However, the description “2D image data” in the description is read as “the outside scene image acquired in step S202”.

  It should be noted that the HMD 100a may repeatedly execute the above-described steps S202 to S210 every time the user's movement is detected by the 9-axis sensor 66.

  FIG. 15 is an explanatory diagram illustrating a result of display processing in the second embodiment. As a result of the display processing step S210, the user U1 of the HMD 100a permeates through the optical image display unit of the HMD 100a in the visual field VR1 and visually recognizes the outside scene SC in the real environment. Furthermore, the user U1 of the HMD 100a visually recognizes the virtual image VI representing the adjustment image AD that is generated and displayed based on any one of the first to third generation methods. As described above, the user U1 of the HMD 100a visually recognizes the adjustment image AD, thereby reducing the visibility of an object (an apple in the illustrated example) that exists in the real environment. For this reason, the user U1 of the HMD 100a can visually recognize the virtual image VI of the object (apple) displayed three-dimensionally by the partial 3D image data more clearly.

  As described above, according to the display process of the second embodiment, the same effect as that of the first embodiment can be obtained in a configuration that does not include other devices (image display device, server).

C. Variation:
In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. In addition, the following modifications are possible.

・ Modification 1:
In the said embodiment, it illustrated about the structure of the image display system. However, the configuration of the image display system can be arbitrarily determined without departing from the gist of the present invention, and for example, addition / deletion / conversion of components can be performed.

  For example, the arrangement of the functional units in the image display system can be arbitrarily changed. For example, at least a part or all of the functional units described to be included in the server may be included in the HMD or may be included in the projector. Similarly, at least a part or all of the functional units described as being provided in the HMD may be provided in the server or may be provided in the projector.

  For example, the image display system includes an HMD, a projector and a screen, and a server. However, the components of the image display system are merely examples, and can be arbitrarily changed. For example, the image display system may include a display instead of the projector and the screen or together with the projector and the screen. In this case, the display functions as an “image display device”. In addition, for example, the image display system may include other components not illustrated (for example, a network device such as a router or an access point). In the image display system, the server is not limited to a mode connected via the Internet. For example, the server may be arranged in the same LAN as the HMD or projector.

  For example, the transmission unit included in the server is described as being realized by the CPU developing and executing a computer program stored in the ROM or hard disk on the RAM. However, the functional unit may be configured using an ASIC (Application Specific Integrated Circuit) designed to realize the function. For example, various changes can be made to the image storage unit of the server. For example, the table format as illustrated may not be adopted. Moreover, you may add, delete, and change a field with respect to the illustrated table.

Modification 2
In the said embodiment, it illustrated about the structure of HMD. However, the configuration of the HMD can be arbitrarily determined without departing from the gist of the present invention. For example, each component can be added, deleted, converted, and the like.

  The allocation of components to the control unit and the image display unit in the above embodiment is merely an example, and various aspects can be employed. For example, the following aspects may be adopted. (I) A mode in which processing functions such as a CPU and a memory are mounted on the control unit, and only a display function is mounted on the image display unit. (Ii) Processing functions such as a CPU and a memory are provided in both the control unit and the image display unit. (Iii) a mode in which the control unit and the image display unit are integrated (for example, a mode in which the control unit is included in the image display unit and functions as a glasses-type wearable computer), (iv) instead of the control unit A mode in which a smartphone or a portable game machine is used. (V) The control unit and the image display unit are connected by connection via a wireless signal transmission path such as a wireless LAN, infrared communication, or Bluetooth, and the connection unit (code) is connected. Abolished aspect. In this case, power supply to the control unit or the image display unit may be performed wirelessly.

  For example, the configurations of the control unit and the image display unit exemplified in the above embodiment can be arbitrarily changed. Specifically, for example, in the above embodiment, the control unit includes the transmission unit and the image display unit includes the reception unit. However, both the transmission unit and the reception unit have a function capable of bidirectional communication. It may be provided and may function as a transmission / reception unit. Further, for example, a part of an operation interface (such as various keys and a trackpad) provided in the control unit may be omitted. The control unit may be provided with another operation interface such as an operation stick. Moreover, it is good also as what receives input from a keyboard or a mouse | mouth as a structure which can connect devices, such as a keyboard and a mouse | mouth, to a control part. For example, although the secondary battery is used as the power source, the power source is not limited to the secondary battery, and various batteries can be used. For example, a primary battery, a fuel cell, a solar cell, a thermal cell, or the like may be used.

  FIG. 16 is an explanatory diagram showing an external configuration of the HMD in the modification. In the example of FIG. 16A, the image display unit 20x includes a right optical image display unit 26x instead of the right optical image display unit 26, and a left optical image display unit 28x instead of the left optical image display unit 28. I have. The right optical image display unit 26x and the left optical image display unit 28x are formed smaller than the optical member of the above-described embodiment, and are respectively disposed obliquely above the right eye and the left eye of the user when the HMD is worn. . 16B, the image display unit 20y includes a right optical image display unit 26y instead of the right optical image display unit 26, and a left optical image display unit 28y instead of the left optical image display unit 28. I have. The right optical image display unit 26y and the left optical image display unit 28y are formed smaller than the optical member of the above-described embodiment, and are respectively disposed obliquely below the right eye and the left eye of the user when the HMD is mounted. . Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is also arbitrary, and the optical image display unit covers only a part of the user's eyes, in other words, the optical image display unit completely covers the user's eyes. It can also be realized as an HMD in an uncovered form.

  For example, each processing unit (for example, an image processing unit, a display control unit, etc.) provided in the control unit is realized by the CPU developing and executing a computer program stored in the ROM or hard disk on the RAM. As described. However, these functional units may be configured using an ASIC designed to realize the function.

  For example, the HMD is a binocular transmissive HMD, but may be a monocular HMD. Further, it may be configured as a non-transparent HMD that blocks the transmission of outside scenes when the user wears the HMD, or may be configured as a video see-through in which a camera is mounted on the non-transparent HMD. Further, for example, instead of the image display unit worn like glasses, an ordinary flat display device (a liquid crystal display device, a plasma display device, an organic EL display device, etc.) may be employed as the image display unit. Also in this case, the connection between the control unit and the image display unit may be a connection via a wired signal transmission path or a connection via a wireless signal transmission path. If it does in this way, a control part can also be utilized as a remote control of a usual flat type display device. For example, instead of the image display unit worn like glasses, an image display unit having another shape such as an image display unit worn like a hat may be adopted as the image display unit. Further, the earphone may be an ear-hook type or a headband type, or may be omitted. Further, for example, it may be configured as a head-up display (HUD) mounted on a vehicle such as an automobile or an airplane, or other transportation means. For example, you may comprise as HMD incorporated in body armor, such as a helmet. For example, it may be configured as a hand-held handheld display.

  For example, in the above embodiment, the image light generation unit is configured using a backlight, a backlight control unit, an LCD, and an LCD control unit. However, the above aspect is merely an example. The image light generation unit may include a configuration unit for realizing another method together with or in place of these configuration units. For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image generation unit can use a digital micromirror device or the like instead of the LCD. For example, the present invention can be applied to a laser retinal projection type HMD.

・ Modification 3:
In the above embodiment, an example of the display process has been described. However, the processing procedure shown in the above embodiment is merely an example, and various modifications are possible. For example, some steps may be omitted, and other steps may be added. Further, the order of the steps to be executed may be changed.

  For example, in step S108 of the display process, a two-dimensional image (a two-dimensional image represented by 2D image data or an outside scene image captured by a camera) and a three-dimensional image (a tertiary represented by comparative 3D image data) are displayed. The original image) is compared with each other, and the area where both match is defined as the “cancellation area”. For this reason, in the said embodiment, the magnitude | size of a cancellation area | region and the magnitude | size of the area | region where a three-dimensional image is displayed become substantially the same. However, the size of the cancellation area may be different from the size of the area where the 3D image is displayed. For example, the size of the cancel area> the size of the area where the 3D image is displayed may be set.

  For example, in step S108 of the display process, a cancel area may be set in consideration of the user's parallax. In this case, the image adjustment unit supplies the right eye adjustment image to be supplied as the right eye image data Data1 and the left eye adjustment image to be supplied as the left eye image data Data2 to the user. It may be prepared separately according to the parallax. The parallax of the user may be stored in advance in the storage unit, and may be specified or changed by the user.

  For example, the server may execute step S108 of the display process. The method for specifying the cancel area in the server is the same as that in step S108 in FIG. After specifying the cancel area, the server may transmit the coordinates of the specified cancel area to the HMD.

  For example, the server may perform steps S108 and S110 of the display process. The method for specifying the cancel area in the server is the same as that in step S108 in FIG. The adjustment image generation method in the server is the same as that in step S110 in FIG. The server should just transmit the produced | generated adjustment image to HMD after the production | generation of an adjustment image.

  For example, in step S112 of the display process, the image adjusting unit processes the image data of the adjusted image and the partial 3D image data according to the movement and posture of the user's head acquired by the 9-axis sensor, Later data may be transmitted to the image processing unit. In this way, it is possible to realize display processing that follows the movement and posture of the user's head.

  For example, in step S112 of the display process, the image processing unit displays both images by synthesizing the received image data of the adjusted image and the partial 3D image data. However, the image processing unit supplies the received image data of the adjusted image and the partial 3D image data to the image display unit in a time-sharing manner so that the user can recognize that both images are displayed simultaneously. Also good.

  For example, the image storage unit of the server may further include “related data” related to the partial 3D data in addition to the 2D image data and the partial 3D image data. The related data is information related to the partial 3D data, and various forms such as image data, character string data, and audio data can be adopted. In that case, in step S112 of the display process, the image processing unit may display (or reproduce, etc.) related data in addition to the image data of the adjusted image and the partial 3D image data.

  For example, in step S204 of the display process, the 3D image acquisition unit may acquire partial 3D image data corresponding to the object designated by the user from the server. For example, gesture detection (pointing to an object to be designated), line-of-sight detection (gazing at an object to be designated), speech recognition (calling the name of the object to be designated), or the like can be used for designation of an object by a user.

  For example, in the display processing of the above-described embodiment, the image adjustment unit reduces the visibility of an object that is visible to the user through the image display unit, and displays a virtual image that represents the object three-dimensionally. To be displayed. However, the image adjustment unit causes the image display unit to form a virtual image that three-dimensionally represents an object, and at that time, a part of the object displayed as a virtual image (for example, a portion corresponding to the background) The visibility of at least a part of the outside scene that is visible to the user through the screen may be reduced. That is, even when there is no two-dimensional image that can be seen by the user through the image display unit or the same object as the “three-dimensionally represented object” in the real world, the image adjustment unit displays the display of the above embodiment. You may implement the process similar to a process. In this way, when augmented reality processing is performed using a transmissive HMD, the visibility of an object displayed as a virtual image is improved with respect to the outside scene that the user sees through the image display unit. Can be improved.

-Modification 4:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

DESCRIPTION OF SYMBOLS 1 ... Image display system 10 ... Control part 11 ... Decision key 12 ... Illumination part 13 ... Display switch key 14 ... Trackpad 15 ... Luminance switch key 16 ... Direction key 17 ... Menu key 18 ... Power switch 20 ... Image display part 21 ... Right holding unit 22 ... Right display driving unit 23 ... Left holding unit 24 ... Left display driving unit 26 ... Right optical image display unit 28 ... Left optical image display unit 30 ... Earphone plug 32 ... Right earphone 34 ... Left earphone 40 ... Connection unit 42 ... Right cord 44 ... Left cord 46 ... Connecting member 48 ... Body code 51 ... Transmitter 52 ... Transmitter 53 ... Receiver 54 ... Receiver 61 ... Camera (imaging unit)
66 ... 9-axis sensor 110 ... Input information acquisition unit 100 ... HMD (head-mounted display device)
120 ... Storage unit 122 ... Partial 3D image data storage unit 130 ... Power source 132 ... Wireless communication unit 140 ... CPU
DESCRIPTION OF SYMBOLS 142 ... Two-dimensional image acquisition part 144 ... Three-dimensional image acquisition part 146 ... Image adjustment part 160 ... Image processing part 170 ... Audio | voice processing part 180 ... Interface 190 ... Display control part 201 ... Right backlight control part 202 ... Left backlight control Unit 211 ... Right LCD control unit 212 ... Left LCD control unit 221 ... Right backlight 222 ... Left backlight 241 ... Right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate 200 ... Projector 250 ... Screen 300 ... Server 310 ... CPU
312 ... Transmission unit 340 ... Communication interface 350 ... Storage unit 352 ... Image storage unit PCLK ... Clock signal VSync ... Vertical synchronization signal HSync ... Horizontal synchronization signal Data ... Image data Data1 ... Right eye image data Data2 ... Left eye image data OA ... External device PC ... Personal computer SC ... Outside view VI ... Virtual image VR ... Field of view VR1 ... Field of view VR2 ... Field of view RE ... Right eye LE ... Left eye ER ... End EL ... End BS ... Communication carrier A1 ... Maximum emission area A2 ... Cancel Area AD ... Adjusted image BD ... Two-dimensional image for background OBJ ... Object INT ... Internet

Claims (10)

A transmissive head-mounted display device that allows a user to visually recognize a virtual image and an outside scene,
An image display unit for allowing the user to visually recognize the virtual image;
The object included in the outside scene, which reduces the visibility of an object that is transmitted through the image display unit and visually recognized by the user, and at least the virtual image that three-dimensionally represents the object is displayed on the image display unit. An image adjustment unit to be displayed;
Equipped with a,
The image adjusting unit is
An adjustment image is generated in which the same color as the object that the user sees through the image display unit is arranged,
A head-mounted display device that reduces the visibility of the object by displaying the virtual image representing the adjusted image on the image display unit . A transmissive head-mounted display device that allows a user to visually recognize a virtual image and an outside scene,
An image display unit for allowing the user to visually recognize the virtual image;
The object included in the outside scene, which reduces the visibility of an object that is transmitted through the image display unit and visually recognized by the user, and at least the virtual image that three-dimensionally represents the object is displayed on the image display unit. An image adjustment unit to be displayed;
Equipped with a,
The image adjusting unit is
An image representing the outside scene that is visible to the user through the image display unit, and a part of the image excluding the object included in the outside scene is transmitted through the image display unit and used. Generating an adjustment image arranged at a position overlapping with the object visually recognized by the person,
A head-mounted display device that reduces the visibility of the object by displaying the virtual image representing the adjusted image on the image display unit . The head-mounted display device according to claim 1 , further comprising:
A three-dimensional image acquisition unit for acquiring a three-dimensional image for representing the object three-dimensionally;
The head-mounted display device, wherein the image adjustment unit determines the color based on the three-dimensional image. The head-mounted display device according to claim 2 , further comprising:
A three-dimensional image acquisition unit for acquiring a three-dimensional image for representing the object three-dimensionally;
A two-dimensional image acquisition unit for acquiring a two-dimensional image representing the outside scene two-dimensionally;
With
The image adjustment unit is a head-mounted display device that identifies the overlapping position in the adjustment image by comparing the two-dimensional image and the three-dimensional image. The head-mounted display device according to claim 2 , further comprising:
A three-dimensional image acquisition unit for acquiring a three-dimensional image for representing the object three-dimensionally;
A two-dimensional image acquisition unit for acquiring a two-dimensional image obtained by imaging the user's visual field direction in a state in which the head-mounted display device is mounted;
With
The image adjustment unit is a head-mounted display device that identifies the overlapping position in the adjustment image by comparing the two-dimensional image and the three-dimensional image. An image display system comprising the head-mounted display device according to any one of claims 1 to 5 , further comprising:
An information processing device;
An image display device;
With
The information processing apparatus includes:
A transmission unit that transmits a two-dimensional image including the object to the image display device and transmits a three-dimensional image for representing at least the object three-dimensionally to the head-mounted display device. ,
The image display device includes:
An image display system comprising a display unit for displaying the two-dimensional image transmitted from the information processing apparatus. A method of controlling a transmissive head-mounted display device that allows a user to visually recognize a virtual image and an outside scene,
A first step of allowing the user to visually recognize the virtual image;
The object included in the outside scene, which is visible through the head-mounted display device and reduces the visibility of the object visually recognized by the user, and displays at least the virtual image representing the object three-dimensionally A second step;
Equipped with a,
The second step includes
Generating an adjustment image in which the same color as the object that the user visually recognizes through the head-mounted display device is arranged;
Displaying the virtual image representing the adjusted image on the head-mounted display device, thereby reducing the visibility of the object . A method of controlling a transmissive head-mounted display device that allows a user to visually recognize a virtual image and an outside scene,
A first step of allowing the user to visually recognize the virtual image;
The object included in the outside scene, which is visible through the head-mounted display device and reduces the visibility of the object visually recognized by the user, and displays at least the virtual image representing the object three-dimensionally A second step;
Equipped with a,
The second step includes
An image representing the outside scene that is transmitted through the head-mounted display device and visually recognized by the user, and a part of the image excluding the object included in the outside scene is displayed on the head-mounted display device. Generating an adjustment image that is disposed at a position that overlaps with the object that the user sees through,
Displaying the virtual image representing the adjusted image on the head-mounted display device, thereby reducing the visibility of the object . A computer program used in a transmissive head-mounted display device that allows a user to visually recognize a virtual image and an outside scene,
A first function for allowing the user to visually recognize the virtual image;
The object included in the outside scene, which is visible through the head-mounted display device and reduces the visibility of the object visually recognized by the user, and displays at least the virtual image representing the object three-dimensionally The second function,
To the computer ,
The second function is:
An adjustment image is generated in which the same color as the object that the user sees through the head-mounted display device is arranged,
A computer program that reduces the visibility of the object by displaying the virtual image representing the adjustment image on the head-mounted display device . A computer program used in a transmissive head-mounted display device that allows a user to visually recognize a virtual image and an outside scene,
A first function for allowing the user to visually recognize the virtual image;
The object included in the outside scene, which is visible through the head-mounted display device and reduces the visibility of the object visually recognized by the user, and displays at least the virtual image representing the object three-dimensionally The second function,
To the computer ,
The second function is:
An image representing the outside scene that is transmitted through the head-mounted display device and visually recognized by the user, and a part of the image excluding the object included in the outside scene is displayed on the head-mounted display device. An adjustment image arranged at a position overlapping the object that the user sees through
A computer program that reduces the visibility of the object by displaying the virtual image representing the adjustment image on the head-mounted display device .

Images