JP2016091348A - Head-mounted display device and control method for the same as well as computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display a predetermined image with an appropriate shielding relation with respect to an actual object in the outside world.SOLUTION: There is provided a head-mounted display device comprising: a superimposed image display control unit that causes a display unit to display a predetermined image in association with the position of an object in the outside world which transmitted and visible; a position information acquisition unit that acquires position information indicating the position of the object in a two-dimensional space; and a depth information acquisition unit that acquires depth information indicating the depth of each part in the outside world. The superimposed image display control unit calculates the position of the object in a three-dimensional space based on the position information and the depth information; defines a display area for a predetermined image in the three-dimensional space based on the position of the object; and if a determination is made that the object is present between the display area and the display unit in the three-dimensional space based on information indicating the display area and the depth information, displays the predetermined image by excluding, from the display area, a predetermined range at least including the area that overlaps the object with respect to the display area in the depth direction.SELECTED DRAWING: Figure 9

Description

  The present invention relates to a head-mounted display device that includes a display unit that can visually recognize the outside world, a control method thereof, and a computer program.

  In recent years, a technique called augmented reality (AR) has attracted attention. Augmented reality is a technology that displays a virtual object superimposed on a real space. Augmented reality devices are classified into a video see-through method and an optical see-through method according to a method for synthesizing real space and augmented reality information. The video see-through method is a method of photographing a real space with a camera and synthesizing augmented reality information with the captured image. The optical see-through method is a method in which a virtual object is optically reflected in a real space by using a half mirror or the like.

  In a video see-through display device, for example, as described in Patent Document 1, based on image data output by an imaging unit and depth map information output by a distance sensor unit, the position of a light source in real space , Estimating the color and intensity of light emitted from the light source and information indicating object reflected light that is reflected by the real object and reaches the virtual object, and applying them to the virtual object A technique for reflecting the influence of light reflected from a light source or an object on a virtual object has been proposed. According to this technology, it is possible to display an image arranged so that a virtual object actually exists on an image obtained by imaging a real space.

JP2013-3848A

  The conventional technique is for a video see-through display device. On the other hand, in an optical see-through display device, only a virtual object is captured in the real space, and what kind of influence the physical space object has on the virtual object in the positional shielding relationship. There was a problem that this point was not fully examined. This problem is a problem that occurs even in an optical see-through head-mounted display device. In addition, in conventional head-mounted display devices, it is desired to improve user convenience, improve detection accuracy, make the device configuration compact, reduce costs, save resources, and facilitate manufacturing. It was.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) One embodiment of the present invention is a head-mounted display device that includes a display unit that can see through the outside. The head-mounted display device includes a superimposed image display control unit that causes the display unit to display a predetermined image in association with a position of an external object that is visible through the display unit, and the object in a two-dimensional space. A position information acquisition unit that acquires position information indicating the position of the object, and a depth information acquisition unit that acquires depth information indicating the depth of each part around at least the object in the outside world. The superimposed image display control unit calculates the position of the target object in a three-dimensional space based on the position information and the depth information, and based on the position of the target object, the third order for the predetermined image Determine the display area in the original space, based on the information representing the display area and the depth information, determine whether an object exists between the display area and the display unit in the three-dimensional space, When it is determined that the object exists, the predetermined image is displayed by excluding a predetermined range including at least a region where the object overlaps the display region in the depth direction from the display region. May be. According to the head-mounted display device of this aspect, when it is determined that an object exists between the display area of the predetermined image and the display unit in the three-dimensional space, the object with respect to the display area in the depth direction A predetermined range including at least an overlapping region is removed from the display region, and a predetermined image is displayed. Therefore, when an object is present on the front side (display unit side) of the display area, the predetermined image is not displayed in a range that overlaps the object in the depth direction in the display area. . Therefore, there is an effect that a predetermined image can be displayed with an appropriate shielding relationship (that is, a longitudinal relationship) with respect to an actual object in the outside world.

(2) In the head-mounted display device of the above aspect, the predetermined image may be extended information as augmented reality to be given to the object. According to the head-mounted display device of this aspect, there is an effect that it is possible to display with an appropriate shielding relationship between the object and the extended information as augmented reality to be given to the object.

(3) In the head-mounted display device according to the aspect described above, the position information acquisition unit may include a camera that images the object. According to the head-mounted display device of this form, position information indicating the position of the object in the two-dimensional space can be acquired with a simple configuration.

(4) In the head-mounted display device according to the aspect described above, the depth information acquisition unit may include a depth sensor that outputs a signal indicating the depth of the outside world. According to the head-mounted display device of this aspect, depth information indicating the depth of at least each part around the object in the outside world can be acquired with a simple configuration.

(5) In the head-mounted display device of the above aspect, the superimposed image display control unit displays the position information acquired by the position information acquisition unit and the depth information acquired by the depth information acquisition unit on the display unit. The display area may be determined based on the position information and the depth information after conversion into a value expressed in a coordinate system viewed through. According to the head-mounted display device of this form, when the position information and the depth information are acquired as values of a sitting system different from the coordinate system seen through the display unit, the position information and It is possible to unify the depth information into values expressed in a coordinate system viewed through the display unit. Therefore, the display accuracy of augmented reality can be increased.

(6) Another embodiment of the present invention is a head-mounted display device that includes a display unit that can see through the outside world. The head-mounted display device includes a superimposed image display control unit that causes the display unit to display a first image in association with a position of an external object that is visible through the display unit, and the target in a two-dimensional space. You may provide the positional information acquisition part which acquires the positional information which shows the position of an object, and the depth information acquisition part which acquires the depth information which shows the depth of each part of the circumference | surroundings of the said target object at least in the external world. The superimposed image display control unit calculates the position of the object in a three-dimensional space based on the position information and the depth information, and based on the position of the object, the superposition image display control unit A display area in a three-dimensional space is defined as a first display area, a display area in the three-dimensional space for a second image different from the first image is acquired as a second display area, and the first display Whether or not the second display area exists between the first display area and the display unit in the three-dimensional space is determined based on information representing the area and information representing the second display area. When it is determined that the second display area exists, a predetermined range including at least an area where the second display area overlaps the first display area in the depth direction is removed from the first display area. Te, it may be the display of the first image. According to the head-mounted display device of this aspect, when it is determined that another image exists between the display area of the predetermined image and the display unit in the three-dimensional space, the display area in the depth direction is compared with the display area. Thus, a predetermined range including at least a region where other images overlap is removed from the display region, and a predetermined image is displayed. Therefore, when another image exists on the front side (display unit side) of the display area, the predetermined image is displayed in a range overlapping the object in the depth direction in the display area. There is no. Therefore, there is an effect that a predetermined image can be displayed with an appropriate shielding relationship (that is, a longitudinal relationship) with respect to other images.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with another new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention is realized as an apparatus including one or more elements among four elements of a display unit, a superimposed image display control unit, a position information acquisition unit, and a depth information acquisition unit. Is possible. That is, this apparatus may or may not have a display unit. Further, the apparatus may or may not have a superimposed image display control unit. Moreover, the apparatus may or may not have a position information acquisition unit. In addition, the apparatus may or may not have a depth information acquisition unit. A display part is good also as what can permeate | transmit and visually recognize the external field, for example. For example, the superimposed image display control unit may cause the display unit to display a predetermined image in association with the position of the object in the outside world that is seen through. The position information acquisition unit may acquire position information indicating the position of the object in the two-dimensional space, for example. For example, the depth information acquisition unit may acquire depth information indicating the depth of at least each part around the object in the outside world. Furthermore, the superimposed image display control unit calculates, for example, the position of the object in the three-dimensional space based on the position information and the depth information, and based on the position of the object, in the three-dimensional space for the predetermined image. When a display area is defined, and it is determined whether or not an object exists between the display area and the display unit in the three-dimensional space based on the information representing the display area and the depth information. In addition, a predetermined image may be displayed by excluding a predetermined range including at least a region where an object overlaps the display region in the depth direction from the display region. Such a device can be realized as, for example, a head-mounted display device, but can also be realized as a device other than the head-mounted display device. According to such a form, at least one of various problems such as improvement of user convenience, improvement of detection accuracy, compactness of the device configuration, cost reduction, resource saving, and ease of manufacture is achieved. Can be solved. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device.

  The present invention can also be realized in various forms other than the head-mounted display device. For example, display device, head-mounted display device and display device control method, head-mounted display system, display device, head-mounted display system, and computer program for realizing the functions of the display device, and the computer program Can be realized in the form of a data signal or the like embodied in a carrier wave including the computer program.

It is explanatory drawing which shows schematic structure of the head mounted display apparatus (HMD) in 1st Embodiment of this invention. It is a block diagram which shows the structure of HMD functionally. It is explanatory drawing which shows an example of the augmented reality display by HMD. It is explanatory drawing which shows an example of the usage form of HMD. It is a flowchart which shows an augmented reality display control process. It is explanatory drawing which shows an example of the depth map obtained by the depth sensor. It is explanatory drawing which shows an example of a marker. It is explanatory drawing which shows an example of the data structure of an extended information database. It is explanatory drawing which shows an example of a display of AR image. It is explanatory drawing which illustrates the form of the display of AR image with respect to a display area. It is explanatory drawing which shows the modification 3. It is explanatory drawing which shows the modification 4. It is explanatory drawing which shows the structure of the external appearance of HMD in another modification.

A. First embodiment:
A-1. Basic configuration of a head-mounted display device:
FIG. 1 is an explanatory diagram showing a schematic configuration of a head-mounted display device according to the first embodiment of the present invention. The head-mounted display device 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The HMD 100 is a see-through head-mounted display device in which an image floats in the external world visually recognized through a glass.

  The HMD 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when attached to the user's head, and a control unit (controller) 10 that controls the image display unit 20.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, and a left optical image display unit 28. It is out. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 is disposed inside the right holding unit 21, in other words, on the side facing the user's head when the user wears the image display unit 20. Further, the left display driving unit 24 is disposed inside the left holding unit 23. Hereinafter, the right holding unit 21 and the left holding unit 23 will be described as “holding units” without being distinguished from each other. Similarly, the right display drive unit 22 and the left display drive unit 24 are described as “display drive units” without being distinguished from each other, and the right optical image display unit 26 and the left optical image display unit 28 are not distinguished from each other as “optical image display units”. Will be described.

  The display driving unit includes a liquid crystal display (hereinafter referred to as “LCD”) 241, 242, projection optical systems 251, 252, and the like (see FIG. 2). Details of the configuration of the display driving unit will be described later. The optical image display unit as an optical member includes light guide plates 261 and 262 (see FIG. 2) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display driving unit to the user's eyes. The light control plate is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 (the side opposite to the user's eye side). The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. Further, by adjusting the light transmittance of the light control plate, it is possible to adjust the amount of external light entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48 connected to the control unit 10, a right cord 42 and a left cord 44 in which the main body cord 48 branches into two, and a connecting member 46 provided at the branch point. . The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24. The connecting member 46 is provided with a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. A connector (not shown) that fits to each other is provided on each end of the main body cord 48 on the opposite side of the connecting member 46 and the control section 10. The connector of the main body cord 48 and the connector of the control section 10 are provided. The control unit 10 and the image display unit 20 are connected to or disconnected from each other by the fitting / releasing. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for controlling the HMD 100. The control unit 10 includes a lighting unit 12, a touch pad 14, a cross key 16, and a power switch 18. The lighting unit 12 notifies the operation state of the HMD 100 (for example, ON / OFF of the power supply) by its light emission mode. For example, an LED (Light Emitting Diode) can be used as the lighting unit 12. The touch pad 14 detects a contact operation on the operation surface of the touch pad 14 and outputs a signal corresponding to the detected content. As the touch pad 14, various touch pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The cross key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power state of the HMD 100 by detecting a slide operation of the switch.

  FIG. 2 is a block diagram functionally showing the configuration of the HMD 100. The control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a wireless communication unit 132, a GPS module 134, a CPU 140, an interface 180, and transmission units (Tx) 51 and 52. The parts are connected to each other by a bus (not shown).

  The input information acquisition unit 110 acquires a signal corresponding to an operation input to the touch pad 14, the cross key 16, the power switch 18, and the like, for example. The storage unit 120 includes a ROM, a RAM, a DRAM, a hard disk, and the like.

  The power supply 130 supplies power to each part of the HMD 100. As the power source 130, for example, a secondary battery such as a lithium polymer battery or a lithium ion battery can be used. Further, instead of the secondary battery, a primary battery or a fuel cell may be used, or the wireless battery may be operated by receiving wireless power feeding. Furthermore, you may make it receive electric power supply from a solar cell and a capacitor. The wireless communication unit 132 performs wireless communication with other devices in accordance with a predetermined wireless communication standard such as a wireless LAN, Bluetooth (registered trademark), or iBeacon (registered trademark). The GPS module 134 detects its current position by receiving a signal from a GPS satellite.

  The CPU 140 reads out and executes the computer program stored in the storage unit 120, thereby operating the operating system (OS) 150, the image processing unit 160, the display control unit 162, the superimposed image display control unit 164, and the sound processing unit 170. Function as.

  The image processing unit 160 generates a signal based on content (video) input via the interface 180 or the wireless communication unit 132. Then, the image processing unit 160 controls the image display unit 20 by supplying the generated signal to the image display unit 20 via the connection unit 40. The signal supplied to the image display unit 20 differs between the analog format and the digital format. In the case of the analog format, the image processing unit 160 generates and transmits a clock signal PCLK, a vertical synchronization signal VSync, a horizontal synchronization signal HSync, and image data Data. Specifically, the image processing unit 160 acquires an image signal included in the content. For example, in the case of a moving image, the acquired image signal is an analog signal generally composed of 30 frame images per second. The image processing unit 160 separates a synchronization signal such as a vertical synchronization signal VSync and a horizontal synchronization signal HSync from the acquired image signal, and generates a clock signal PCLK by a PLL circuit or the like according to the period. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like. The image processing unit 160 stores the converted digital image signal as image data Data of RGB data in the DRAM in the storage unit 120 for each frame.

  On the other hand, in the case of the digital format, the image processing unit 160 generates and transmits the clock signal PCLK and the image data Data. Specifically, when the content is in digital format, the clock signal PCLK is output in synchronization with the image signal, so that the generation of the vertical synchronization signal VSync and the horizontal synchronization signal HSync and the A / D conversion of the analog image signal are performed. It becomes unnecessary. The image processing unit 160 performs image processing such as resolution conversion processing, various tone correction processing such as adjustment of luminance and saturation, and keystone correction processing on the image data Data stored in the storage unit 120. May be executed.

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. The image data Data transmitted via the transmission unit 51 is also referred to as “right eye image data Data1”, and the image data Data transmitted via the transmission unit 52 is also referred to as “left eye image data Data2”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control unit 10 and the image display unit 20.

  The display control unit 162 generates a control signal for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 162 uses the control signal to turn on / off the right LCD 241 by the right LCD control unit 211, turn on / off the right backlight 221 by the right backlight control unit 201, and control the left LCD. Each of the right display drive unit 22 and the left display drive unit 24 is controlled by individually controlling ON / OFF driving of the left LCD 242 by the unit 212 and ON / OFF driving of the left backlight 222 by the left backlight control unit 202. Controls the generation and emission of image light. The display control unit 162 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. Similarly, the display control unit 162 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The superimposed image display control unit 164 causes the display control unit 162 to display an image as augmented reality in association with the position of the object in the outside world that is visible through the display control unit 162. The superimposed image display control unit 164 will be described in detail later.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and transmits the acquired audio signal to a speaker (not shown) in the right earphone 32 and a speaker (not shown) in the left earphone 34 connected to the connecting member 46. To supply. For example, when a Dolby (registered trademark) system is adopted, processing is performed on an audio signal, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, respectively.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer PC, a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. (See also FIG. 1), a depth sensor 62, and a nine-axis sensor 66.

  The camera 61 is an RGB camera and is disposed at a position corresponding to the user's nose root when the user wears the image display unit 20. Therefore, the camera 61 performs color imaging of a predetermined direction of the HMD 100, that is, the outside world in the direction in which the user is facing in a state where the user wears the image display unit 20 on the head. The camera 61 can be a monochrome camera instead of the RGB camera.

  The depth sensor 62 is a kind of depth (distance) sensor, and is arranged side by side with the camera 61.

  The nine-axis sensor 66 is a motion sensor that detects acceleration (three axes), angular velocity (three axes), and geomagnetism (three axes), and is disposed at a position corresponding to the user's eyebrows in this embodiment. Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, the movement of the user's head is detected. The direction of the image display unit 20, that is, the field of view of the user is specified from the detected head movement.

  The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, a right LCD control unit 211 that functions as a display element, and a right An LCD 241 and a right projection optical system 251 are included. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are also collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data1 input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby converting the illumination light emitted from the right backlight 221 into an image. Modulate into effective image light to represent.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The optical image display unit can use any method as long as a virtual image is formed in front of the user's eyes using image light. For example, a diffraction grating or a transflective film may be used. Good. Note that the emission of image light by the HMD 100 is also referred to as “displaying an image” in this specification.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. That is, the left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, and a left LCD control unit 212 that functions as a display element. And a left LCD 242 and a left projection optical system 252. Similar to the right LCD 241, the left LCD 242 is irradiated from the left backlight 222 by changing the transmittance of light transmitted through the left LCD 242 by driving the liquid crystal at each pixel position arranged in a matrix. The illumination light is modulated into effective image light representing the image. In this embodiment, the backlight method is adopted, but image light may be emitted using a front light method or a reflection method.

A-2. About augmented reality display:
FIG. 3 is an explanatory diagram illustrating an example of augmented reality display by the HMD 100. FIG. 3 illustrates the user's field of view VR. As described above, the image light guided to both eyes of the user of the HMD 100 forms an image on the retina of the user, so that the user visually recognizes the image VI as an augmented reality (AR). In the example of FIG. 3, the image VI is a standby screen of the OS of the HMD 100. Further, the optical image display units 26 and 28 transmit the light from the outside world SC, so that the user visually recognizes the outside world SC. As described above, the user of the HMD of this embodiment can see the image VI and the external environment SC behind the image VI for the portion of the visual field VR where the image VI is displayed. Further, only the outside SC can be seen in the portion of the field of view VR1 where the image VI is not displayed.

  FIG. 4 is an explanatory diagram illustrating an example of a usage pattern of the HMD 100. In FIG. 4, the user U visually recognizes the wall WL existing in the outside through the optical image display units 26 and 28 of the HMD 100. The wall WL is a historically important building, for example, and the user UL visually recognizes the wall WL using the HMD 100, so that, for example, the scenery at the time when the wall WL was established can be seen on the wall surface of the wall WL. The shown image PI can be visually recognized as augmented reality. It is the superimposed image display control unit 164 in FIG. 2 that provides such augmented reality. Giving augmented reality to the wall WL is merely an example of how the HMD 100 is used. For example, various colors are given to a white car as augmented reality, or an advertisement image is given to a white advertising tower. It can be set as various usage forms, such as. That is, an image as an augmented reality is displayed in a predetermined display area related to the front surface of the object to which the augmented reality is assigned. The superimposed image display control unit 164 is functionally realized by the CPU 140 executing a predetermined program stored in the storage unit 120. Details of the predetermined program will be described below.

  FIG. 5 is a flowchart showing the augmented reality display control process. This augmented reality display control process corresponds to the predetermined program, and is repeatedly executed by the CPU 140 at predetermined time intervals. When the processing is started, the CPU 140 first obtains an RGB image from the camera 61 (step S110), captures the outside world as a two-dimensional image from the output signal of the depth sensor 62, and calculates the depth at each pixel of the image. A depth map (distance image) indicating the pixel density by the shading of the pixel is generated (step S120).

  FIG. 6 is an explanatory diagram illustrating an example of a depth map. As illustrated, the depth map DP is a grayscale image, and the depth (distance) at each pixel is represented by the grayscale. The depth sensor 62 and the process of step S120 executed by the CPU 140 implement the “depth information acquisition unit” described in the “Summary of Invention” column.

  After execution of step S120 in FIG. 5, the CPU 140 recognizes the marker from the RGB image acquired in step S110, and acquires the two-dimensional position coordinates of the marker (step S130). In the present embodiment, as shown in FIG. 4, a marker MK is attached in advance to one corner of a wall WL that is an object to which augmented reality is added, and the object is recognized by recognizing the marker MK from within the RGB image. It is possible to recognize.

  FIG. 7 is an explanatory diagram illustrating an example of the marker MK. As shown in the figure, the marker MK is a two-dimensional marker, on which an image of a predetermined pattern that serves as a mark for designating an object to which augmented reality is assigned is printed. The object can be identified by the image of this pattern. In step S130 in FIG. 5, the marker MK is recognized from the RGB image, and a coordinate value indicating the position of the marker MK in the two-dimensional space is acquired as a two-dimensional position coordinate. The “position information acquisition unit” described in the “Summary of Invention” section is realized by the camera 61 and the processing in steps S110 and S130 executed by the CPU 140.

  In this embodiment, the object can be recognized by pasting a marker to the object in advance. Instead, the shape pattern of the “wall” that is the object is stored in advance. The object may be recognized by pattern recognition in the RGB image.

  After execution of step S130, the CPU 140 recognizes the marker MK from the depth map acquired in step S120, and acquires the depth of the marker MK (step S140). Instead of the marker MK, the shape pattern of the “wall” that is the object may be stored in advance, and the object may be recognized by pattern recognition in the depth map (distance image).

  Next, the CPU 140 converts the two-dimensional position coordinates of the marker acquired in step S130 and the depth map acquired in step S120 into values expressed in a coordinate system viewed through the optical image display units 26 and 28. Field of view conversion is performed (step S150). The camera 61 and the depth sensor 62 are provided at positions different from those of the optical image display units 26 and 28, and the two-dimensional position coordinates of the marker acquired in step S130 are expressed in a two-dimensional coordinate system viewed from the camera 61. The depth map acquired in step S130 is expressed in a two-dimensional coordinate system viewed from the depth sensor 62. For this reason, in step S160, visual field conversion is performed to convert the two-dimensional position coordinates and the depth map into values expressed in a two-dimensional coordinate system viewed through the optical image display units 26 and 28.

  After execution of step S150, the CPU 140 converts the marker image pattern recognized in step S130 into a marker identification number, and reads extended information corresponding to the marker identification number from the extended information database (step S160). The extended information database is stored in the storage unit 120 in advance. The extended information database may be stored in advance in a storage device on a network, particularly the Internet, instead of the storage unit 120.

  FIG. 8 is an explanatory diagram showing an example of the data structure of the extended information database DB. The extended information database DB includes a plurality of records including three items C1, C2, and C3 of “marker identification number”, “display image”, and “display area”. One record corresponds to one type of marker. In the item “C1 of marker identification number”, an integer of 1 or more for identifying the marker is recorded. In the item “display image” C2, a file name of image data representing an image to be added as augmented reality (hereinafter referred to as “AR image”) is recorded. The image data is still image data and is stored in the storage unit 120 in advance. As one of the AR images, for example, an image showing a landscape at the time when the wall WL was established.

  In the item “display area” C3, display area information representing a display area for displaying an AR image is recorded. The display area information is data indicating the position and size of the display area based on the position of a pair of vertices out of the two pairs of vertices of the rectangle. The position of the paired vertex is indicated by a relative distance with respect to the two-dimensional position coordinate of the marker after visual field conversion in step S150 in the coordinate system viewed through the optical image display units 26 and 28 described above.

  In step S160 (FIG. 5), specifically, the extended information database DB is searched using the marker identification number as a key. A record whose key matches the item C1 of the “marker identification number” is selected, and the extended information included in the record, that is, the information of the items C2 and C3 of the “display image” and “display area” is stored in the extended information database DB. Read from.

  Subsequently, the CPU 140 sets the position and size of the display area in the three-dimensional space based on the display area information of the item “C3” of “display area” read out in step S160 and the depth of the marker acquired in step S140. (Step S170). The “three-dimensional space” here means that the X axis and the Y axis are the two coordinate axes of the two-dimensional coordinate system seen through the optical image display units 26 and 28, and the Z axis is the optical image display unit. This space is represented by a coordinate system (that is, a viewpoint coordinate system) having directions of 26 and 28. According to step S170, the position in the X-axis / Y-axis space is determined by the display area information of the item C3 of “display area”, and the position in the Z-axis direction is regarded as a position that matches the depth of the marker. The position and size of the display area in the three-dimensional space are determined.

  Thereafter, the CPU 140 compares the position and size of the display area set in step S170 with the depth map after the visual field conversion in step S150, and in the three-dimensional space described above, the front side of the display area (in the Z-axis direction). It is determined whether there is any object represented by the information of the depth map between the optical image display units 26 and 28 side, that is, between the display area and the optical image display units 26 and 28 (step S180). The “object” here is a real-world object indicated by the information of the depth map. Specifically, depth information at a position corresponding to the display area is searched from the depth map, and when the depth information indicates a value on the near side of the display area, it is determined that an object exists. In this determination, if the pixel indicating the near side value forms a small aggregate of 1 pixel or 2 pixels, it may be determined that the object is noise or a minute object and not determined as an object. Moreover, it is good also as a structure which does not determine with an object about the aggregate | assembly of specific shapes, such as a linear shape like an electric wire, not only a small aggregate | assembly.

  If it is determined in step S180 that an object is present, a region where the object that is present in the Z-axis direction overlaps the display region is obtained, and a predetermined range including this overlapping region is removed from the display region (step S190). ). The predetermined range is a predetermined shape inscribed in the overlapping area, for example, a rectangular range. In addition, it can change to a rectangle, and can also be made into various shapes, such as a triangle, a pentagon, a hexagon, an ellipse shape. Furthermore, the overlapping area itself can be set within a predetermined range.

  Subsequently, the CPU 140 reads out the AR image having the file name of the item “C2” “display image” read out in step S160 from the storage unit 120, and displays the AR image in the display area (step S200). On the other hand, when it is determined in step S180 that no object exists between the display area and the optical image display units 26 and 28, the CPU 140 performs the process in step S200 without executing the process in step S190. Proceed. After execution of step S200, the extended image display control process ends. Note that the superimposed image display control unit 164 of FIG. 2 is realized by the processing from step S150 to step S200.

A-3. About effect of embodiment:
FIG. 9 is an explanatory diagram illustrating an example of display of an AR image. This illustration is a display in the usage mode shown in FIG. 4 and illustrates the user's field of view VR1. As shown in the drawing, the field of view VR1 includes a wall WL and a car CR stopped in front of the wall WL. Further, the AR image is displayed so as to be superimposed on the wall WL. The AR image display area PA set in step S170 in FIG. 5 is an area having a rectangular shape with P1 and P2 in the figure as opposite vertices, and having a depth on the front surface of the wall WL. For this reason, the vehicle CR, which is an object, exists on the front side of the display area PA. In step S180 of FIG. 5, an affirmative determination is made, and in step S190, an area where the vehicle CR overlaps the display area PA in the depth direction (Z-axis direction) is obtained, and a predetermined range including this overlapping area, for example, in the figure A rectangular portion having P3 and P2 as the opposite vertices is removed from the display area PA. The new display area PA * is a hatched portion in the figure, and the AR image is displayed in the display area PA *.

  FIG. 10 is an explanatory diagram illustrating a display form of the AR image PI with respect to the display area PA *. FIG. 10A shows an example of the original AR image, FIG. 10B shows the first display form, and FIG. 10C shows the second display form. In the present embodiment, the first display form is adopted. As shown in FIG. 10B, in the first display form, the AR image PI is enlarged while maintaining the aspect ratio to the full display area PA before the predetermined range is removed, and then corresponds to the predetermined range. The portion is cut, and the AR image PI1 after the cut is displayed in the display area PA * after the predetermined range is removed.

  As shown in FIG. 10C, in the second display mode, the AR image PI is enlarged to the maximum possible display in the display area PA * after the predetermined range is removed while maintaining the aspect ratio. The enlarged AR image PI2 is displayed in the display area PA *. As a modification of the present embodiment, the second display form may be adopted.

  For this reason, according to the HMD 100 of the present embodiment, when an object is present on the front side (display unit side) of the display area, the AR overlaps the range overlapping the object in the depth direction in the display area. The image is not displayed. Therefore, there is an effect that an AR image can be displayed with an appropriate occlusion relationship (ie, a front-rear relationship) with respect to an actual object in the real space. In the case of the prior art, since the AR image is superimposed and displayed on the entire display area PA (P1-P2 rectangle in FIG. 9) before the predetermined range is removed, a part of the AR image also overlaps the car CR. Although the visibility of the user's field of view is reduced, according to the HMD 100 of the present embodiment, the above-described reduction in visibility can be prevented.

  Further, according to the HMD 100 of the first embodiment, since the field of view is changed in step S150 of FIG. 5, the camera 61 and the depth sensor 62 are provided at positions different from the optical image display units 26 and 28. Nevertheless, the two-dimensional position coordinates of the marker and the depth map can be unified to values expressed in a coordinate system viewed through the optical image display units 26 and 28. Therefore, the display accuracy of augmented reality can be increased.

B. Variations:
The present invention is not limited to the first embodiment and the modifications thereof, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible. is there.

B-1. Modification 1:
In the first embodiment and the modification thereof, the depth sensor 62 detects the depth of at least each part around the object in the outside world. However, the depth sensor 62 may be used instead of the depth sensor. Further, the depth of each part in the outside world may be configured to be indicated in advance as CAD information, and the depth around the object may be acquired from the CAD information.

B-2. Modification 2:
In the first embodiment, the extended information to be given to the target object as augmented reality is still image data. However, it can be replaced with character data, graphic data, moving image data, or the like. The image may be a 3D image. Furthermore, a combination of these various types of data can also be used.

B-3. Modification 3:
In 1st Embodiment, it was set as the structure by which AR image is displayed on the predetermined | prescribed display area | region in connection with the front surface of the target object which provides augmented reality. On the other hand, the predetermined display area need not be limited to the front surface of the object, and may be any position as long as the position is determined based on the position of the object. For example, as shown in FIG. 11 (a), a predetermined display region 210 may be positioned at a position that is the same depth as the front surface 200a of the object 200 and is above the object 200. As shown in (b), the predetermined display area 220 may be positioned at a position farther away from the front surface 200a of the object 200 and above the upper surface 210a of the object 200. In the case of FIG. 11B, the item C3 of “display area” in the extended information database DB includes depth information, and the display area is determined by the relative distance in the three-dimensional direction with respect to the marker. You can do it.

B-4. Modification 4:
In the first embodiment, as shown in FIG. 9, the display area PA is deformed when the vehicle CR, which is an object, is present on the front side of the AR image display area PA. On the other hand, the object can be changed to a display area of an AR image given to another object. That is, as shown in FIG. 12, when an AR image (second image) display region 330 for another object 320 is present in front of the first image display region 310 that is an AR image. The first image may be displayed except for an area where the display area 330 overlaps the display area 310 in the depth direction. Therefore, there is an effect that the AR image can be displayed in an appropriate shielding relationship (that is, a front-rear relationship) with respect to the AR image of another object. In the fourth modification, the second image does not necessarily have to be an AR image given to the object. If the AR image is an AR image, the second image may be displayed regardless of the object. it can. In short, when there is a display area of another AR image on the near side of the display area of the AR image assigned to the object, the display area of the AR image of the object is changed to that of the other AR image. Any configuration can be used as long as it excludes a predetermined range including at least a region where the display region overlaps.

B-5. Other variations:
In the said embodiment, it illustrated about the structure of the head mounted display. However, the configuration of the head-mounted display can be arbitrarily determined without departing from the gist of the present invention. For example, each component can be added, deleted, converted, and the like.

  The allocation of components to the control unit and the image display unit in the above embodiment is merely an example, and various aspects can be employed. For example, the following aspects may be adopted. (I) A mode in which processing functions such as a CPU and a memory are mounted in the control unit, and only a display function is mounted in the image display unit. (Iii) a mode in which the control unit and the image display unit are integrated (for example, a mode in which the control unit is included in the image display unit and functions as a glasses-type wearable computer), (iv) instead of the control unit A mode in which a smartphone or a portable game machine is used, and (v) a mode in which the connection unit (code) is eliminated by adopting a configuration in which the control unit and the image display unit can perform wireless communication and wireless power feeding.

  In the above embodiment, for convenience of explanation, the control unit includes a transmission unit, and the image display unit includes a reception unit. However, each of the transmission unit and the reception unit of the above-described embodiment has a function capable of bidirectional communication, and can function as a transmission / reception unit. For example, the control unit shown in FIG. 2 is connected to the image display unit via a wired signal transmission path. However, the control unit and the image display unit may be connected by a connection via a wireless signal transmission path such as a wireless LAN, infrared communication, or Bluetooth (registered trademark).

  For example, the configurations of the control unit and the image display unit described in the above embodiment can be arbitrarily changed. Specifically, for example, the touch pad may be omitted from the control unit and the operation may be performed using only the cross key. The control unit may be provided with another operation interface such as an operation stick. Moreover, it is good also as what receives an input from a keyboard or a mouse | mouth as a structure which can connect devices, such as a keyboard and a mouse | mouth, to a control part. Further, for example, in addition to an operation input using a touch pad or a cross key, an operation input using a foot switch (a switch operated by a user's foot) may be acquired. For example, after providing a line-of-sight detection unit such as an infrared sensor in the image display unit, the user's line of sight may be detected to obtain an operation input by a command associated with the movement of the line of sight. For example, a user's gesture may be detected using a camera, and an operation input by a command associated with the gesture may be acquired. When detecting a gesture, a user's fingertip, a ring attached to the user's hand, a medical device to be used by the user's hand, or the like can be used as a mark for motion detection. If it is possible to acquire an operation input using a foot switch or a line of sight, the input information acquisition unit can acquire an operation input from the user even in a task in which it is difficult for the user to release his / her hand.

  FIG. 13 is an explanatory diagram showing the configuration of the appearance of the HMD in the modification. 13A, the image display unit 20x includes a right optical image display unit 26x instead of the right optical image display unit 26, and a left optical image display unit 28x instead of the left optical image display unit 28. I have. The right optical image display unit 26x and the left optical image display unit 28x are formed smaller than the optical member of the above-described embodiment, and are respectively disposed obliquely above the right eye and the left eye of the user when the HMD is mounted. . 13B, the image display unit 20y includes a right optical image display unit 26y instead of the right optical image display unit 26, and a left optical image display unit 28y instead of the left optical image display unit 28. I have. The right optical image display unit 26y and the left optical image display unit 28y are formed smaller than the optical member of the above-described embodiment, and are respectively disposed obliquely below the right eye and the left eye of the user when the HMD is worn. . Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is also arbitrary, and the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit completely covers the user's eye. It can also be realized as an HMD in an uncovered form.

  For example, the head mounted display is a binocular transmissive head mounted display, but may be a monocular head mounted display. Further, it may be configured as a non-transmissive head mounted display in which the transmission of the outside scene is blocked when the user wears the head mounted display.

  For example, the functional units such as the image processing unit, the display control unit, and the audio processing unit are described as being realized by the CPU developing and executing a computer program stored in the ROM or hard disk on the RAM. However, these functional units may be configured using an ASIC (Application Specific Integrated Circuit) designed to realize the function.

  For example, in the above-described embodiment, it is assumed that the image display unit is a head mounted display that is mounted like glasses, but the image display unit is a normal flat display device (liquid crystal display device, plasma display device, organic EL display device, etc. ). Also in this case, the connection between the control unit and the image display unit may be a connection via a wired signal transmission path or a connection via a wireless signal transmission path. If it does in this way, a control part can also be utilized as a remote control of a usual flat type display device.

  Further, as the image display unit, instead of the image display unit worn like glasses, an image display unit of another shape such as an image display unit worn like a hat may be adopted. The earphone may be an ear-hook type or a headband type, or may be omitted. Further, for example, it may be configured as a head-up display (HUD, Head-Up Display) mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet.

  For example, in the above embodiment, the display driving unit is configured using a backlight, a backlight control unit, an LCD, an LCD control unit, and a projection optical system. However, the above aspect is merely an example. The display drive unit may include a configuration unit for realizing another method together with or in place of these configuration units. For example, the display driving unit may include an organic EL (Organic Electro-Luminescence) display, an organic EL control unit, and a projection optical system. For example, the display driving unit can use a DMD (digital micromirror device) or the like instead of the LCD. For example, the display driving unit includes a color light source for generating each color light of RGB and a signal light modulation unit including a relay lens, a scanning optical system including a MEMS mirror, and a drive control circuit for driving them. May be configured. As described above, even if an organic EL, DMD, or MEMS mirror is used, the “emission area in the display driving unit” is still an area where image light is actually emitted from the display driving unit. By controlling the emission area in the (display drive unit) in the same manner as in the above embodiment, the same effect as in the above embodiment can be obtained. Further, for example, the display driving unit may be configured to include one or more lasers that emit a laser having an intensity corresponding to the pixel signal to the retina of the user. In this case, the “emission area in the display drive unit” represents an area in which laser light representing an image is actually emitted from the display drive unit. By controlling the emission region of the laser beam in the laser (display drive unit) in the same manner as in the above embodiment, the same effect as in the above embodiment can be obtained.

  The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

10. Control unit (controller)
DESCRIPTION OF SYMBOLS 12 ... Lighting part 14 ... Touch pad 16 ... Cross key 18 ... Power switch 20 ... Image display part 21 ... Right holding part 22 ... Right display drive part 23 ... Left holding part 24 ... Left display drive part 26 ... Right optical image display part 28 ... Left optical image display unit 30 ... Earphone plug 32 ... Right earphone 34 ... Left earphone 40 ... Connection unit 42 ... Right cord 44 ... Left cord 46 ... Connecting member 48 ... Body code 51 ... Transmission unit 52 ... Transmission unit 53 ... Reception 54: Receiver 61 ... Camera 62 ... Depth sensor 100 ... Head-mounted display device (HMD)
DESCRIPTION OF SYMBOLS 110 ... Input information acquisition part 120 ... Memory | storage part 130 ... Power supply 132 ... Wireless communication part 140 ... CPU
DESCRIPTION OF SYMBOLS 160 ... Image processing part 162 ... Display control part 164 ... Superimposed image display control part 170 ... Sound processing part 180 ... Interface 201 ... Right backlight control part 202 ... Left backlight control part 211 ... Right LCD control part 212 ... Left LCD control Part 221 ... Right backlight 222 ... Left backlight 241 ... Right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate

Claims (10)

A head-mounted display device provided with a display unit capable of visually observing the outside world,
A superimposed image display control unit that displays a predetermined image in association with the position of an object in the outside world that is visible through the display unit;
A position information acquisition unit that acquires position information indicating the position of the object in a two-dimensional space;
A depth information acquisition unit that acquires depth information indicating the depth of each part around at least the object in the outside world;
With
The superimposed image display control unit
Based on the position information and the depth information, calculate the position of the object in a three-dimensional space, and based on the position of the object, determine a display area in the three-dimensional space for the predetermined image,
Based on the information representing the display area and the depth information, it is determined whether or not an object exists between the display area and the display unit in the three-dimensional space,
A head that displays the predetermined image by excluding from the display area a predetermined range including at least an area where the object overlaps the display area in the depth direction when the object is determined to exist. Part-mounted display device. The head-mounted display device according to claim 1,
The head-mounted display device, wherein the predetermined image is extended information as augmented reality to be given to the object. The head-mounted display device according to claim 1 or 2,
The position information acquisition unit is a head-mounted display device including a camera that images the object. The head-mounted display device according to any one of claims 1 to 3,
The depth information acquisition unit is a head-mounted display device including a depth sensor that outputs a signal indicating the depth of the outside world. The head-mounted display device according to any one of claims 1 to 4,
The superimposed image display control unit
The position information acquired by the position information acquisition unit and the depth information acquired by the depth information acquisition unit are converted into values expressed in a coordinate system viewed through the display unit, and the converted position information and depth A head-mounted display device that determines the display area based on information. A head-mounted display device provided with a display unit capable of visually observing the outside world,
A superimposed image display control unit that causes the display unit to display a first image in association with the position of an object in the outside world that is visible through the screen;
A position information acquisition unit that acquires position information indicating the position of the object in a two-dimensional space;
A depth information acquisition unit that acquires depth information indicating the depth of each part around at least the object in the outside world;
With
The superimposed image display control unit
Based on the position information and the depth information, the position of the object in a three-dimensional space is calculated, and on the basis of the position of the object, a display area in the three-dimensional space for the first image, As the first display area,
Obtaining a display area in the three-dimensional space for a second image different from the first image as a second display area;
Whether the second display area exists between the first display area and the display unit in the three-dimensional space based on the information representing the first display area and the information representing the second display area Determine whether or not
When it is determined that the second display area exists, a predetermined range including at least an area where the second display area overlaps the first display area in the depth direction is excluded from the first display area. A head-mounted display device that displays the first image. A method for controlling a head-mounted display device including a display unit capable of visually recognizing the outside world,
A step of displaying a predetermined image in association with the position of an object in the outside world that is visible through the display unit;
Obtaining position information indicating the position of the object in a two-dimensional space;
Obtaining depth information indicating the depth of each part around at least the object in the outside world;
With
The step of displaying the predetermined image includes
Based on the position information and the depth information, calculate the position of the object in a three-dimensional space, and based on the position of the object, determine a display area in the three-dimensional space for the predetermined image,
Based on the information representing the display area and the depth information, it is determined whether or not an object exists between the display area and the display unit in the three-dimensional space,
A head that displays the predetermined image by excluding from the display area a predetermined range including at least an area where the object overlaps the display area in the depth direction when the object is determined to exist. Part-mounted display device control method. A method for controlling a head-mounted display device including a display unit capable of visually recognizing the outside world,
A step of causing the display unit to display a first image in association with the position of an object in the outside world that is seen through.
Obtaining position information indicating the position of the object in a two-dimensional space;
Obtaining depth information indicating the depth of each part around at least the object in the outside world;
With
The step of displaying the first image includes:
Based on the position information and the depth information, the position of the object in a three-dimensional space is calculated, and on the basis of the position of the object, a display area in the three-dimensional space for the first image, As the first display area,
Obtaining a display area in the three-dimensional space for a second image different from the first image as a second display area;
Whether the second display area exists between the first display area and the display unit in the three-dimensional space based on the information representing the first display area and the information representing the second display area Determine whether or not
When it is determined that the second display area exists, a predetermined range including at least an area where the second display area overlaps the first display area in the depth direction is excluded from the first display area. A method for controlling a head-mounted display device that displays the first image. A computer program for controlling a head-mounted display device provided with a display unit capable of seeing through the outside world,
A function of causing the display unit to display a predetermined image in association with the position of an object in the outside world that is visible through the screen;
A function of acquiring position information indicating the position of the object in a two-dimensional space;
A function of acquiring depth information indicating the depth of each part around at least the object in the outside world;
To the computer,
The function of displaying the predetermined image is as follows:
Based on the position information and the depth information, calculate the position of the object in a three-dimensional space, and based on the position of the object, determine a display area in the three-dimensional space for the predetermined image,
Based on the information representing the display area and the depth information, it is determined whether or not an object exists between the display area and the display unit in the three-dimensional space,
A computer that, when it is determined that the object exists, removes a predetermined range including at least an area where the object overlaps the display area in the depth direction from the display area and displays the predetermined image; program. A computer program for controlling a head-mounted display device provided with a display unit capable of seeing through the outside world,
A function of causing the display unit to display a first image in association with a position of an object in the outside world that is visible through the screen;
A function of acquiring position information indicating the position of the object in a two-dimensional space;
A function of acquiring depth information indicating the depth of each part around at least the object in the outside world;
To the computer,
The function of displaying the first image is as follows:
Based on the position information and the depth information, the position of the object in a three-dimensional space is calculated, and on the basis of the position of the object, a display area in the three-dimensional space for the first image, As the first display area,
Obtaining a display area in the three-dimensional space for a second image different from the first image as a second display area;
Whether the second display area exists between the first display area and the display unit in the three-dimensional space based on the information representing the first display area and the information representing the second display area Determine whether or not
When it is determined that the second display area exists, a predetermined range including at least an area where the second display area overlaps the first display area in the depth direction is excluded from the first display area. A computer program for displaying the first image.

Images