JP2016186658A - Head-mount type display device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an optical transmission type head-mount type display device, in which the sense of discomfort in realizing the augmented reality is reduced.SOLUTION: A head-mount type display device that allows a user to see the virtual image and the outside scenery at the same time includes: an imaging unit that images the outside scenery and acquires the outside scenery image; an augmented reality processing unit that analyzes the acquired outside scenery image and generates the additional image data for the augmented reality in accordance with the analysis result; a display element that generates the image light using the generated additional image data; and a light guide unit that forms a virtual image before the eyes of the user with the generated image light. The augmented reality processing unit acquires an overlapping region where the display region of the virtual image and the imaging region of the imaging unit overlap, and generates the additional image data using the overlapping region.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a head-mounted display device and a method for controlling the head-mounted display device.

  There is known a technique called augmented reality (AR) that adds information to a real environment using a computer. Augmented reality is mounted on smartphones and head-mounted displays (hereinafter referred to as “head-mounted display devices”). FIG. 1 is an explanatory diagram for explaining augmented reality. FIG. 1A shows a state in which augmented reality is realized by a non-transmissive head-mounted display device. The head-mounted display device captures an outside scene with a camera, recognizes an image CP obtained by the imaging, and generates information (object OB in the example in the figure) for additional presentation. The head-mounted display device displays an image obtained by superimposing the captured image CP and the generated object OB on the liquid crystal screen. As a result, the user of the head-mounted display device can experience augmented reality. In addition, although the example which implement | achieves augmented reality using a non-transmissive head-mounted display apparatus was demonstrated here, it can implement | achieve also with a smart phone. For example, Patent Document 1 discloses a non-transmissive head-mounted display device equipped with augmented reality technology.

JP 2011-203823 A JP 2010-146481 A

  FIG. 1B shows how augmented reality is realized by an optically transmissive head-mounted display device. In the optically transmissive head-mounted display device, as in the non-transmissive type, the image CP obtained by imaging is recognized and information (object OB) for additional presentation is generated. Thereafter, in the optically transmissive head-mounted display device, only the generated object OB is displayed on the liquid crystal screen. The user can experience augmented reality by visually recognizing both the object OB displayed on the liquid crystal screen and displayed as the virtual image VI and the outside scene SC that can be seen through the front lens. Note that the area obtained by imaging shown in the first image from the left in FIG. 1B is also called “imaging field of view”, and the area displayed as the virtual image VI shown in the second image from the left in FIG. The area that can be visually recognized as the outside scene SC shown as the third image from the left in FIG. 1B is also referred to as “real field of view”.

  Here, the size of the region where the virtual image VI is projected in front of the user's eyes (the portion where the real field of view and the video field of view overlap) may not match the size of the region that can be imaged by the camera (imaging field of view). If the sizes of the two do not match, the optically transmissive head-mounted display device uses a shift between the object OB displayed as the virtual image VI and the outside scene SC that can be seen through the front lens. There was a problem of feeling uncomfortable. Specifically, for example, when a region (video field of view) that can be captured by a camera is vast, in a conventional optically transmissive head-mounted display device, an object OB corresponding to the vast range is generated, It is displayed as a virtual image VI in front of the user's eyes. In such a case, the user feels uncomfortable due to a crease that occurs between the outside scene SC in the region where the virtual image VI is projected in front of his / her eyes (the portion where the real view and the video view overlap) and the object OB. End up.

  An object of the present invention is to reduce a sense of incongruity that occurs when an augmented reality is realized in an optically transmissive head-mounted display device.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms or application examples.

[Application Example 1]
A head-mounted display device that allows a user to view a virtual image and an outside scene at the same time,
An imaging unit that captures an outside scene and obtains an outside scene image;
An augmented reality processing unit that analyzes the acquired outside scene image and generates additional image data for augmented reality according to a result of the analysis;
A display element that generates image light using the generated additional image data;
A light guide unit that forms a virtual image in front of the user's eyes using the generated image light; and
With
The augmented reality processing unit obtains an overlapping region in which the virtual image display region and the imaging region of the imaging unit overlap in the outside scene image, and generates the additional image data using the overlapping region. Part-mounted display device.
With such a configuration, the augmented reality processing unit acquires an overlapping region where the display region of the virtual image and the imaging region of the imaging unit overlap, and generates additional image data using the overlapping region. The uncomfortable feeling that occurs when the unit realizes augmented reality can be reduced.

[Application Example 2]
The head-mounted display device according to Application Example 1,
A storage unit storing guide image data for indicating a display area of the virtual image for the user;
The augmented reality processing unit generates image light using the guide image data, guides the user to indicate a predetermined mark in the display area of the virtual image, and acquires and acquires the outside scene image. A head-mounted display device that acquires a position of the mark obtained by analyzing the outside scene image as a boundary between the overlapping region and another region.
With such a configuration, the augmented reality processing unit generates image light using the guide image data, and guides the user to indicate a predetermined mark in the display area of the virtual image, thereby acquiring an outside scene image. The position of the landmark obtained by analyzing the acquired outside scene image is acquired as a boundary between the overlapping region and another region. As a result, the augmented reality processing unit can acquire an overlapping region in which the virtual image display region and the imaging region of the imaging unit overlap.

[Application Example 3]
A head-mounted display device according to Application Example 2,
The guide image data is a rectangular shape having the same aspect ratio as the display element, and includes guide marks arranged at at least two of the four corners,
The augmented reality processing unit is a head-mounted display device that guides the user to indicate the mark by pointing the guide mark with a finger.
With such a configuration, the augmented reality processing unit guides the user to indicate the mark by pointing the guide mark with a finger, so that no special device is required and the user can easily understand. With a simple operation, a predetermined mark can be given to the display area of the virtual image.

[Application Example 4]
A head-mounted display device according to Application Example 2,
The guide image data is a rectangular shape having the same aspect ratio as the display element, and includes frame-shaped guide marks arranged at at least two of the four corners,
The augmented reality processing unit guides the user so as to show the mark by arranging an object having a rectangular side surface in the frame of the guide mark. .
With such a configuration, the augmented reality processing unit guides the user so as to show the mark by arranging an object having a rectangular side surface in the guide mark frame. The mark can be given to the display area of the virtual image by the above operation.

[Application Example 5]
The head-mounted display device according to any one of Application Examples 1 to 4, further comprising:
A second imaging unit that is disposed at a position corresponding to the right eye or temple of the user or the periphery of the user when the head-mounted display device is mounted, and that captures an external scene and obtains a second external scene image; ,
A third imaging unit that is arranged at a position corresponding to the left eye or temple of the user or the periphery of the user when the head-mounted display device is worn, and that captures the outside scene and obtains a third outside scene image; ,
With
The augmented reality processing unit further obtains a deviation amount between the specific object included in the second outside scene image and the specific object included in the third outside scene image, and calculates the deviation. A head-mounted display device that acquires information about the depth of the object in the outside world from a quantity.
With such a configuration, the augmented reality processing unit further includes the second imaging arranged at a position corresponding to the right eye or the temple of the user or the periphery thereof when the head-mounted display device is worn. A third object disposed at a position corresponding to the specific object included in the second outside scene image photographed by the unit and the left eye or the temple of the user when the head-mounted display device is worn or the periphery thereof. The amount of deviation between the specific object included in the third outside scene image photographed by the imaging unit and the depth of the object in the outside world can be obtained from the obtained amount of deviation.

  The present invention can be realized in various modes. For example, a head-mounted display device, a method for controlling a head-mounted display device, an image display system, and a function of these methods, devices, or systems Can be realized in the form of a computer program for realizing the above, a recording medium on which the computer program is recorded, and the like.

It is explanatory drawing for demonstrating augmented reality. It is explanatory drawing which shows the structure of the external appearance of the head mounted display apparatus in one Example of this invention. It is a block diagram which shows the composition of a head mount display functionally. It is explanatory drawing which shows a mode that image light is inject | emitted by the image light production | generation part. It is explanatory drawing which shows an example of the virtual image recognized by the user. It is a flowchart which shows the procedure of an area | region determination process. It is explanatory drawing for demonstrating area | region determination processing. It is explanatory drawing which shows an example of the outside scene image imaged with the camera. It is a flowchart which shows the procedure of the area | region determination process in 2nd Example. It is explanatory drawing for demonstrating the area | region determination process in 2nd Example. It is explanatory drawing which shows the structure of the external appearance of the head mounted display apparatus in 3rd Example. It is explanatory drawing for demonstrating the depth information acquired by a depth acquisition process. It is explanatory drawing for demonstrating a depth acquisition process.

  Next, embodiments of the present invention will be described based on examples.

A. First embodiment:
(A-1) Configuration of the head-mounted display device:
FIG. 2 is an explanatory diagram showing an external configuration of a head-mounted display device according to an embodiment of the present invention. The head-mounted display device HM is a head-mounted display device mounted on the head, and is also called a head mounted display (HMD). The head-mounted display HM of the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene.

  The head-mounted display HM includes an image display unit 20 that allows the user to visually recognize a virtual image while being mounted on the user's head, and a control unit (controller) 10 that controls the image display unit 20.

  The image display unit 20 is a wearing body that is worn on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. Including. The right optical image display unit 26 and the left optical image display unit 28 are disposed at positions corresponding to the right and left eyes of the user when the head mounted display HM is mounted. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are respectively connected at positions corresponding to the eyebrows of the user when the head mounted display HM is mounted. The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28.

  The right holding unit 21 is substantially perpendicular to the right optical image display unit 26 from the end ER of the right optical image display unit 26 to a position corresponding to the user's temporal region when the head mounted display HM is mounted. It is a member provided by being stretched. Similarly, the left holding unit 23 is substantially perpendicular to the left optical image display unit 28 from the end EL of the left optical image display unit 28 to a position corresponding to the user's temporal region when the head mounted display HM is mounted. It is the member provided by extending | stretching so that. The right holding unit 21 and the left holding unit 23 hold the head mounted display HM on the user's head like a temple of glasses.

  The right display drive unit 22 is located on the inner side of the right holding unit 21, in other words, on the side facing the user's head when the head mounted display HM is mounted, on the end ER side of the right optical image display unit 26. Is arranged. The left display driving unit 24 is disposed inside the left holding unit 23 and on the end EL side of the left optical image display unit 28. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”, and the right optical unit. The image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

  The display driving unit includes an LCD (Liquid Crystal Display), a projection optical system, and the like (not shown). Details will be described later. The optical image display unit as an optical member includes a light guide plate (not shown) and a light control plate. The light guide plate is formed of a light transmissive resin material or the like, and emits image light taken from the display driving unit toward the user's eyes. The light control plate is a thin plate-like optical element, and is disposed so as to cover the front side of the head mounted display HM (the side opposite to the user's eye side). The light control plate protects the light guide plate, suppresses damage to the light guide plate, adhesion of dirt, etc., and adjusts the light transmittance of the light control plate to adjust the amount of external light entering the user's eyes. The ease of visual recognition of the virtual image can be adjusted. The light control plate can be omitted.

  The camera 61 is disposed at a position corresponding to the eyebrow of the user when the head mounted display HM is attached. The camera 61 captures an outside scene image (external scenery) in the front side direction of the head mounted display HM, in other words, the direction opposite to the user's eye side. In this embodiment, the camera 61 corresponds to an “imaging unit”. The camera 61 in this embodiment is exemplified as a monocular camera, but a stereo camera may be adopted.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48 connected to the control unit 10, a right cord 42 in which the main body cord 48 branches into two, a left cord 44, and a connecting member 46 provided at the branch point. Yes. The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. Each end of the main body cord 48 opposite to the connecting member 46 and the control unit 10 are provided with connectors (not shown) that are fitted to each other. The connector of the main body cord 48 and the control unit 10 The control unit 10 and the image display unit 20 are connected to or disconnected from each other by the fitting / releasing of the connector. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for operating the head mounted display HM. The control unit 10 includes a lighting unit 12, a touch pad 14, a cross key 16, and a power switch 18. The lighting unit 12 notifies the operation state of the head mounted display HM (for example, ON / OFF of the power supply) according to the light emission state. For example, an LED (Light Emitting Diode) can be used as the lighting unit 12. The touch pad 14 detects a user's finger operation on the operation surface of the touch pad 14 and outputs a signal corresponding to the detected content. The cross key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power-on state of the head mounted display HM by detecting a slide operation of the switch.

  FIG. 3 is a block diagram functionally showing the configuration of the head mounted display HM. The control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a CPU 140, an interface 180, and transmission units (Tx) 51 and 52, and these units are connected to each other via a bus (not shown). Has been.

  The input information acquisition unit 110 has a function of acquiring a signal corresponding to an operation input by a user such as an operation input to the touch pad 14, the cross key 16, and the power switch 18, for example. The storage unit 120 is a storage unit including a ROM, a RAM, a DRAM, a hard disk, etc. (not shown). The storage unit 120 includes a coordinate information storage unit 122. The coordinate information storage unit 122 is a storage region for storing coordinates obtained in the region determination process executed by the AR processing unit 142. The power supply 130 supplies power to each part of the head mounted display HM. As the power supply 130, for example, a secondary battery can be used.

  The CPU 140 provides a function as an operating system (OS) 150 by executing a program installed in advance. The CPU 140 also functions as an AR processing unit 142, an image processing unit 160, an audio processing unit 170, and a display control unit 190 by developing firmware and computer programs stored in a ROM and a hard disk in a RAM and executing them. To do.

  The AR processing unit 142 executes processing for realizing augmented reality (hereinafter, also referred to as “AR processing”) using a processing start request from the OS 150 or a specific game application as a trigger. In the AR processing, the AR processing unit 142 analyzes the outside scene image acquired by the camera 61, and generates additional image data for augmented reality according to the analysis result. Since an outside scene image analysis method and additional image data generation method are well known, description thereof will be omitted. Further, the AR processing unit 142 executes an area determination process prior to the AR process. The area determination process is a process for determining an analysis range when “analyzing an outside scene image” in the AR process. Details will be described later. The AR processing unit 142 is also referred to as an “augmented reality processing unit”.

  The image processing unit 160 generates a clock signal PCLK, a vertical synchronization signal VSync, a horizontal synchronization signal HSync, and image data Data based on the additional image data generated in the AR processing, content input via the interface 180, and the like. Then, these signals are supplied to the image display unit 20 via the connection unit 40.

  When supplying based on the designated additional image data, the image processing unit 160 acquires display settings (VSync, HSync, etc.) that are predetermined as default values and stored in the storage unit 120. The image processing unit 160 generates a clock signal PCLK using a PLL (Phase Locked Loop) circuit (not shown) or the like according to the acquired display setting. Thereafter, the image processing unit 160 stores the additional image data Data (RGB data) in the DRAM in the storage unit 120.

  On the other hand, when supplying based on content, the image processing unit 160 acquires an image signal included in the content. For example, in the case of a moving image, the acquired image signal is generally an analog signal composed of 30 frame images per second. The image processing unit 160 separates synchronization signals such as the vertical synchronization signal VSync and the horizontal synchronization signal HSync from the acquired image signal. Further, the image processing unit 160 generates a clock signal PCLK using a PLL circuit (not shown) or the like according to the period of the separated vertical synchronization signal VSync and horizontal synchronization signal HSync. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like (not shown). Thereafter, the image processing unit 160 stores the converted digital image signal as target image data Data (RGB data) in the DRAM in the storage unit 120 for each frame.

  When the additional image data or content is given in a digital format, the clock signal PCLK is output in synchronization with the image signal. Therefore, the A / D conversion of the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the analog image signal is performed. It is unnecessary. Hereinafter, the additional image data Data and the target image data Data are also collectively referred to as “image data Data”. The image processing unit 160 performs image processing such as various tone correction processing such as resolution conversion processing, brightness and saturation adjustment, and keystone correction processing on the image data Data stored in the storage unit 120 as necessary. May be executed.

  The image processing unit 160 transmits the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync generated as described above, and the image data Data stored in the DRAM in the storage unit 120 to the transmission units 51 and 52. To send each via. Note that the image data Data transmitted through the transmission unit 51 is also referred to as “right-eye image data Data1”, and the image data Data transmitted through the transmission unit 52 is also referred to as “left-eye image data Data2”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control unit 10 and the image display unit 20.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 uses the control signal to turn on / off the right LCD 241 by the right LCD control unit 211, turn on / off the right backlight 221 by the right backlight control unit 201, and control the left LCD. The left display drive unit 22 and the left display drive unit 24 are controlled by individually controlling the ON / OFF of the left LCD 242 by the unit 212 and the ON / OFF of the left backlight 222 by the left backlight control unit 202. Each controls the generation and emission of image light. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light.

  The display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals for the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and applies to a speaker (not shown) in the right earphone 32 and a speaker (not shown) in the left earphone 34 of the image display unit 20. Supplied via the connection 40. For example, when a Dolby (registered trademark) system is adopted, processing is performed on an audio signal, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, respectively.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer PC, a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, a wireless LAN interface, or the like can be provided.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. And a right earphone 32 and a left earphone 34.

  The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, a right LCD control unit 211 that functions as a display element, and a right An LCD 241 and a right projection optical system 251 are included. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are also collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 has a function of driving the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 has a function of driving the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data1 input via the reception unit 53. Have The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  FIG. 4 is an explanatory diagram illustrating a state in which image light is emitted by the image light generation unit. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal corresponding to each pixel position arranged in a matrix, thereby changing the illumination light IL emitted from the right backlight 221. It has a function of modulating into effective image light PL representing an image. As shown in FIG. 4, in this embodiment, the backlight method is adopted. However, the image light may be emitted using a front light method or a reflection method.

  The right projection optical system 251 in FIG. 3 is configured by a collimating lens that converts image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The right projection optical system 251 and the right light guide plate 261 are collectively referred to as “light guide unit”. In addition, as long as a light guide part forms a virtual image in front of a user's eyes using image light, arbitrary systems can be used, for example, a diffraction grating may be used and a transflective film is used. Also good.

  The left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, a left LCD control unit 212 and a left that function as a display element. An LCD 242 and a left projection optical system 252 are included. The left backlight control unit 202, the left LCD control unit 212, the left backlight 222, and the left LCD 242 are collectively referred to as an “image light generation unit”, the left projection optical system 252, and the left light guide plate 262. Are also collectively referred to as “light guides”. The right display drive unit 22 and the left display drive unit 24 are paired. Since each unit of the left display drive unit 24 has the same configuration and operation as each unit described in the right display drive unit 22, a detailed description will be given. Omitted.

  FIG. 5 is an explanatory diagram illustrating an example of a virtual image recognized by the user. As described above, the image light guided to both eyes of the user of the head mounted display HM forms an image on the retina of the user, so that the user can visually recognize the virtual image. As shown in FIG. 5, a virtual image VI is displayed in the visual field VR of the user of the head mounted display HM. In addition, regarding the portion of the user's visual field VR where the virtual image VI is displayed, the user can see the outside scene SC via the virtual image VI of the optical image display unit. Further, the user can directly view the outside scene SC through the optical image display unit except for the portion of the user's visual field VR where the virtual image VI is displayed.

  The area where the virtual image VI is displayed is also referred to as “video field of view”. In addition, an area in which the user can visually recognize the outside scene SC, in other words, an area in the visual field VR is also referred to as “real field of view”. At this time, it can be said that the area where the virtual image VI is projected in front of the user's eyes is a portion where the real view and the image view overlap.

(A-2) Area determination processing:
FIG. 6 is a flowchart showing the procedure of the area determination process. FIG. 7 is an explanatory diagram for explaining the region determination processing. The area determination process is a process executed by the AR processing unit 142 prior to the AR process, and is a process for determining an analysis range when “analyzing an outside scene image” in the AR process. The area determination process may operate as a subroutine of the AR process.

  First, the AR processing unit 142 initializes a variable by setting “1” to a variable n used in the region determination process (step S100). Next, the AR processing unit 142 displays the guide image as a virtual image (step S102). FIG. 7A is an explanatory diagram illustrating an example of the guide image CG. The guide image CG is an image for showing the end of the virtual image display area to the user of the head mounted display HM. The guide image CG in this embodiment is a rectangular shape having the same aspect ratio as that of the right LCD 241 and the left LCD 242 (display element) and has a white background, and has four corners and portions located at intersections of diagonal lines. In addition, circular guide marks P1 to P5 are provided. Image data of the guide image CG (hereinafter also referred to as “guide image data”) is stored in the storage unit 120 in advance.

  In step S <b> 102, the AR processing unit 142 reads the guide image data stored in the storage unit 120 and transmits the guide image data to the image processing unit 160. The image processing unit 160 that has received the guide image data executes the process described with reference to FIG. As a result, the guide image CG of FIG. 7A is displayed as the virtual image VI in the visual field VR of the user of the head mounted display HM.

  In step S104, the AR processing unit 142 guides the user to point to the nth guide mark in the guide image CG displayed as the virtual image VI. Here, the “n-th point” varies according to the value of the variable n initialized in step S100. Further, the AR processing unit 142 may display a message using a dialog box or the like as a guidance method for the user, or may perform guidance by voice. Voice guidance is preferable because guidance can be provided without blocking the currently displayed guide image CG.

  FIG. 7B shows a situation where the user points to the first guide mark P1 of the guide image CG displayed as a virtual image VI in his / her visual field VR. In FIG. 7B, for the convenience of illustration, the illustration of the outside scene SC that passes through the optical image display unit is omitted. After guidance to the user, the AR processing unit 142 causes the camera 61 to capture an outside scene image in step S106 of FIG. FIG. 8 is an explanatory diagram illustrating an example of an outside scene image CP captured by the camera 61.

  In step S108, the AR processing unit 142 recognizes the captured outside scene image CP and identifies the position of the human fingertip. The AR processing unit 142 acquires the coordinates of the fingertip position and stores them in the coordinate information storage unit 122 (step S110). Specifically, the AR processing unit 142 acquires the coordinates of the position of the fingertip specified in step S108. This coordinate can be defined as, for example, the amount of movement in the X and Y directions when the pixel in the upper left part of the outside scene image CP is (0, 0). The AR processing unit 142 stores the acquired coordinates and the value of the variable n in the coordinate information storage unit 122 in association with each other.

  The AR processing unit 142 determines whether or not the value of the variable n is equal to or greater than the number of guide marks arranged in the guide image CG (Step S112). When the value of the variable n is less than the number of guide marks (step S112: NO), the AR processing unit 142 increments the variable n and shifts the process to step S104. Then, the AR processing unit 142 repeats a series of processes of acquiring the outside scene image CP, acquiring the coordinates of the fingertip position, and storing the coordinates for the next guidance mark. On the other hand, when the value of the variable n is equal to or greater than the number of guide marks (step S112: YES), the AR processing unit 142 transmits a request to end the display of the guide image to the image processing unit 160 and the display control unit 190. (Step S116), the process is terminated.

  In this way, the AR processing unit 142 displays the guide image virtual image VI in the user's field of view in a state where the guide image is displayed in the full displayable area of the right LCD 241 and the left LCD 242 (display element). The coordinates indicating which position of the outside scene image CP imaged by the camera 61 corresponds to the guide marks P1 to P4 at the four corners of FIG. In other words, in the area determination process, the AR processing unit 142 acquires in advance an area where the imaging area of the camera 61 (imaging unit) and the display area of the virtual image VI overlap, that is, the area EA in FIG. A region where the imaging region of the camera 61 and the display region of the virtual image VI overlap is also referred to as an “overlapping region”.

  As described above, according to the first embodiment, the AR processing unit 142 (augmented reality processing unit) acquires the overlapping area EA prior to the AR processing. In the subsequent AR processing, the AR processing unit 142 uses the acquired overlapping area EA, specifically, by setting an image in the overlapping area EA in the outside scene image CP captured by the camera 61 as an analysis target. Generate additional image data for augmented reality. Thereby, the size of the area (video field of view) that can be captured by the camera 61 and the size of the area projected as the virtual image VI in front of the user's eyes (the part where the real field of view and the video field of view overlap) are made to coincide in a pseudo manner. (Calibrate). As a result, it is possible to suppress the occurrence of a deviation between the additional presentation information (object OB) displayed in the AR process and the outside scene SC that the user sees through the optical image display unit in front of him. In the optical transmission type head mounted display HM (head-mounted display device), it is possible to reduce a sense of incongruity that occurs when the AR processing unit 142 realizes augmented reality.

  In the above embodiment, the “limit analysis target” method is shown as the method of using the overlapping area EA when the AR processing unit 142 generates the additional image data. However, the above is only an example, and the overlapping area EA can be used in various ways. For example, the AR processing unit 142 analyzes all the outside scene images CP captured by the camera 61, and displays a display method of information for additional presentation within the overlapping area EA and information for additional presentation outside the overlapping area EA. The changed additional image data may be generated. Specifically, the information for additional presentation in the overlapping area EA is displayed in a form in which the contents can be recognized, and the information for additional presentation outside the overlapping area EA is displayed in an aspect indicating only the presence with an icon such as an arrow. be able to.

  The AR processing unit 142 (augmented reality processing unit) generates image light using the guide image data and guides the user to indicate a predetermined mark in the display area of the virtual image VI. CP is acquired, and the position of the mark obtained by analyzing the acquired outside scene image CP is acquired as a boundary between the overlapping area EA and another area. As a result, the AR processing unit 142 can acquire an overlapping area EA in which the display area of the virtual image VI and the imaging area of the camera 61 overlap.

  Further, since the AR processing unit 142 (augmented reality processing unit) guides the user to indicate the mark by pointing the guide marks P1 to P5 with a finger, no special device is required and the user It is possible to give a predetermined mark to the display area of the virtual image VI by a simple operation that is easy to understand.

B. Second embodiment:
In the second embodiment of the present invention, a configuration in which the overlapping area acquisition method in the area determination process is different will be described. Below, only the part which has a different structure and operation | movement from 1st Example is demonstrated. In the figure, the same components as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment described above, and detailed description thereof is omitted.

(B-1) Configuration of the head-mounted display device:
The configuration of the head mounted display HMa in the second embodiment is substantially the same as that of the first embodiment described with reference to FIGS.

  However, the head mounted display HMa in the second embodiment includes an AR processing unit 142a instead of the AR processing unit 142, but is different from the first example. Further, the head mounted display HMa in the second embodiment is different from the first embodiment in the content of the guide image data stored in the storage unit 120 and the content of the region determination process.

(B-2) Area determination processing:
FIG. 9 is a flowchart showing the procedure of region determination processing in the second embodiment. FIG. 10 is an explanatory diagram for explaining the region determination processing in the second embodiment. Similarly to the first embodiment, the area determination process of the second embodiment is a process executed by the AR processing unit 142a prior to the AR process, and an analysis range when “analyzing an outside scene image” is performed in the AR process. It is a process for determining.

  First, the AR processing unit 142a displays a guide image (step S202). FIG. 10A is an explanatory diagram illustrating an example of the guide image CGa. The guide image CGa is an image for showing the end of the virtual image display area to the user of the head mounted display HM. The guide image CGa in the present embodiment is a rectangular shape having the same aspect ratio as the right LCD 241 and the left LCD 242 (display element), and has a white background, and has frame-shaped guide marks P1 to P4 at the four corners. doing. Image data of the guide image CGa is stored in the storage unit 120 in advance.

  In step S200, the AR processing unit 142a reads the guide image data stored in the storage unit 120, and transmits the guide image data to the image processing unit 160. In the image processing unit 160 that has received the guide image data, the processing described in FIG. 3 is executed, and the guide image CGa in FIG. 10A is displayed as a virtual image VI in the visual field VR of the head mounted display HMa.

  In step S202, the AR processing unit 142a guides the user to align the paper within the frame of the guide image CGa displayed as the virtual image VI. The guidance method may be a message using a dialog box or the like, or may be voice guidance.

  FIG. 10B shows how the user aligns the paper PP within the guide marks P1 to P4 of the guide image CGa displayed as a virtual image VI in his / her visual field VR. In FIG. 10B, for the convenience of illustration, the illustration of the outside scene SC that passes through the optical image display unit is omitted. After guidance to the user, the AR processing unit 142a causes the camera 61 to capture an outside scene image in step S204 of FIG.

  In step S206, the AR processing unit 142a recognizes the captured outside scene image and specifies the boundary between the paper and the outside scene. The AR processing unit 142a acquires the coordinates of the positions of the vertices of the rectangle among the specified boundaries, and stores them in the coordinate information storage unit 122 (step S208). Thereafter, the AR processing unit 142a transmits a request to end the display of the guide image to the image processing unit 160 and the display control unit 190 (Step S210), and ends the process.

  As described above, also in the region determination processing of the second embodiment, the AR processing unit 142a (augmented reality processing unit) overlaps the imaging region of the camera 61 (imaging unit) and the display region of the virtual image VI. Can be acquired. For this reason, also in the structure of 2nd Example, the effect similar to 1st Example can be acquired.

  Further, the AR processing unit 142a (augmented reality processing unit) places a mark on the user by arranging an object (for example, paper) having a rectangular side surface in the frame of the guide marks P1 to P4. In order to guide as shown, a mark can be given to the display area of the virtual image VI by a single operation.

C. Third embodiment:
In the third embodiment of the present invention, a configuration capable of executing depth acquisition processing in addition to region determination processing will be described. The depth acquisition process is a process of acquiring “depth information” of the external environment, and is executed by the AR processing unit 142b. The AR processing unit 142b can realize augmented reality in consideration of the depth of the external environment by using the acquired depth information when generating additional image data in the AR processing. Below, only the part which has a different structure and operation | movement from 1st Example is demonstrated. In the figure, the same components as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment described above, and detailed description thereof is omitted.

(C-1) Configuration of the head-mounted display device:
FIG. 11 is an explanatory diagram showing an external configuration of the head-mounted display device according to the third embodiment. The difference from the first embodiment shown in FIGS. 1 to 5 (particularly FIG. 2) is that a control unit 10b is provided instead of the control unit 10 and an image display unit 20b is provided instead of the image display unit 20. It is. The control unit 10b includes an AR processing unit 142b instead of the AR processing unit 142. The AR processing unit 142b executes depth acquisition processing in addition to the region determination processing described in the first embodiment. The image display unit 20b further includes a camera 62 and a camera 63.

  The camera 62 is disposed at a position corresponding to the right eye or temple of the user or the periphery thereof when the head mounted display HMb is mounted. The camera 63 is disposed at a position corresponding to the user's left eye or temple or the periphery thereof when the head mounted display HMb is mounted. Similarly to the camera 61, the camera 62 and the camera 63 capture an outside scene in the direction opposite to the user's eye side, and obtain an outside scene image. The outside scene image captured by the camera 62 and the camera 63 is used in a depth acquisition process described later. In this embodiment, the camera 62 corresponds to a “second imaging unit”, and the camera 63 corresponds to a “third imaging unit”.

(C-2) Area determination processing:
The procedure of the area determination process in the third embodiment is the same as that in the first embodiment described with reference to FIGS.

(C-3) Depth acquisition processing:
FIG. 12 is an explanatory diagram for describing the depth information acquired in the depth acquisition process. The depth information is information relating to the depth of an object existing in the external environment of the user wearing the head mounted display HMb. For example, as shown in FIG. 12, when the user of the head mounted display HMb looks at an apple that is present in the front, there is a difference in the visible image between the right eye RE and the left eye LE of the user. Hereinafter, this difference is also referred to as “binocular parallax”, and is expressed as an angle of convergence θ1 between the apple and the user's right eye RE and left eye LE. The binocular parallax is generated, so that the user can perceive the depth of the apple.

On the other hand, when the user of the head mounted display HMb sees a flower that exists farther than the apple, the convergence angle θ2 formed by the flower and the user's right eye RE and left eye LE is smaller than the convergence angle θ1. Become. That is, in FIG. 12, when the distance between the user's eyes and the apple is the depth distance L1, and the distance between the user's eyes and the flower is the depth distance L2, binocular parallax θ1, θ2 And the depth distances L1 and L2 are
θ1> θ2 and L1 <L2
Satisfy the relationship. In other words, the magnitude of the binocular parallax θ is inversely proportional to the depth distance L between the user's eyes and the object. Therefore, in the depth acquisition process, at least one of binocular parallax θ or depth distance L is acquired as “depth information”.

  FIG. 13 is an explanatory diagram for explaining the depth acquisition process. In the depth determination process, the AR processing unit 142b first causes the camera 62 and the camera 63 to capture an outside scene image. The upper part of FIG. 13 shows an outside scene image CP62 captured by the camera 62 provided at a position corresponding to the right eye of the user. The lower part of FIG. 13 shows an outside scene image CP63 captured by the camera 63 provided at a position corresponding to the left eye of the user. The AR processing unit 142b recognizes each of the captured outside scene image CP62 and the outside scene image CP63, and specifies a reference point for a specific object included in both images. The reference point is a point used as a reference for specifying the position of the object, and can be arbitrarily determined. The reference point can be, for example, the center point of the object.

  The AR processing unit 142b obtains a deviation amount between the reference point in the outside scene image CP62 and the reference point in the outside scene image CP63. Thereafter, the AR processing unit 142b calculates the binocular parallax θ or the depth distance L from the obtained deviation amount using a table or the like that defines the correspondence between the deviation amount and the binocular parallax θ (or the depth distance L). . Note that the larger the deviation amount, the larger the binocular parallax θ and the smaller the depth distance L.

  For example, in the example of FIG. 12, the AR processing unit 142b specifies the apple center point O1 included in the outside scene image CP62 and the apple center point O2 included in the outside scene image CP63, and the center point O1 and the center A deviation amount DE1 from the point O2 is obtained. Then, the AR processing unit 142b calculates the binocular parallax θ (or the depth distance L) from the shift amount DE1 using the table described above.

  As described above, according to the third embodiment, the AR processing unit 142b (augmented reality processing unit) performs the right eye or temple of the user when wearing the head mounted display HMb (head mounted display device) When a specific object included in the outside scene image CP62 (second outside scene image) captured by the camera 62 (second imaging unit) arranged at a position corresponding to the periphery of the head mounted display HMb is mounted. A specific object included in the outside scene image CP63 (third outside scene image) captured by the camera 63 (third imaging unit) disposed at a position corresponding to the left eye or temple of the user or the periphery thereof; , And the information about the depth of the object in the outside world can be acquired from the calculated amount of deviation. As a result, the AR processing unit 142b can provide virtual reality using information related to the depth of the object in the AR processing.

D. Variations:
In addition, this invention is not restricted to said Example and embodiment, A various structure can be taken in the range which does not deviate from the summary. For example, a function realized by software may be realized by hardware. In addition, the following modifications are possible.

D1. Modification 1:
In the said Example, it illustrated about the structure of the head mounted display. However, the configuration of the head-mounted display can be arbitrarily determined without departing from the gist of the present invention. For example, each component can be added, deleted, converted, and the like.

  In the above embodiment, for convenience of explanation, the control unit includes a transmission unit, and the image display unit includes a reception unit. However, each of the transmission unit and the reception unit in the above embodiment has a function capable of bidirectional communication, and can function as a transmission / reception unit.

  For example, the connection unit may be omitted, and the control unit and the image display unit may be configured to perform wireless communication. Specifically, the control unit further includes a first wireless communication unit, and the image display unit further includes a second wireless communication unit and a power source. In this case, the first wireless communication unit functions as a transmission unit in the above embodiment, and the second wireless communication unit functions as a reception unit in the above embodiment.

  For example, the configurations of the control unit and the image display unit illustrated in FIG. 2 can be arbitrarily changed. Specifically, for example, the touch pad may be omitted from the control unit, and the operation may be performed using only the cross key. Further, the control unit may be provided with another operation interface such as an operation stick. Moreover, it is good also as what receives an input from a keyboard or a mouse | mouth as a structure which can connect devices, such as a keyboard and a mouse | mouth, to a control part. In addition, a communication unit using Wi-Fi (wireless fidelity) or the like may be provided in the control unit.

  For example, the control unit illustrated in FIG. 2 is connected to the image display unit via a wired signal transmission path. However, the control unit and the image display unit may be connected by a connection via a wireless signal transmission path such as a wireless LAN, infrared communication, or Bluetooth (registered trademark).

  For example, the head mounted display is a binocular transmissive head mounted display, but may be a monocular head mounted display. Further, it may be configured as a non-transmissive head mounted display in which the transmission of the outside scene is blocked when the user wears the head mounted display. In the case of the non-transmissive head mounted display, the AR processing unit generates new additional image data in which the outside scene image and the additional image are superimposed, and the image processing unit displays the new additional image data.

  For example, functional units such as an image processing unit, a display control unit, an AR processing unit, and an audio processing unit are realized by the CPU developing and executing a computer program stored in the ROM or hard disk on the RAM. Described. However, these functional units may be configured using an ASIC (Application Specific Integrated Circuit) designed to realize the function.

  For example, in the above-described embodiment, the image display unit is a head-mounted display that is mounted like glasses, but the image display unit is a normal flat display device (liquid crystal display device, plasma display device, organic EL display device, etc.) ). Also in this case, the connection between the control unit and the image display unit may be a connection via a wired signal transmission path or a connection via a wireless signal transmission path. If it does in this way, a control part can also be utilized as a remote control of a usual flat type display device.

  Further, as the image display unit, instead of the image display unit worn like glasses, an image display unit of another shape such as an image display unit worn like a hat may be adopted. Further, the earphone may be an ear-hook type or a headband type, or may be omitted.

  For example, in the above embodiment, the secondary battery is used as the power source. However, the power source is not limited to the secondary battery, and various batteries can be used. For example, a primary battery, a fuel cell, a solar cell, a thermal cell, or the like may be used.

D2. Modification 2:
In the above embodiment, an example of the guide image has been described. However, the guide image is not limited to the above example, and various modifications are possible.

  For example, in the first embodiment (FIG. 7), as an example of the guide image, a rectangular image having circular guide marks at the intersections of the four corners and the diagonal line is shown. However, for example, the image may be a rectangular image having guide marks only at two of the four corners (for example, two points on a diagonal line). Furthermore, for example, the shape of the guide mark does not have to be circular, and a polygon, a pictograph such as a character, an icon, or the like may be employed.

  For example, in the second embodiment (FIG. 10), as an example of the guide image, an image having a rectangular shape and frame-shaped guide marks at the four corners is shown. However, for example, the image may be a rectangle and have frame-shaped guide marks only at two of the four corners (for example, two points on a diagonal line). For example, it is good also as an image which has the frame-shaped guide mark surrounding all four sides of a rectangle. Further, for example, the shape of the guide mark may not be a frame shape.

D3. Modification 3:
In the said Example, an example of the area | region determination process was shown. However, the processing procedure shown in FIGS. 6 and 9 is merely an example, and various modifications are possible. For example, some steps may be omitted, and other steps may be added. Further, the order of the steps to be executed may be changed.

  For example, in the area determination process shown in FIG. 6, the user's fingertip is used as the “predetermined mark”. However, any mark can be used. For example, a tapered shape such as a pen tip or a pointer can be used.

  For example, in the area determination process illustrated in FIG. 6, the AR processing unit performs a series of processes of guiding the user to point the guidance mark, acquiring an outside scene image, acquiring the coordinates of the fingertip position, and storing the coordinates. It was decided to repeat for the number of guide marks. However, the flow of processing is to sequentially guide the user to point to the guide marks, acquire the outside scene images for all the guide marks, and then acquire the coordinates of the fingertip positions from the plurality of acquired outside scene images, and save the coordinates. It is good.

  For example, in the area determination process shown in FIG. 9, paper is used as the “predetermined mark”. However, any mark can be used as long as it has a rectangular side surface. For example, an underlay, a notebook, a box, or a book may be used.

D4. Modification 4:
In the said Example, an example of the depth acquisition process was shown. However, the processing procedures shown in FIGS. 12 and 13 are merely examples, and various modifications are possible.

D5. Modification 5:
In the above embodiment, the image light generation unit is configured using a backlight, a backlight control unit, an LCD, and an LCD control unit. However, the above aspect is merely an example. The image light generation unit may include a configuration unit for realizing another method together with or in place of these configuration units.

  For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the present invention can be applied to a laser retinal projection type head-mounted display device.

D6. Modification 6:
Of the constituent elements in the above-described embodiments and modifications, elements other than the elements corresponding to the configurations described in the independent claims in the claims are additional elements and can be omitted.

10, 10b ... Control unit (controller)
DESCRIPTION OF SYMBOLS 12 ... Illuminating part 14 ... Touch pad 16 ... Cross key 18 ... Power switch 20 ... Image display part 21 ... Ear hook part 22 ... Right display drive part 24 ... Left display drive part 26 ... Right optical image display part 28 ... Left optical image Display unit 32 ... Right earphone 34 ... Left earphone 40 ... Connecting unit 42 ... Right cord 44 ... Left cord 46 ... Connecting member 48 ... Body cord 51 ... Transmitting unit 52 ... Transmitting unit 53 ... Receiving unit 54 ... Receiving unit 61 ... Camera ( Imaging unit)
62 ... Camera (second imaging unit)
63 ... Camera (third imaging unit)
DESCRIPTION OF SYMBOLS 110 ... Input information acquisition part 120 ... Memory | storage part 122 ... Coordinate information memory | storage part 130 ... Power supply 140 ... CPU
142, 142a, 142b ... AR processing unit 160 ... image processing unit 170 ... audio processing unit 180 ... interface 190 ... display control unit 201 ... right backlight control unit (image light generation unit)
202 ... Left backlight control unit (image light generation unit)
211 ... Right LCD controller (display element, image light generator)
212 ... Left LCD controller (display element, image light generator)
221 ... Right backlight (image light generator)
222: Left backlight (image light generator)
241 ... Right LCD (display element, image light generator)
242 ... Left LCD (display element, image light generator)
251 ... Right projection optical system (light guide unit)
252 ... Left projection optical system (light guide)
261 ... Right light guide plate (light guide part)
262 ... Left light guide plate (light guide part)
280 ... Power supply PCLK ... Clock signal VSync ... Vertical synchronization signal HSync ... Horizontal synchronization signal Data ... Image data Data1 ... Right eye image data Data2 ... Left eye image data OA ... External device SC ... Outside view VI ... Virtual image IL ... Light illumination PL ... image light VR ... field of view n ... variable CP62 ... outside scene image (second outside scene image)
CP63 ... outside scene image (third outside scene image)
O1, O2 ... Center points P1 to P51 ... Guidance marks EA ... Overlapping area OB ... Object RE ... Right eye LE ... Left eye CG, CGa ... Guide image EL, ER ... End HM, HMa, HMb ... Head-mounted display Device (head mounted display)
AP ... tip part PP ... paper CP ... outside scene image

SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms or application examples.
[Form 1]
A head-mounted display device that displays a virtual image so as to be visible at the same time as an outside scene,
Imaging means for imaging the outside scene;
Processing means for generating an additional image according to the imaged outside scene;
Display means for displaying the generated additional image as the virtual image;
With
The head-mounted display device, wherein the processing unit generates the additional image using an overlapping region in which the virtual image display region and the imaging region of the imaging unit overlap among the captured outside scene.

Claims (6)

A head-mounted display device that allows a user to view a virtual image and an outside scene at the same time,
An imaging unit that captures an outside scene and obtains an outside scene image;
An augmented reality processing unit that analyzes the acquired outside scene image and generates additional image data for augmented reality according to a result of the analysis;
A display element that generates image light using the generated additional image data;
A light guide unit that forms a virtual image in front of the user's eyes using the generated image light; and
With
The augmented reality processing unit obtains an overlapping region in which the virtual image display region and the imaging region of the imaging unit overlap in the outside scene image, and generates the additional image data using the overlapping region. Part-mounted display device. The head-mounted display device according to claim 1, further comprising:
A storage unit storing guide image data for indicating a display area of the virtual image for the user;
The augmented reality processing unit generates image light using the guide image data, guides the user to indicate a predetermined mark in the display area of the virtual image, and acquires and acquires the outside scene image. A head-mounted display device that acquires a position of the mark obtained by analyzing the outside scene image as a boundary between the overlapping region and another region. The head-mounted display device according to claim 2,
The guide image data is a rectangular shape having the same aspect ratio as the display element, and includes guide marks arranged at at least two of the four corners,
The augmented reality processing unit is a head-mounted display device that guides the user to indicate the mark by pointing the guide mark with a finger. The head-mounted display device according to claim 2,
The guide image data is a rectangular shape having the same aspect ratio as the display element, and includes frame-shaped guide marks arranged at at least two of the four corners,
The augmented reality processing unit guides the user so as to show the mark by arranging an object having a rectangular side surface in the frame of the guide mark. . The head-mounted display device according to any one of claims 1 to 4, further comprising:
A second imaging unit that is disposed at a position corresponding to the right eye or temple of the user or the periphery of the user when the head-mounted display device is mounted, and that captures an external scene and obtains a second external scene image; ,
A third imaging unit that is arranged at a position corresponding to the left eye or temple of the user or the periphery of the user when the head-mounted display device is worn, and that captures the outside scene and obtains a third outside scene image; ,
With
The augmented reality processing unit further obtains a deviation amount between the specific object included in the second outside scene image and the specific object included in the third outside scene image, and calculates the deviation. A head-mounted display device that acquires information about the depth of the object in the outside world from a quantity. A method for controlling a head-mounted display device in which a user can view a virtual image and an outside scene at the same time,
(A) capturing an outside scene and obtaining an outside scene image;
(B) analyzing the acquired outside scene image and generating additional image data for augmented reality according to the result of the analysis;
(C) generating image light using the generated additional image data;
(D) forming a virtual image in front of the user's eyes using the generated image light;
With
In the step (b), an overlapping area in which the virtual image display area and the imaging area in the step (a) overlap is acquired from the outside scene image, and the additional image data is generated using the overlapping area. Head mounted display device.

Images