JP2019004471A - Head-mounted type display device, and control method of head-mounted type display device - Google Patents

Abstract

To provide augmented reality in which visual discomfort of a user has been alleviated by fusing a virtual object in an outside view in consideration of various conditions related to display environment of a virtual image.SOLUTION: A head-mounted type display device that allows a user to visually recognize a virtual image and an outside view includes: an image display section that allows the user to visually recognize the virtual image; and an augmented reality processing section that causes the image display section to form a virtual image, which represents a virtual object being an object for providing augmented reality to the user, the virtual object having alleviated visual discomfort of the user.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a head-mounted display device.

  2. Description of the Related Art A head-mounted display device that forms a virtual image in a visual field region of an observer by being used on the observer's head is known. This head-mounted display device is also called a head-mounted display (HMD), and is a non-transparent head-mounted display device that blocks the user's field of view when the head-mounted display device is mounted. There are a display device and a transmissive head-mounted display device in which the user's field of view is not blocked when the head-mounted display device is mounted.

  On the other hand, a technique called augmented reality (AR) is known in which information is additionally presented to a real environment using a computer. As methods for realizing augmented reality, a method based on image recognition and a method based on a pass-through method are known. In the case of the method based on image recognition, for example, a virtual object is generated by recognizing an image of an outside scene captured by a WEB camera or the like. In the case of a method based on the pass-through method, a virtual object is generated using current position information acquired by, for example, GPS and azimuth information acquired by, for example, an electronic compass. In the non-transmissive head-mounted display device, an image obtained by superimposing the image of the outside scene and the virtual object generated as described above is displayed on the liquid crystal display. Thereby, the user can experience augmented reality. In the transmissive head-mounted display device, only the virtual object generated as described above is displayed on the liquid crystal display. The user can experience augmented reality by visually recognizing both the virtual object displayed as a virtual image via the liquid crystal display and the actual outside scene that is seen through the front lens. For this reason, in order to provide an augmented reality that alleviates the user's visual discomfort in an optically transmissive head-mounted display device, it is displayed as a virtual image with respect to the outside scene that the user actually sees. It is necessary to fuse the virtual object to be created.

  Patent Document 1 describes a technique in which a distance from a user to a real object included in the outside scene and a distance to the virtual object are approximately the same distance in order to fuse the virtual object to the outside scene. . In Patent Document 1, a distance D between a user and a real object is obtained, a convergence angle θ of the real object is determined from the distance D, and a convergence angle of the virtual object such that the convergence angle θ ± 40 minutes of the real object is obtained. To generate image data for the right eye that realizes the calculated convergence angle and image data for the left eye. Japanese Patent Application Laid-Open No. 2004-228561 describes a technique that enables the convergence angle of a virtual object to be adjusted in order to fuse the virtual object to the outside scene. In Patent Document 3, in order to provide an augmented reality in which a user's visual discomfort is alleviated in a non-transmissive head-mounted display device, a camera for capturing an outside scene image is measured in advance. A technique for adjusting according to the parameters is described.

  In Patent Document 4 and Patent Document 5, a specific color region such as skin color is cut out from an image in a user's line-of-sight direction captured by a camera in order to reduce a sense of discomfort when a virtual object is superimposed on an outside scene. A technique for masking a virtual object is described.

Japanese Patent No. 3717653 JP-A-5-328408 JP 2008-227865 A JP 2005-346468 A JP 2003-296759 A

  In the head-mounted display device, in order to stereoscopically display the virtual image of the virtual object at the targeted convergence angle, for example, the size of the liquid crystal display, the distance between the virtual images displayed in front of the left and right eyes of the user, the user It is necessary to consider various conditions relating to the virtual image display environment, such as the interocular distance. However, the techniques described in Patent Document 1 and Patent Document 2 have a problem that sufficient consideration has not been made in this regard. The technique described in Patent Document 3 does not target an optically transmissive head-mounted display device. For this reason, an optically transmissive head mounted that can provide augmented reality that eases the user's visual discomfort by fusing virtual objects to the outside scene in consideration of various conditions related to the virtual image display environment A type display device has been desired.

  Further, in the techniques described in Patent Document 4 and Patent Document 5, a virtual object is processed based on an image photographed by a camera. There is a problem that the accuracy of the processing of the virtual object cannot be secured for the objects having similarities. For this reason, there has been a demand for an optically transmissive head-mounted display device that can provide an augmented reality in which a user's visual discomfort is reduced by fusing a virtual object to the outside scene with higher accuracy.

  As described above, there has been a demand for an optically transmissive head-mounted display device that can provide an augmented reality that reduces the user's visual discomfort.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to an aspect of the present invention, a head-mounted display device that allows a user to visually recognize a virtual image and an outside scene is provided. The head-mounted display device includes: an image display unit that allows the user to visually recognize the virtual image; and a virtual object that is an object for giving the user an augmented reality, A head-mounted display device comprising: an augmented reality processing unit that causes the image display unit to form the virtual image representing the virtual object with a sense of incongruity. According to this form of the head-mounted display device, the image display unit causes the user to visually recognize the virtual object in which the user's visual discomfort is alleviated as a virtual image. Therefore, it is possible to provide an optically transmissive head-mounted display device that can provide an augmented reality that alleviates the user's visual discomfort.

(2) In the head-mounted display device of the above aspect; the augmented reality processing unit uses the three-dimensional information representing the environment around the user as a virtual three-dimensional object in a three-dimensional space, The uncomfortable feeling with respect to the virtual object may be alleviated by harmonizing the virtual object with the environment. According to this form of the head-mounted display device, the augmented reality processing unit harmonizes the virtual object with the surrounding environment. Thereby, the image display part can make a user visually recognize the virtual object by which the user's visual discomfort was eased as a virtual image. The augmented reality processing unit uses the three-dimensional information to harmonize the virtual object with the surrounding environment. For this reason, compared with the case where a virtual object is harmonized with the surrounding environment based on the image image | photographed with the camera, the precision of harmony can be improved.

(3) In the head-mounted display device of the above aspect; the augmented reality processing unit; arranges the virtual object in the three-dimensional information; for at least one of the virtual object and the three-dimensional information Then, after applying a visual effect according to the environment, the sense of discomfort with the virtual object may be alleviated by making the virtual object two-dimensional. According to this form of the head-mounted display device, the augmented reality processing unit can add a visual effect corresponding to the surrounding environment to the virtual object using the three-dimensional information.

(4) In the head-mounted display device according to the above aspect; the visual effect according to the environment is at least; trimming a portion of the virtual object that is behind the three-dimensional object in the three-dimensional information; Lighting the virtual object according to the control; adjusting the behavior of the virtual object based on at least one of the coefficient of restitution and the coefficient of friction of the three-dimensional object in the three-dimensional information set according to the environment; Any one of them may be included. According to this form of the head-mounted display device, the augmented reality processing unit performs at least one of trimming according to the environment, lighting according to the environment, and behavior adjustment according to the environment with respect to the virtual object. Can add visual effects.

(5) The head-mounted display device according to the above aspect further includes: an image acquisition unit that acquires an image of the user's visual field direction when the head-mounted display device is mounted; and the augmented reality processing unit May estimate the environment by recognizing the image in the viewing direction acquired by the image acquisition unit. According to the head-mounted display device of this aspect, the augmented reality processing unit can automatically estimate the environment around the user by recognizing the image in the user's field of view.

(6) In the head-mounted display device of the above aspect, the augmented reality processing unit calculates a distance from the user's eye to the user's gaze point when the virtual object is two-dimensionalized. You may consider it. According to this form of the head-mounted display device, the augmented reality processing unit considers the distance from the user's eye to the user's gaze point when two-dimensionalizing the virtual object. For this reason, for example, it is possible to add a defocus effect to a virtual object at a position deviating from the user's gaze point.

(7) In the head-mounted display device of the above aspect, the augmented reality processing unit generates right eye image data for the right eye and left eye image data for the left eye representing the virtual object. Thus, a visual effect for enabling stereoscopic viewing of the virtual object is added to the virtual object to alleviate the uncomfortable feeling with respect to the virtual object; and the image display unit includes the right-eye image data and The left eye image data is used to cause the left and right eyes of the user to visually recognize different virtual images; and further, displayed based on the same right eye image data and left eye image data. The difference between the first convergence angle of the virtual image and the second convergence angle of the virtual image displayed based on the right-eye image data and the left-eye image data shifted to the left and right is stored. A parallax angle storage unit; and the augmented reality processing unit uses the difference stored in the pixel parallax angle storage unit to fuse the virtual object to the outside scene. And the left-eye image data may be generated. According to the head-mounted display device of this aspect, the difference stored in the pixel parallax angle storage unit is the first virtual image displayed based on the same right-eye image data and left-eye image data. And the second convergence angle of the virtual image displayed based on the right-eye image data and the left-eye image data shifted to the left and right. For this reason, the difference stored in the pixel parallax angle storage unit is a parallax angle realized by image data shifted to the left and right, and is a parallax angle determined in consideration of various conditions regarding the virtual image display environment. It can be said that there is. Therefore, the augmented reality processing unit uses the difference stored in the pixel parallax angle storage unit and considers various conditions related to the virtual image display environment, and fuses the virtual object to the outside scene. Data and image data for the left eye can be generated. In this way, the augmented reality processing unit fuses the virtual object to the outside scene in consideration of various conditions related to the virtual image display environment when generating the image data, so that the user's visual discomfort is alleviated. A head-mounted display device that can provide a sense of reality can be realized.

(8) In the head-mounted display device of the above aspect; the augmented reality processing unit; determines a target distance for allowing the user to visually recognize the virtual object; and a convergence angle at the target distance from the determined target distance A target convergence angle is calculated; using the calculated target convergence angle and the first convergence angle, a target parallax angle that is a parallax angle at the target distance is calculated; the target parallax angle and the pixel The right eye image data and the left eye image data may be generated from a single image data representing the virtual object using the difference stored in the parallax angle storage unit. According to this form of the head-mounted display device, the augmented reality processing unit determines a target distance for allowing the user to visually recognize the virtual object, and calculates a target convergence angle that is a convergence angle at the target distance from the determined target distance. Then, using the calculated target convergence angle and the first convergence angle, a target parallax angle that is a parallax angle at the target distance is calculated. The augmented reality processing unit uses the target parallax angle calculated in this way and the difference stored in the pixel parallax angle storage unit, so that different right-eye images can be obtained from a single image data representing the virtual object. Data and image data for the left eye can be generated.

(9) In the head-mounted display device of the above aspect; the augmented reality processing unit; determines a target distance for allowing the user to visually recognize the virtual object; on the 3D model space based on the determined target distance Two virtual cameras for acquiring a 2D projection image are set; a target convergence angle that is a convergence angle at the target distance is calculated from the determined target distance; the calculated target convergence angle and the first The target parallax angle, which is a parallax angle at the target distance, is calculated using the convergence angle of the target parallax; using the target parallax angle and the difference stored in the pixel parallax angle storage unit, The right-eye image data is generated by scaling the 2D projection image by the virtual camera, and the left-eye image data is generated by scaling the 2D projection image by the other virtual camera. Good. According to this form of the head-mounted display device, the augmented reality processing unit determines a target distance for allowing the user to visually recognize the virtual object, and calculates a target convergence angle that is a convergence angle at the target distance from the determined target distance. Then, using the calculated target convergence angle and the first convergence angle, a target parallax angle that is a parallax angle at the target distance is calculated. The augmented reality processing unit uses the target parallax angle calculated in this way and the difference stored in the pixel parallax angle storage unit to scale the 2D projection images from the two virtual cameras, respectively, Eye image data and left eye image data can be generated.

(10) The head-mounted display device of the above aspect further includes: an interocular distance storage unit that stores the interocular distance of the user; the augmented reality processing unit sets the two virtual cameras At this time, based on the determined target distance, a virtual viewpoint, which is a virtual viewpoint, is arranged in the 3D model space, and one virtual camera is arranged at a position away from the arranged virtual viewpoint by an interocular distance / 2. Then, the other virtual camera may be arranged at a position that is a distance of the interocular distance / 2 from the arranged virtual viewpoint. According to the head mounted display device of this aspect, when setting the virtual camera on the 3D model space, the augmented reality processing unit first arranges the virtual viewpoint based on the target distance that allows the user to visually recognize the virtual object. To do. Then, the augmented reality processing unit arranges one virtual camera at a position that is an interocular distance / 2 away from the arranged virtual viewpoint, and the other virtual camera at a position that is an interocular distance / 2 away from the arranged virtual viewpoint. Place. As a result, the virtual camera can acquire a 2D projection image in consideration of both the target distance for allowing the user to visually recognize the virtual object and the interocular distance of the user.

(11) The head-mounted display device of the above aspect further includes: an interpupillary distance measuring unit that measures the interpupillary distance of the user; a measurement result by the interpupillary distance measuring unit is the interocular distance. It may be stored in the interocular distance storage unit. According to the head-mounted display device of this aspect, the interpupillary distance according to the user is determined by measuring the interpupillary distance for each user and updating the interocular distance storage unit. Can be memorized.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with a new component, and partially delete the limited contents of some of the plurality of components. In order to solve some or all of the above-described problems or achieve some or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, an aspect of the present invention can be realized as an apparatus including some or all of the three elements of the augmented reality processing unit, the image display unit, and the pixel parallax angle storage unit. That is, this apparatus may or may not have the augmented reality processing unit. Moreover, this apparatus may or may not have an image display unit. In addition, this apparatus may or may not have a pixel parallax angle storage unit. Such a device can be realized, for example, as a head-mounted display device, but can also be realized as a device other than the head-mounted display device. Any or all of the technical features of each form of the head-mounted display device described above can be applied to this device.

  The present invention can be realized in various modes. For example, a head-mounted display device and a head-mounted display device control method, a head-mounted display system, these methods, devices, or The present invention can be realized in the form of a computer program for realizing the function of the system, a recording medium on which the computer program is recorded, or the like.

It is explanatory drawing which shows schematic structure of the head mounted display apparatus in one Embodiment of this invention. It is a block diagram which shows the composition of a head mount display functionally. It is explanatory drawing which shows an example of the virtual image visually recognized by the user. It is explanatory drawing for demonstrating the procedure of an augmented reality process. It is explanatory drawing for demonstrating an interocular distance and a pixel parallax angle. It is explanatory drawing which shows a mode that AR process part produces | generates the image data for right eyes, and the image data for left eyes from image data. It is a flowchart showing the procedure of the augmented reality process in 2nd Example. It is explanatory drawing for demonstrating the detail of the augmented reality process in 2nd Example. It is explanatory drawing which shows a mode that AR process part scales the image obtained by projection using the 2D projection image of the virtual camera arrange | positioned at the virtual viewpoint. It is a flowchart showing the procedure of the augmented reality process in 3rd Example. It is a block diagram which shows functionally the structure of the head mounted display in 2nd Embodiment. It is a flowchart which shows the procedure of the augmented reality process in 2nd Embodiment. It is explanatory drawing which shows an example of a mask object. It is explanatory drawing which shows an example of arrangement | positioning of the virtual object with respect to a mask object. It is explanatory drawing which shows the mode of the virtual object after implementation of lighting. It is explanatory drawing which shows a mode that the behavior of a virtual object is adjusted based on a restitution coefficient and a friction coefficient. It is explanatory drawing which shows the image data obtained by 2D projection of a virtual camera. It is explanatory drawing which shows an example of the virtual image visually recognized by the user in the augmented reality process of 2nd Embodiment. It is explanatory drawing which shows the structure of the external appearance of the head mounted display in a modification.

A. First embodiment:
A-1. Configuration of head mounted display device:
FIG. 1 is an explanatory diagram showing a schematic configuration of a head-mounted display device according to an embodiment of the present invention. The head-mounted display device 100 is a display device mounted on the head, and is also called a head mounted display (HMD). The head mounted display 100 according to the present embodiment is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene.

  The head-mounted display 100 includes an image display unit 20 that allows a user to visually recognize a virtual image when attached to the user's head, and a control unit (controller) 10 that controls the image display unit 20.

  The image display unit 20 is a mounting body that is mounted on the user's head, and has a glasses shape in the present embodiment. The image display unit 20 includes a right holding unit 21, a right display driving unit 22, a left holding unit 23, a left display driving unit 24, a right optical image display unit 26, a left optical image display unit 28, and a camera 61. And an interpupillary distance measuring unit 62. The right optical image display unit 26 and the left optical image display unit 28 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 20, respectively. One end of the right optical image display unit 26 and one end of the left optical image display unit 28 are connected to each other at a position corresponding to the eyebrow of the user when the user wears the image display unit 20.

  The right holding unit 21 extends from the end ER which is the other end of the right optical image display unit 26 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member. Similarly, the left holding unit 23 extends from the end EL which is the other end of the left optical image display unit 28 to a position corresponding to the user's temporal region when the user wears the image display unit 20. It is a member provided. The right holding unit 21 and the left holding unit 23 hold the image display unit 20 on the user's head like a temple of glasses.

  The right display drive unit 22 is disposed inside the right holding unit 21, in other words, on the side facing the user's head when the user wears the image display unit 20. Further, the left display driving unit 24 is disposed inside the left holding unit 23. Hereinafter, the right holding unit 21 and the left holding unit 23 are collectively referred to simply as “holding unit”, and the right display driving unit 22 and the left display driving unit 24 are collectively referred to simply as “display driving unit”. The right optical image display unit 26 and the left optical image display unit 28 are collectively referred to simply as “optical image display unit”.

  The display driving unit includes a liquid crystal display (hereinafter referred to as “LCD”) 241, 242, projection optical systems 251, 252, and the like (see FIG. 2). Details of the configuration of the display driving unit will be described later. The optical image display unit as an optical member includes light guide plates 261 and 262 (see FIG. 2) and a light control plate. The light guide plates 261 and 262 are formed of a light transmissive resin material or the like, and guide the image light output from the display driving unit to the user's eyes. The light control plate is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 20 (the side opposite to the user's eye side). The light control plate protects the light guide plates 261 and 262 and suppresses damage to the light guide plates 261 and 262 and adhesion of dirt. Further, by adjusting the light transmittance of the light control plate, it is possible to adjust the external light quantity entering the user's eyes and adjust the ease of visual recognition of the virtual image. The light control plate can be omitted.

  The camera 61 is disposed at a position corresponding to the user's eyebrow when the user wears the image display unit 20. The camera 61 captures an outside scene image in the front side direction of the image display unit 20, in other words, an outside scene (external scenery) in the user's viewing direction with the head mounted display 100 attached. The camera 61 is a so-called visible light camera, and the outside scene image acquired by the camera 61 is an image representing the shape of the object from the visible light emitted from the object. The camera 61 in the present embodiment is a monocular camera, but may be a stereo camera. The camera 61 functions as an “image acquisition unit”.

  The interpupillary distance measuring unit 62 measures the interpupillary distance of the user. The interpupillary distance is a distance between the center of the iris of the user's right eye RE and the center of the iris of the user's left eye LE. As shown in FIG. 1, the interpupillary distance measuring unit 62 is disposed on the inner surface of the image display unit 20, and two cameras that capture images of the right eye RE and left eye LE of the user, and the captured images. For example, it comprises a processing unit that analyzes using a method by triangulation and calculates the distance between the centers of the irises of the left and right eyes. The interpupillary distance measuring unit 62 may measure the interpupillary distance of the user using ultrasonic waves or infrared rays instead of the camera. Moreover, the interpupillary distance measuring unit 62 can also measure the distance between the pupils of the user by combining a plurality of the above-described methods.

  The image display unit 20 further includes a connection unit 40 for connecting the image display unit 20 to the control unit 10. The connection unit 40 includes a main body cord 48 connected to the control unit 10, a right cord 42 and a left cord 44 in which the main body cord 48 branches into two, and a connecting member 46 provided at the branch point. . The right cord 42 is inserted into the casing of the right holding unit 21 from the distal end AP in the extending direction of the right holding unit 21 and connected to the right display driving unit 22. Similarly, the left cord 44 is inserted into the housing of the left holding unit 23 from the distal end AP in the extending direction of the left holding unit 23 and connected to the left display driving unit 24. The connecting member 46 is provided with a jack for connecting the earphone plug 30. A right earphone 32 and a left earphone 34 extend from the earphone plug 30.

  The image display unit 20 and the control unit 10 transmit various signals via the connection unit 40. Each end of the main body cord 48 opposite to the connecting member 46 and the control unit 10 are provided with connectors (not shown) that are fitted to each other. The connector of the main body cord 48 and the control unit 10 The control unit 10 and the image display unit 20 are connected to or disconnected from each other by the fitting / releasing of the connector. For the right cord 42, the left cord 44, and the main body cord 48, for example, a metal cable or an optical fiber can be adopted.

  The control unit 10 is a device for controlling the head mounted display 100. The control unit 10 includes a lighting unit 12, a touch pad 14, a cross key 16, and a power switch 18. The lighting unit 12 notifies the operation state of the head mounted display 100 (for example, ON / OFF of the power supply) by the light emission mode. For example, an LED (Light Emitting Diode) can be used as the lighting unit 12. The touch pad 14 detects a contact operation on the operation surface of the touch pad 14 and outputs a signal corresponding to the detected content. As the touch pad 14, various touch pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. The cross key 16 detects a pressing operation on a key corresponding to the up / down / left / right direction, and outputs a signal corresponding to the detected content. The power switch 18 switches the power state of the head mounted display 100 by detecting a slide operation of the switch.

  FIG. 2 is a block diagram functionally showing the configuration of the head mounted display 100. The control unit 10 includes an input information acquisition unit 110, a storage unit 120, a power supply 130, a wireless communication unit 132, a GPS module 134, a CPU 140, an interface 180, and transmission units (Tx) 51 and 52. The parts are connected to each other by a bus (not shown).

  The input information acquisition unit 110 acquires a signal corresponding to an operation input to the touch pad 14, the cross key 16, the power switch 18, and the like, for example. The storage unit 120 includes a ROM, a RAM, a DRAM, a hard disk, and the like. The storage unit 120 includes an interocular distance 122 and a pixel parallax angle 124. Details will be described later. The power supply 130 supplies power to each part of the head mounted display 100. As the power supply 130, for example, a secondary battery can be used. The wireless communication unit 132 performs wireless communication with other devices in accordance with a predetermined wireless communication standard such as a wireless LAN or Bluetooth. The GPS module 134 detects its current position by receiving a signal from GPS hygiene.

  The CPU 140 functions as an operating system (OS) 150, an image processing unit 160, an audio processing unit 170, a display control unit 190, and an AR processing unit 142 by reading out and executing a computer program stored in the storage unit 120. . The AR processing unit 142 executes processing for realizing augmented reality (hereinafter, also referred to as “augmented reality processing”), triggered by a processing start request from the OS 150 or a specific application. Details will be described later. The AR processing unit 142 corresponds to the “augmented reality processing unit” in the claims.

  The image processing unit 160 generates a signal based on content (video) input via the interface 180 or the wireless communication unit 132. Then, the image processing unit 160 supplies the generated signal to the image display unit 20 via the connection unit 40. The signal supplied to the image display unit 20 differs between the analog format and the digital format. In the case of the analog format, the image processing unit 160 generates and transmits a clock signal PCLK, a vertical synchronization signal VSync, a horizontal synchronization signal HSync, and image data Data. Specifically, the image processing unit 160 acquires an image signal included in the content. For example, in the case of a moving image, the acquired image signal is an analog signal generally composed of 30 frame images per second. The image processing unit 160 separates a synchronization signal such as a vertical synchronization signal VSync and a horizontal synchronization signal HSync from the acquired image signal, and generates a clock signal PCLK by a PLL circuit or the like according to the period. The image processing unit 160 converts the analog image signal from which the synchronization signal is separated into a digital image signal using an A / D conversion circuit or the like. The image processing unit 160 stores the converted digital image signal in the DRAM in the storage unit 120 for each frame as image data Data of RGB data. On the other hand, in the digital format, the image processing unit 160 generates and transmits a clock signal PCLK and image data Data. Specifically, when the content is in digital format, the clock signal PCLK is output in synchronization with the image signal, so that the generation of the vertical synchronization signal VSync and the horizontal synchronization signal HSync and the A / D conversion of the analog image signal are performed. It becomes unnecessary. The image processing unit 160 performs image processing such as resolution conversion processing, various tone correction processing such as adjustment of luminance and saturation, and keystone correction processing on the image data Data stored in the storage unit 120. May be executed.

  The image processing unit 160 transmits the generated clock signal PCLK, vertical synchronization signal VSync, horizontal synchronization signal HSync, and image data Data stored in the DRAM in the storage unit 120 via the transmission units 51 and 52, respectively. To do. Note that the image data Data transmitted through the transmission unit 51 is also referred to as “right-eye image data Data1”, and the image data Data transmitted through the transmission unit 52 is also referred to as “left-eye image data Data2”. The transmission units 51 and 52 function as a transceiver for serial transmission between the control unit 10 and the image display unit 20.

  The display control unit 190 generates control signals for controlling the right display drive unit 22 and the left display drive unit 24. Specifically, the display control unit 190 uses the control signal to turn on / off the right LCD 241 by the right LCD control unit 211, turn on / off the right backlight 221 by the right backlight control unit 201, and control the left LCD. Each of the right display drive unit 22 and the left display drive unit 24 is controlled by individually controlling ON / OFF driving of the left LCD 242 by the unit 212 and ON / OFF driving of the left backlight 222 by the left backlight control unit 202. Controls the generation and emission of image light. For example, the display control unit 190 may cause both the right display driving unit 22 and the left display driving unit 24 to generate image light, generate only one image light, or neither may generate image light. Further, the display control unit 190 transmits control signals for the right LCD control unit 211 and the left LCD control unit 212 via the transmission units 51 and 52, respectively. In addition, the display control unit 190 transmits control signals for the right backlight control unit 201 and the left backlight control unit 202, respectively.

  The audio processing unit 170 acquires an audio signal included in the content, amplifies the acquired audio signal, and transmits the acquired audio signal to a speaker (not shown) in the right earphone 32 and a speaker (not shown) in the left earphone 34 connected to the connecting member 46. To supply. For example, when a Dolby (registered trademark) system is adopted, processing is performed on an audio signal, and different sounds with different frequencies or the like are output from the right earphone 32 and the left earphone 34, respectively.

  The interface 180 is an interface for connecting various external devices OA that are content supply sources to the control unit 10. Examples of the external device OA include a personal computer PC, a mobile phone terminal, and a game terminal. As the interface 180, for example, a USB interface, a micro USB interface, a memory card interface, or the like can be used.

  The image display unit 20 includes a right display drive unit 22, a left display drive unit 24, a right light guide plate 261 as a right optical image display unit 26, a left light guide plate 262 as a left optical image display unit 28, and a camera 61. And a 9-axis sensor 66.

  The 9-axis sensor 66 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 66 is provided in the image display unit 20, when the image display unit 20 is mounted on the user's head, it functions as a motion detection unit that detects the movement of the user's head. . Here, the movement of the head includes changes in the speed, acceleration, angular velocity, direction, and direction of the head.

  The right display driving unit 22 includes a receiving unit (Rx) 53, a right backlight (BL) control unit 201 and a right backlight (BL) 221 that function as a light source, a right LCD control unit 211 that functions as a display element, and a right An LCD 241 and a right projection optical system 251 are included. The right backlight control unit 201, the right LCD control unit 211, the right backlight 221 and the right LCD 241 are also collectively referred to as “image light generation unit”.

  The receiving unit 53 functions as a receiver for serial transmission between the control unit 10 and the image display unit 20. The right backlight control unit 201 drives the right backlight 221 based on the input control signal. The right backlight 221 is a light emitter such as an LED or electroluminescence (EL). The right LCD control unit 211 drives the right LCD 241 based on the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data Data1 input via the reception unit 53. The right LCD 241 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix. The right LCD 241 changes the transmittance of the light transmitted through the right LCD 241 by driving the liquid crystal at each pixel position arranged in a matrix, thereby converting the illumination light emitted from the right backlight 221 into an image. Modulate into effective image light to represent. In this embodiment, the backlight method is adopted, but image light may be emitted using a front light method or a reflection method.

  The right projection optical system 251 is configured by a collimator lens that converts the image light emitted from the right LCD 241 to light beams in a parallel state. The right light guide plate 261 as the right optical image display unit 26 guides the image light output from the right projection optical system 251 to the right eye RE of the user while reflecting the image light along a predetermined optical path. The optical image display unit can use any method as long as a virtual image is formed in front of the user's eyes using image light. For example, a diffraction grating or a transflective film may be used. good.

  The left display drive unit 24 has the same configuration as the right display drive unit 22. That is, the left display driving unit 24 includes a receiving unit (Rx) 54, a left backlight (BL) control unit 202 and a left backlight (BL) 222 that function as a light source, and a left LCD control unit 212 that functions as a display element. And a left LCD 242 and a left projection optical system 252.

  FIG. 3 is an explanatory diagram illustrating an example of a virtual image visually recognized by the user. FIG. 3A illustrates the user's visual field VR during a normal display process. As described above, the image light guided to both eyes of the user of the head mounted display 100 forms an image on the retina of the user, so that the user visually recognizes the virtual image VI. In the example of FIG. 3A, the virtual image VI is an OS standby screen of the head mounted display 100. Further, the user views the outside scene SC through the right optical image display unit 26 and the left optical image display unit 28. As described above, the user of the head mounted display 100 of the present embodiment can see the virtual image VI and the outside scene SC behind the virtual image VI for the portion of the visual field VR where the virtual image VI is displayed. In addition, the portion of the visual field VR where the virtual image VI is not displayed can be seen through the optical scene display portion and directly through the outside scene SC. In the present specification, “the head-mounted display 100 displays an image” includes causing the user of the head-mounted display 100 to visually recognize a virtual image as described above.

  FIG. 3B illustrates the visual field VR of the user during the augmented reality process. In the augmented reality processing, the AR processing unit 142 of the head mounted display 100 is a virtual object for expanding the outside scene SC perceived by the user, and represents a virtual object in which the user's visual discomfort is reduced. Are formed on the image display unit 20. Specifically, the AR processing unit 142 generates image data in which a visual effect that can alleviate the user's visual discomfort is added to the virtual object, and the generated image data is transmitted to the image display unit 20. To do. “Extending the outside scene SC” means adding, deleting, emphasizing, and attenuating information to the real environment that the user sees, that is, the outside scene SC. In the augmented reality processing of the first embodiment, the AR processing unit 142 generates different right-eye image data Data1 and left-eye image data Data2 in order to fuse a virtual object to the outside scene SC, thereby generating a virtual image. Add visual effects to enable stereoscopic viewing of objects. “Making a virtual object into the outside scene” means that the position of the outside scene SC that the user actually sees is a predetermined distance from the user (hereinafter also referred to as “target distance”). This means that a virtual image VI that gives the user a feeling as if a virtual object exists is displayed. In the example of FIG. 3B, an image representing an apple is displayed as a virtual image VI so as to overlap an actual road included in the outside scene SC. As a result, the user can feel as if an apple is falling on an empty road. In the example of FIG. 3B, the distance between the position of the user and the position at which the user feels “the apple is falling” corresponds to the target distance. In the example of FIG. 3B, an apple corresponds to the virtual object OB.

A-2. Augmented reality processing (first embodiment):
The AR processing unit 142 executes the augmented reality process according to the following procedures a1 to a3.
(A1) A target distance for visually recognizing the virtual object is determined.
(A2) Image data representing a virtual object is generated.
(A3) Image data for right eye Data1 and image data for left eye Data2 are generated from the image data.

FIG. 4 is an explanatory diagram for explaining the procedure of the augmented reality process. When the image processing unit 160 supplies the same right-eye image data Data1 and left-eye image data Data2 to the image display unit 20, the user moves the object to a position CO1 that is separated from the user by the initial imaging distance La. Recognize The convergence angle at this time is referred to as “initial convergence angle θa”. In step a1, the AR processing unit 142 determines a target distance Lb for allowing the user to visually recognize the virtual object OB in order to enable display of the virtual object OB with a sense of perspective. For example, the AR processing unit 142 can determine the target distance Lb by any of the following methods.
Analyzing the outside scene image obtained by the camera 61 in the direction of the visual field of the user.
・ Analyze the user's current position coordinates and head movements. In this case, the current position coordinates of the user are acquired from the position information of the control unit 10 detected by the GPS module 134. The movement of the user's head is acquired from the movement information detected by the 9-axis sensor 66.

  In procedure a2, the AR processing unit 142 generates image data representing the virtual object OB. In the augmented reality processing of the first embodiment, the AR processing unit 142 acquires image data corresponding to the analysis result of the procedure a1 from a plurality of image data stored in advance in the storage unit 120.

  In procedure a3, the AR processing unit 142 generates right-eye image data Data1 and left-eye image data Data2 from the image data generated in procedure a2. At this time, the AR processing unit 142 uses the pixel parallax angle 124 stored in the storage unit 120.

  FIG. 5 is an explanatory diagram for explaining the interocular distance 122 and the pixel parallax angle 124. In the present embodiment, the interocular distance 122 is the distance between the center of the optical image display unit 26 and the center of the left optical image display unit 28, and the distance DL between the right eye RE and the left eye LE of the user. It is considered. Therefore, the interocular distance 122 is a distance (for example, 65 mm) between the center of the right optical image display unit 26 and the center of the left optical image display unit 28 based on the design value of the head mounted display 100 in advance. Stored. The interocular distance 122 may store an actual distance DL between the user's right eye RE and left eye LE. Details will be described in a second embodiment of augmented reality processing. The interocular distance 122 may be adjustable according to the user's preference. The interocular distance 122 corresponds to an “interocular distance storage unit” in the claims.

The pixel parallax angle 124 is a parallax angle realized by image data shifted by one pixel left and right, and is represented by θpix (°). Specifically, the pixel parallax angle 124 is a convergence angle between a convergence angle of a virtual image displayed based on the same image data on the left and right and a virtual image displayed based on image data shifted by one pixel on the left and right. Difference. θpix is obtained as follows and is stored in advance in the pixel parallax angle 124.
The initial convergence angle θa when the right LCD 241 and the left LCD 242 are driven is measured based on the same right-eye image data Data1 and left-eye image data Data2.
The convergence angle θc when the right LCD 241 and the left LCD 242 are driven is measured based on the right-eye image data Data1 and the left-eye image data Data2 shifted by one pixel.
The difference between the initial convergence angle θa and the convergence angle θc is obtained (θc−θa). Since it is usually assumed that θc> θa, here, the initial convergence angle θa is subtracted from the convergence angle θc.
In order to obtain a difference in convergence angle per eye, the difference between the initial convergence angle θa and the convergence angle θc is divided by 2.

  That is, it can be expressed as θpix = (θc−θa) / 2. The initial convergence angle θa corresponds to the “first convergence angle” in the claims, the convergence angle θc corresponds to the “second convergence angle” in the claims, and the pixel parallax angle 124 is claimed. This corresponds to a “pixel parallax angle storage unit” in the range.

Returning to FIG. 4, the description of the augmented reality processing procedure will be continued. The target distance Lb (distance that allows the user to visually recognize the virtual object OB), the convergence angle θb, and the interocular distance 122 (DL / 2) determined in the procedure a1 are expressed by the following formula 1 using a trigonometric function: Can be expressed as Therefore, the AR processing unit 142 calculates the target convergence angle θb by applying the target distance Lb determined in the procedure a1 and the interocular distance 122 to Equation 1.
tan (θb / 2) = (DL / 2) / Lb
tan (θb / 2) = DL / (2 × Lb)
θb / 2 = arctan {DL / (2 × Lb)}
θb = 2 × arctan {DL / (2 × Lb)} (1)

The AR processing unit 142 calculates a target parallax angle θx that is a parallax angle at the target distance Lb (a distance that allows the user to visually recognize the virtual object OB) by using the obtained target convergence angle θb and initial convergence angle θa. Specifically, the AR processing unit 142 applies the values of the initial convergence angle θa and the target convergence angle θb to Equation 2, and calculates the target parallax angle θx.
θx = (θb−θa) / 2 (2)
In Expression 2, for example, when the set position is set to infinity, θx is expressed as θx = −θa / 2.

  The AR processing unit 142 processes the image data of the procedure a2 using the obtained target parallax angle θx and the pixel parallax angle 124, and generates right-eye image data Data1 and left-eye image data Data2. Specifically, θpix stored in the pixel parallax angle 124 is a parallax angle realized by image data shifted by one pixel left and right. Therefore, in order to generate the right-eye image data Data1 for realizing the target parallax angle θx, the original image data may be shifted by θx / θpix pixels in the left direction. Similarly, in order to generate the image data Data2 for the left eye for realizing the target parallax angle θx, the original image data may be shifted by θx / θpix pixels in the right direction.

  FIG. 6 shows how the AR processing unit 142 generates right-eye image data Data1 and left-eye image data Data2 from the image data DT as described above.

  The AR processing unit 142 transmits the right eye image data Data1 and the left eye image data Data2 generated as described above to the image processing unit 160. The image processing unit 160 transmits the received right-eye image data Data1 to the image display unit 20 via the transmission unit 51. Similarly, the received left-eye image data Data 2 is transmitted to the image display unit 20 via the transmission unit 52. Thereafter, the display process described with reference to FIG. 2 is executed. As shown in FIG. 3B, the user of the head mounted display 100 can visually recognize the three-dimensional virtual object OB in the visual field VR.

  Note that the human resolution when the visual acuity is 1.0 can be expressed by arctan (1.5 mm / 5000 mm), which is about 0.017 °. Therefore, if the error between the virtual image VI displayed by the right-eye image data Data1 and the left-eye image data Data2 and the outside scene SC is designed to be 0.017 ° or less, the actual size virtual image VI matching the outside scene SC can be obtained. Display is possible.

  As described above, according to the augmented reality processing of the first embodiment, the augmented reality processing unit (AR processing unit 142) generates image data (right-eye image data Data1 and left-eye image data Data2). The virtual object OB is fused to the outside scene SC in consideration of various conditions regarding the display environment of the virtual image VI (the size of the right LCD 241 and the left LCD 242 and the distance between the virtual images VI displayed in front of the user's left and right eyes). Therefore, it is possible to realize a head-mounted display device (head mounted display 100) that can provide an augmented reality that reduces the user's visual discomfort. Specifically, the difference (θpix) stored in the pixel parallax angle storage unit (pixel parallax angle 124) is a virtual image displayed based on the same right-eye image data Data1 and left-eye image data Data2. The second convergence angle θc of the virtual image displayed based on the first convergence angle θa (initial convergence angle θa) of VI and the right-eye image data Data1 and the left-eye image data Data2 shifted by one pixel to the left and right. And the difference. For this reason, the difference (θpix) stored in the pixel parallax angle storage unit (pixel parallax angle 124) is a parallax angle realized by image data shifted by one pixel to the left and right, and is various in relation to the display environment of the virtual image VI. It can be said that the parallax angle is determined in consideration of the above conditions, that is, the size of the right LCD 241 and the left LCD 242 and the distance between the virtual images VI displayed in front of the left and right eyes of the user. Therefore, the augmented reality processing unit (AR processing unit 142) uses the pixel parallax angle storage unit (pixel parallax angle 124) stored in the pixel parallax angle storage unit (pixel parallax angle 124) to display the virtual image VI display environment. The right eye image data Data1 and the left eye image data Data2 for fusing the virtual object OB to the outside scene SC can be generated in consideration of various conditions relating to the above.

  Furthermore, according to the augmented reality processing of the first embodiment, the augmented reality processing unit (AR processing unit 142) determines the target distance Lb that allows the user to visually recognize the virtual object OB, and determines the target distance from the determined target distance Lb. A target convergence angle θb that is a convergence angle at Lb is calculated, and a target parallax angle θx that is a parallax angle at the target distance Lb is calculated using the calculated target convergence angle θb and the first convergence angle θa (initial convergence angle θa). Is calculated. The augmented reality processing unit (AR processing unit 142) uses the target parallax angle θx calculated in this way and the difference (θpix) stored in the pixel parallax angle storage unit (pixel parallax angle 124). Different right-eye image data Data1 and left-eye image data Data2 can be generated from a single image data DT (FIG. 6) representing the virtual object OB.

A-3. Augmented reality processing (second embodiment):
In the augmented reality process of the second embodiment, the AR processing unit 142 generates image data representing a virtual object corresponding to the user's movement from a virtual three-dimensional object in a 3D (Three Dimensions) model space. Image data for right eye Data1 and image data for left eye Data2 are generated from the image data. The difference from the augmented reality process of the first embodiment is that the procedure of FIG. 7 is executed instead of the procedures a2 and a3. Since the procedure a1 is the same as that in the first embodiment, the description is omitted.

  FIG. 7 is a flowchart showing the procedure of augmented reality processing in the second embodiment. The AR processing unit 142 performs initial setting of the interocular distance 122 (step S102). Specifically, the AR processing unit 142 acquires the interpupillary distance of the user measured by the interpupillary distance measuring unit 62 (FIG. 2), and stores the acquired distance in the interocular distance 122. Thus, according to step S102, the interpupillary distance according to the user can be stored in the interocular distance 122.

  FIG. 8 is an explanatory diagram for explaining details of the augmented reality processing in the second embodiment. In step S104 of FIG. 7, the AR processing unit 142 sets a virtual viewpoint at a predetermined position in the 3D model space. Specifically, the AR processing unit 142 sets the virtual viewpoint IV at a position away from the virtual object OB1 in the 3D model space by the distance Lb obtained in the procedure a1 (FIG. 8). The virtual viewpoint IV is a position corresponding to the user's temple, and the direction of the virtual viewpoint IV (viewpoint direction VD) faces the virtual object OB1. Note that the position of the virtual viewpoint IV is defined by the movement amount (x, y, z) with respect to the origin coordinate in the 3D model space. The viewpoint direction VD is defined by angles (θx, θy, θz) with respect to the virtual viewpoint IV. Note that (θx, θy, θz) is also expressed as (roll angle, pitch angle, yaw angle) = (φ, θ, ψ). After setting the virtual viewpoint IV, the AR processing unit 142 executes the processes of steps S110 to S116 and the processes of steps S120 to S126 in parallel.

  After setting the virtual viewpoint IV, the AR processing unit 142 sets the virtual camera CML at a position corresponding to the left eye based on the position and orientation of the virtual viewpoint IV (step S110). Specifically, the AR processing unit 142 sets a virtual camera CML facing the viewpoint direction VD at a position away from the virtual viewpoint IV to the left side (interocular distance 122/2). Similarly, the AR processing unit 142 sets the virtual camera CMR at a position corresponding to the right eye based on the position and orientation of the virtual viewpoint IV (step S120). Specifically, the AR processing unit 142 sets a virtual camera CMR facing the viewpoint direction VD at a position away from the virtual viewpoint IV to the right side (interocular distance 122/2). FIG. 8 shows the virtual cameras CML and CMR set as described above.

  After setting the virtual camera, the AR processing unit 142 performs 2D (Two Dimensions) projection of the virtual camera CML (step S112). Specifically, the AR processing unit 142 converts a three-dimensional object (virtual objects OB1, OB2) in the 3D model space into two-dimensional planar information obtained by the virtual camera CML, and has a sense of depth. Generate an image. Similarly, the AR processing unit 142 performs 2D projection of the virtual camera CMR (step S122). Specifically, the AR processing unit 142 converts a three-dimensional object in the 3D model space into information on a two-dimensional plane obtained by the virtual camera CMR, and generates an image with a sense of depth.

After 2D projection of the virtual camera CML, the AR processing unit 142 scales the image obtained by the projection (step S114). Specifically, the AR processing unit 142 executes the following procedures b1 to b3.
(B1) The AR processing unit 142 applies the target distance Lb determined in the procedure a1 to the above equation 1, and calculates the target convergence angle θb.
(B2) The AR processing unit 142 calculates the target parallax angle θx by applying the obtained target convergence angle θb and initial convergence angle θa to the above equation 2.
(B3) The virtual image obtained in step S112 so that the 2D projection image of the virtual camera CML obtained in step S112 and the 2D projection image of the virtual camera CM arranged at the virtual viewpoint IV are shifted by θx / θpix pixels. Enlarge / reduce the 2D projection image of the camera CML. At this time, the AR processing unit 142 considers the resolution of the right LCD 241.

Similarly, after 2D projection of the virtual camera CMR, the AR processing unit 142 scales the image obtained by the projection (step S124). Specifically, the AR processing unit 142 executes the procedures b1 and b2 and the procedure c3 below.
(B3) The virtual image obtained in step S122 so that the 2D projection image of the virtual camera CMR obtained in step S122 and the 2D projection image of the virtual camera CM arranged at the virtual viewpoint IV are shifted by θx / θpix pixels. The 2D projection image of the camera CMR is enlarged / reduced. At this time, the AR processing unit 142 considers the resolution of the left LCD 242.

  FIG. 9 shows how the AR processing unit 142 scales an image obtained by projection using the 2D projection image DT of the virtual camera CM arranged at the virtual viewpoint IV as described above. In the above example, the virtual camera CMR is set to a position corresponding to the right eye, and the virtual camera CML is set to a position corresponding to the left eye. However, the position and the number of virtual cameras to be set can be changed. Thereafter, the AR processing unit 142 transmits the right-eye image data Data1 and the left-eye image data Data2 generated as described above to the image processing unit 160 (steps S116 and S126). Thereafter, the display process described with reference to FIG. 2 is executed, so that the user of the head mounted display 100 can visually recognize the three-dimensional virtual object OB in the visual field VR.

  As described above, the augmented reality processing unit (AR processing unit 142) also generates the image data (right eye image data Data1 and left eye image data Data2) by the augmented reality processing of the second embodiment. In consideration of various conditions regarding the display environment of the virtual image VI (the size of the right LCD 241 and the left LCD 242 and the distance between the virtual images VI displayed in front of the user's left and right eyes), the virtual objects OB1 and OB2 are fused to the outside scene SC. Therefore, it is possible to realize a head-mounted display device (head mounted display 100) that can provide an augmented reality that reduces the user's visual discomfort.

  Further, according to the augmented reality processing of the second embodiment, the augmented reality processing unit (AR processing unit 142) determines the target distance Lb for allowing the user to visually recognize the virtual objects OB1 and OB2, and from the determined target distance Lb. A target convergence angle θb that is a convergence angle at the target distance Lb is calculated, and a target parallax that is a parallax angle at the target distance Lb is calculated using the calculated target convergence angle θb and the first convergence angle θa (initial convergence angle θa). The angle θx is calculated. The augmented reality processing unit (AR processing unit 142) uses the target parallax angle θx calculated in this way and the difference (θpix) stored in the pixel parallax angle storage unit (pixel parallax angle 124). The right-eye image data Data1 and the left-eye image data Data2 can be generated by scaling the 2D projection images from the two virtual cameras (virtual cameras CMR, CML), respectively.

  Furthermore, according to the augmented reality processing of the second embodiment, the augmented reality processing unit (AR processing unit 142) first sets the virtual camera (virtual cameras CMR, CML) on the 3D model space to the user. The virtual viewpoint IV is arranged based on the target distance Lb for visually recognizing the virtual object. Then, the augmented reality processing unit (AR processing unit 142) arranges one virtual camera (virtual camera CMR) at a position away from the arranged virtual viewpoint IV by the interocular distance / 2, and the eye is arranged from the arranged virtual viewpoint. The other virtual camera (virtual camera CML) is arranged at a position at a distance of / 2. As a result, the virtual camera (virtual cameras CMR, CML) performs 2D projection in consideration of both the target distance Lb that allows the user to visually recognize the virtual objects OB1 and OB2 and the interocular distance (interocular distance 122) of the user. Images can be acquired.

A-4. Augmented reality processing (third embodiment):
In the augmented reality process of the third embodiment, in the augmented reality process of the second embodiment, a virtual object to be fused with the outside scene SC can be changed as the user moves. Hereinafter, only a procedure different from the augmented reality process of the second embodiment will be described. In the figure, the same parts as those of the second embodiment are denoted by the same reference numerals as those of the second embodiment described above, and detailed description thereof is omitted.

  FIG. 10 is a flowchart showing the procedure of augmented reality processing in the third embodiment. The difference from the second embodiment shown in FIG. 7 is that steps S202 and S204 are executed after step S104, and the process is shifted to step S202 after steps S116 and S126 are completed.

  In step S202, the AR processing unit 142 determines whether the user's viewpoint has moved. Specifically, the AR processing unit 142 determines that the user's viewpoint has moved when there is a change in at least one of the current position coordinates of the user and the movement of the head. Note that the current position coordinates of the user are acquired from the position information of the control unit 10 detected by the GPS module 134. The movement of the user's head is acquired from the movement information detected by the 9-axis sensor 66. When the user's viewpoint has not moved (step S202: NO), the AR processing unit 142 causes the process to transition to steps S110 and S120.

  When the viewpoint of the user has moved (step S202: YES), the AR processing unit 142 determines the virtual viewpoint at a predetermined position in the 3D model space based on the current position coordinates of the user and the movement of the head detected in step S202. Set. Details are the same as step S104 in FIG.

  As described above, according to the augmented reality process of the third embodiment, along with the movement of the viewpoint of the user, that is, with the change of at least one of the current position coordinates of the user and the movement of the head. The 2D projection image of the virtual camera CML and the 2D projection image of the virtual camera CMR can be generated and scaled to generate right-eye image data Data1 and left-eye image data Data2. As a result, in addition to the effects of the second embodiment, the virtual object to be fused with the outside scene SC can be changed with the movement of the user's viewpoint.

  As described above, according to the head-mounted display device (head mounted display 100) of the first embodiment, the image display unit 20 can reduce the visual discomfort of the user (that is, the virtual object). A virtual object OB to which a visual effect for enabling stereoscopic viewing) is attached is visually recognized by a user as a virtual image VI. Therefore, it is possible to provide an optically transmissive head-mounted display device that can provide an augmented reality that alleviates the user's visual discomfort.

B. Second embodiment:
In the second embodiment of the present invention, instead of the “visual effect for enabling stereoscopic viewing of a virtual object” exemplified in the first embodiment, the “visual effect that can alleviate the user's visual discomfort” is used. A configuration in the case of adopting “visual effect for harmonizing a virtual object with the environment around the user” will be described. Below, only the part which has a different structure and operation | movement from 1st Embodiment is demonstrated. In the figure, the same components as those of the first embodiment are denoted by the same reference numerals as those of the first embodiment described above, and detailed description thereof is omitted.

B-1. Configuration of head mounted display device:
FIG. 11 is a block diagram functionally showing the configuration of the head mounted display 100a in the second embodiment. The difference from the first embodiment shown in FIG. 2 is that a control unit 10 a is provided instead of the control unit 10. The storage unit 120 of the control unit 10a does not include the area information 122 and the identification image storage unit 124 (FIG. 2). The control unit 10 a includes an AR processing unit 142 a instead of the AR processing unit 142. The AR processing unit 142a differs from the first embodiment in the processing content of the “augmented reality processing” to be executed. Specifically, the AR processing unit 142a executes the augmented reality process described below instead of the augmented reality process of the first embodiment described as the first to third examples.

B-2. Augmented reality processing:
FIG. 12 is a flowchart showing the procedure of augmented reality processing in the second embodiment. In step S302, the AR processing unit 142a acquires three-dimensional information representing the environment around the user as a virtual three-dimensional object in the three-dimensional space. This three-dimensional information is a so-called “3D model” that represents the environment around the user. The AR processing unit 142a can acquire the three-dimensional information using any one of the following methods c1 to c3.

(C1) The AR processing unit 142a acquires three-dimensional information from an external device such as a cloud server connected to the head mounted display 100a.
(C2) When the three-dimensional information is stored in the storage unit 120 in advance, the AR processing unit 142a acquires the three-dimensional information from the storage unit 120.
(C3) The AR processing unit 142a uses the outside scene image captured by the camera 61 or the infrared light irradiation in front of the user's eyes, and the like to display the three-dimensional information representing the environment around the user. Generate the generated 3D information.

  In the case of the methods c1 and c2, the AR processing unit 142a can determine the acquisition range of the three-dimensional information based on the current position coordinates of the user detected by the GPS module 134. In addition, when generating three-dimensional information using the camera 61 in the method c3, the AR processing unit 142a causes the camera 61 to acquire a plurality of outside scene images whose angles are continuously changed. Then, the AR processing unit 142a generates three-dimensional information based on difference information between the nth (n is an arbitrary integer) outer scene image and the (n + 1) th outer scene image. Can do. The cameras 61 used at this time are preferably a plurality of cameras arranged so that the imaging areas intersect. Furthermore, it is preferable that at least one of the plurality of cameras is fixed at a position corresponding to the viewing direction of the user. In order to cause the camera 61 to acquire an outside scene image with continuously changing angles, the AR processing unit 142a determines that the camera 61 is in a case where the user's movement detected by the 9-axis sensor 66 exceeds a predetermined threshold. An instruction to capture an outside scene image may be issued.

  In step S304, the AR processing unit 142a identifies the position and direction in the three-dimensional information corresponding to the current position and line-of-sight direction of the user in the real space. In other words, it can be said that in step S304, the AR processing unit 142a identifies the current position and line-of-sight direction of the virtual user in the three-dimensional information. Specifically, the AR processing unit 142a uses any of the following methods d1 to d3.

(D1) The AR processing unit 142a compares the three-dimensional information acquired in step S302 with the outside scene image in the user's line-of-sight direction acquired by the camera 61 at the time of step S304. The current position and the line-of-sight direction of the virtual user are identified.
(D2) The AR processing unit 142a obtains the latitude / longitude information included in the three-dimensional information acquired in step S302 and the current position coordinates (latitude / longitude) of the user acquired by the GPS module 134 at the time of step S304. By comparing, the current position of the virtual user in the three-dimensional information is specified. In addition, the AR processing unit 142a specifies the virtual user's line-of-sight direction in the three-dimensional information using the movement of the user's head acquired by the 9-axis sensor 66.
(D3) When a plurality of signal generation devices are arranged in advance in the real space, the AR processing unit 142a uses the reception intensity of radio waves and ultrasonic waves received from the signal generation device, and virtually Specify the current position and line-of-sight direction of the user.

  In step S306, the AR processing unit 142a generates a mask object. The “mask object” is three-dimensional information viewed from the position and direction corresponding to the current position and line-of-sight direction of the user in the real space, in other words, 3 viewed from the virtual user's current position and line-of-sight direction. Dimensional information. The AR processing unit 142a generates a mask object using the three-dimensional information acquired in step S302 and the position and direction in the three-dimensional information specified in step S304.

  FIG. 13 is an explanatory diagram illustrating an example of a mask object. FIG. 13A shows the outside scene SC viewed from the current position and line-of-sight direction of the user in the real space. The outside scene SC is a scenery of an office district where buildings are lined up. FIG. 13B shows the mask object VM viewed from the position and direction in the three-dimensional information corresponding to the current position and line-of-sight direction of the user in the real space. The mask object VM includes “buildings”, “roads”, “tree stands”, “electric poles” in which the environment (outside scene SC) around the user shown in FIG. 13A is virtually represented in a three-dimensional space. The three-dimensional object is included.

  This mask object is used to add a visual effect to the virtual object, and is not displayed as an image. In step S306 in FIG. 12, three-dimensional information is acquired from an external device such as a cloud server connected to the head mounted display 100a based on the position and direction in the three-dimensional information specified in step S304. The mask object may be complemented.

  In step S306 of FIG. 12, the AR processing unit 142a generates image data representing a virtual object that the AR processing unit 142a displays as a virtual image VI in the augmented reality processing. Specifically, for example, the AR processing unit 142a acquires arbitrary image data according to the user's request or the purpose of processing from a plurality of image data stored in advance in the storage unit 120.

In step S310, the AR processing unit 142a places the virtual object generated in step S308 in the mask object generated in step S306. In this arrangement, the AR processing unit 142a may consider a target distance for arranging the virtual object in order to display the virtual object OB with a sense of perspective. The target distance can be determined by any of the following methods, for example.
Analyzing the outside scene image obtained by the camera 61 in the direction of the visual field of the user.
Analyzing the current position coordinates of the user acquired by the GPS module 134 and the movement of the user's head acquired by the 9-axis sensor 66.

  FIG. 14 is an explanatory diagram illustrating an example of the arrangement of the virtual object OB with respect to the mask object VM. The virtual object OB illustrated in FIG. 14 is “bicycle”. In the mask object VM, the position where the virtual object OB is arranged is “side of the sidewalk in front of the grove”. Here, from the position and direction in the three-dimensional information specified in step S304 (virtual user's current position and line-of-sight direction), the virtual object (bicycle) arranged in the mask object, the user, Between them, “electric poles” which are part of the mask object are arranged.

  In step S <b> 312 of FIG. 12, the AR processing unit 142 a gives an effect corresponding to the surrounding environment to the mask object VM in a state where the virtual object OB is arranged. In the present embodiment, the “effect according to the surrounding environment” means at least one of the following effects e1 to e3. Note that the effects e1 to e3 may be employed alone or in combination.

(E1) The AR processing unit 142a performs trimming of a portion of the virtual object OB that is behind the three-dimensional object included in the mask object VM. For example, in the case of the example of FIG. 14, the AR processing unit 142a is a virtual part of the bicycle (virtual object OB) that is behind the utility pole (a solid object included in the mask object VM), that is, as shown in FIG. If the bicycle is viewed from the current user's current position and line of sight, the area where the bicycle is hidden behind the utility pole is trimmed. Thus, according to the effect e1, the AR processing unit 142a allows the virtual object OB to be part of the virtual object OB so that the virtual object OB appears to be in the shadow of a three-dimensional object (electric pole in the illustrated example) that exists in the real space. Can be trimmed. For this reason, the image display part 20 can make a user visually recognize the virtual image as if the virtual object OB is behind the thing (electric pole) which exists in real space.

(E2) The AR processing unit 142a performs lighting according to the surrounding environment. Specifically, the AR processing unit 142a collects luminance information of each area of the mask object VM by recognizing an outside scene image captured by the camera 61. The luminance information includes, for example, a shadow area, specular reflection, and diffuse reflection. The AR processing unit 142a determines the type of light source to be added to the mask object VM, the color temperature of the light source, the intensity of the light source, and the like based on the collected luminance information. Here, the types of light sources include, for example, point light sources (Point Light), spot lights (Spot Light), parallel light sources (Directional Light), ambient light (Ambient Light), sky light (Hemisphere Light), and IBL (Image Based). Light) etc. Then, the AR processing unit 142a performs lighting of the virtual object OB by arranging a light source having the determined type, color temperature, and intensity in the mask object VM.

  FIG. 15 is an explanatory diagram showing a state of the virtual object OB after the writing is performed. In the example of FIG. 15, the shadow OBS of the virtual object OB is generated as a result of the lighting of the virtual object OB being performed by the parallel light source DL. As described above, according to the effect e2, the AR processing unit 142a can correct the virtual object OB so that the virtual object OB has light and dark in accordance with the real space. For this reason, the image display part 20 can make a user visually recognize a virtual image as if the virtual object OB is mixed with the thing which exists in real space. In addition, the light source illustrated in step S312 of FIG. 12 may be provided alone or in combination. Further, the AR processing unit 142a may collect the luminance information of each area of the mask object VM by a method other than the image analysis of the outside scene image (for example, calculation based on the current time and direction).

(E3) The AR processing unit 142a adjusts the behavior of the virtual object OB based on at least one of the restitution coefficient and the friction coefficient of the three-dimensional object included in the mask object VM. Specifically, the AR processing unit 142a obtains at least one of the coefficient of restitution and the coefficient of friction of the surface of the three-dimensional object included in the mask object VM by recognizing the outside scene image captured by the camera 61. . At this time, the AR processing unit 142a may obtain an average restitution coefficient and an average friction coefficient for one three-dimensional object, or may obtain a restitution coefficient and a friction coefficient for each region of the three-dimensional object. The AR processing unit 142a adjusts the behavior of the virtual object OB that changes over time based on the obtained coefficient of restitution, coefficient of friction, or both.

  FIG. 16 is an explanatory diagram showing how the behavior of the virtual object OB is adjusted based on the restitution coefficient and the friction coefficient. In the example of FIG. 16, the restitution coefficient XX (X is an arbitrary number) and the friction coefficient XX are obtained and stored for each region of the three-dimensional object included in the mask object VM. The AR processing unit 142a changes the motion DI of the ball-shaped virtual object OB according to the restitution coefficient XX and the friction coefficient XX of the three-dimensional object on the motion DI. In the example shown in the figure, the AR processing unit 142a adjusts the magnitude of the virtual object OB bouncing according to the coefficient of restitution and the friction coefficient of the center lane of a three-lane road. As described above, according to the effect e3, the AR processing unit 142a can adjust the behavior of the virtual object OB according to at least one of the restitution coefficient and the friction coefficient of an object existing in the real space. For this reason, the image display part 20 can make a user visually recognize the virtual image as if the virtual object OB is moving under the influence of the thing which exists in real space.

  In step S314 in FIG. 12, the AR processing unit 142a performs setting of the virtual camera and 2D projection by the virtual camera. Specifically, the AR processing unit 142a determines the position and direction in the mask object VM corresponding to the current position and line-of-sight direction of the user in the real space (in other words, the virtual user in the mask object VM). Current position and line-of-sight direction). Then, the AR processing unit 142a converts the virtual object OB in the mask object VM into two-dimensional planar information obtained by the virtual camera. Thereby, the AR processing unit 142a can obtain the image data of the virtual object OB to which the visual effect is added by two-dimensionalizing the virtual object OB in the mask object VM.

  FIG. 17 is an explanatory diagram showing image data obtained by 2D projection of a virtual camera. In the example of FIG. 17, the image data DT includes a virtual object (bicycle) with a part of the front wheel trimmed as a result of the effect e1 being added and a shadow OBS added as a result of the effect e2 being added. ing. Note that when setting the virtual camera in step S314, the AR processing unit 142a performs setting of the depth of field of the virtual camera and the like according to the distance from the user's eye to the user's gaze point. Is preferred. Then, the defocus effect can be added to the virtual object OB in the image data of the virtual object OB obtained by the 2D projection of the virtual camera. Further, the number of virtual cameras set in step S314 may be one, two virtual cameras corresponding to the left and right eyes of the user, or three or more virtual cameras.

  FIG. 18 is an explanatory diagram illustrating an example of a virtual image VI visually recognized by the user in the augmented reality processing according to the second embodiment. In step S316 of FIG. 12, the AR processing unit 142a transmits the image data DT generated as described above to the image processing unit 160. Thereafter, the display process described with reference to FIG. 2 is executed, so that the user of the head mounted display 100a visually recognizes the virtual object OB in which the visual effect (trimming and shadow OBS) is added to the visual field VR. Can do.

  As described above, according to the head-mounted display device (head-mounted display 100a) of the second embodiment, the image display unit 20 is a virtual object OB in which the user's visual discomfort is reduced, that is, the use The virtual object OB to which a visual effect (that is, a visual effect for harmonizing the virtual object with the surrounding environment) that can alleviate the user's visual discomfort is visually recognized as a virtual image VI. Therefore, it is possible to provide an optically transmissive head-mounted display device that can provide an augmented reality that alleviates the user's visual discomfort. The augmented reality processing unit (AR processing unit 142a) adds a visual effect to the virtual object OB using the mask object VM generated from the three-dimensional information, and harmonizes the virtual object OB with the surrounding environment. For this reason, the accuracy of harmony can be improved as compared with the conventional case where the virtual object is harmonized with the surrounding environment by applying a visual effect to the virtual object based on the image taken by the camera.

  Further, according to the head mounted display device (head mounted display 100a) of the second embodiment, the augmented reality processing unit (AR processing unit 142a) performs trimming (effect e1) according to the environment on the virtual object OB. ), Lighting according to the environment (effect e2), and behavior adjustment according to the environment (effect e3), at least one of the visual effects can be added. Further, the augmented reality processing unit (AR processing unit 142a) automatically recognizes the outside scene image (image in the user's field of view) captured by the image acquisition unit (camera 61), and automatically The surrounding environment (specifically, the luminance information of the effect e2, the restitution coefficient and the friction coefficient in the case of the effect e3) can be estimated.

C. Variation:
In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. In addition, the following modifications are possible.

・ Modification 1:
In the said embodiment, it illustrated about the structure of the head mounted display. However, the configuration of the head-mounted display can be arbitrarily determined without departing from the gist of the present invention. For example, each component can be added, deleted, converted, and the like.

  The allocation of components to the control unit and the image display unit in the above embodiment is merely an example, and various aspects can be employed. For example, the following aspects may be adopted. (I) A mode in which processing functions such as a CPU and a memory are mounted on the control unit, and only a display function is mounted on the image display unit. (Ii) Processing functions such as a CPU and a memory are provided in both the control unit and the image display unit. (Iii) a mode in which the control unit and the image display unit are integrated (for example, a mode in which the control unit is included in the image display unit and functions as a glasses-type wearable computer), (iv) instead of the control unit A mode in which a smartphone or a portable game machine is used, and (v) a mode in which the connection unit (code) is eliminated by adopting a configuration in which the control unit and the image display unit can perform wireless communication and wireless power feeding.

  In the above embodiment, for convenience of explanation, the control unit includes a transmission unit, and the image display unit includes a reception unit. However, each of the transmission unit and the reception unit of the above-described embodiment has a function capable of bidirectional communication, and can function as a transmission / reception unit. For example, the control unit shown in FIG. 2 is connected to the image display unit via a wired signal transmission path. However, the control unit and the image display unit may be connected by a connection via a wireless signal transmission path such as a wireless LAN, infrared communication, or Bluetooth (registered trademark).

  For example, the configurations of the control unit and the image display unit illustrated in FIG. 2 can be arbitrarily changed. Specifically, for example, the touch pad may be omitted from the control unit, and the operation may be performed using only the cross key. Further, the control unit may be provided with another operation interface such as an operation stick. Moreover, it is good also as what receives input from a keyboard or a mouse | mouth as a structure which can connect devices, such as a keyboard and a mouse | mouth, to a control part. Further, for example, in addition to an operation input using a touch pad or a cross key, an operation input using a foot switch (a switch operated by a user's foot) may be acquired. For example, after providing a line-of-sight detection unit such as an infrared sensor in the image display unit, the user's line of sight may be detected, and an operation input by a command associated with the movement of the line of sight may be acquired. For example, a user's gesture may be detected using a camera, and an operation input by a command associated with the gesture may be acquired. When detecting a gesture, a user's fingertip, a ring attached to the user's hand, a medical device to be used by the user's hand, or the like can be used as a mark for motion detection. If it is possible to acquire an operation input using a foot switch or a line of sight, the input information acquisition unit can acquire an operation input from the user even in a task in which it is difficult for the user to release his / her hand.

  For example, the head mounted display is a binocular transmissive head mounted display, but may be a monocular head mounted display. Further, it may be configured as a non-transmissive head mounted display in which the transmission of the outside scene is blocked when the user wears the head mounted display.

  FIG. 19 is an explanatory diagram showing an external configuration of a head mounted display in a modified example. In the case of the example of FIG. 19A, the difference from the head mounted display 100 shown in FIG. 1 is that the image display unit 20a includes a right optical image display unit 26a instead of the right optical image display unit 26. Instead of the left optical image display unit 28, a left optical image display unit 28a is provided. The right optical image display unit 26a is formed smaller than the optical member of the first embodiment, and is disposed obliquely above the right eye of the user when the head mounted display is mounted. Similarly, the left optical image display unit 28a is formed smaller than the optical member of the first embodiment, and is disposed obliquely above the left eye of the user when the head mounted display is mounted. In the example of FIG. 19B, the difference from the head-mounted display 100 shown in FIG. 1 is that the image display unit 20b includes a right optical image display unit 26b instead of the right optical image display unit 26. Instead of the left optical image display unit 28, a left optical image display unit 28b is provided. The right optical image display unit 26b is formed smaller than the optical member of the first embodiment, and is disposed obliquely below the right eye of the user when the head mounted display is mounted. The left optical image display unit 28b is formed smaller than the optical member of the first embodiment, and is disposed obliquely below the left eye of the user when the head mounted display is mounted. Thus, it is sufficient that the optical image display unit is disposed in the vicinity of the user's eyes. The size of the optical member forming the optical image display unit is also arbitrary, and the optical image display unit covers only a part of the user's eye, in other words, the optical image display unit completely covers the user's eye. It can also be realized as an unmounted head mounted display.

  For example, functional units such as an image processing unit, a display control unit, an AR processing unit, and an audio processing unit are realized by the CPU developing and executing a computer program stored in the ROM or hard disk on the RAM. Described. However, these functional units may be configured using an ASIC (Application Specific Integrated Circuit) designed to realize the function.

  For example, in the above-described embodiment, it is assumed that the image display unit is a head mounted display that is mounted like glasses, but the image display unit is a normal flat display device (liquid crystal display device, plasma display device, organic EL display device, etc. ). Also in this case, the connection between the control unit and the image display unit may be a connection via a wired signal transmission path or a connection via a wireless signal transmission path. If it does in this way, a control part can also be utilized as a remote control of a usual flat type display device.

  Further, as the image display unit, instead of the image display unit worn like glasses, an image display unit of another shape such as an image display unit worn like a hat may be adopted. Further, the earphone may be an ear-hook type or a headband type, or may be omitted. Further, for example, it may be configured as a head-up display (HUD, Head-Up Display) mounted on a vehicle such as an automobile or an airplane. Further, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet.

  For example, in the above-described embodiment, the secondary battery is used as the power source. However, the power source is not limited to the secondary battery, and various batteries can be used. For example, a primary battery, a fuel cell, a solar cell, a thermal cell, or the like may be used.

  For example, in the above embodiment, the image light generation unit is configured using a backlight, a backlight control unit, an LCD, and an LCD control unit. However, the above aspect is merely an example. The image light generation unit may include a configuration unit for realizing another method together with or in place of these configuration units. For example, the image light generation unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image generation unit can use a digital micromirror device or the like instead of the LCD. Further, for example, the present invention can be applied to a laser retinal projection type head-mounted display device.

Modification 2
In the first embodiment, an example of augmented reality processing has been described. However, the procedure of the augmented reality process is merely an example, and various modifications can be made. For example, some steps may be omitted, and other steps may be added. Further, the order of the steps to be executed may be changed.

  In the above-described embodiment, the pixel parallax angle is a convergence angle between a convergence angle of a virtual image displayed based on the same image data on the left and right and a virtual image displayed based on image data shifted by one pixel on the left and right. The difference was assumed. However, the pixel parallax angle may be a difference between the convergence angle of the virtual image displayed based on the same left and right image data and the convergence angle of the virtual image displayed based on the image data shifted left and right. The shift amount of the image data is not limited to one pixel (pixel).

  For example, in the augmented reality process, when the calculated target parallax angle θx is larger than parallel (that is, when θx <−θa / 2), the AR processing unit is an image for realizing such a target parallax angle θx. Data generation can be suppressed. By doing so, it is possible to relieve the user's uncomfortable feeling.

  If the parallax angle is too large, the user tends to feel uncomfortable. Therefore, for example, in the augmented reality process, when the calculated target parallax angle θx is equal to or larger than a predetermined threshold (that is, when θx> θlim), the AR processing unit realizes such a target parallax angle θx. Generation of image data can be suppressed. The predetermined threshold (θlim) is preferably “a boundary between parallax in which an object looks double and single parallax for the user”. The predetermined threshold (θlim) may be adjustable according to the user's preference.

  The AR processing unit adjusts the saturation, brightness, and contrast of the right-eye image data and the left-eye image data according to the determined target distance Lb or the calculated target parallax angle θx. May be. For example, in the case of image data representing a virtual object to be displayed in the distance, the sense of perspective felt by the user can be enhanced by reducing at least one of saturation, brightness, and contrast.

  The parallax angle θpix stored in the pixel parallax angle may be adjustable according to the user's preference. The user can adjust the parallax angle θpix small to enhance the perspective, and can adjust the parallax angle θpix large to reduce the perspective.

・ Modification 3:
In the first embodiment, the AR processing unit may realize the augmented reality process by pattern matching the outside scene image in the user's view direction acquired by the camera with the pixel parallax angle. Specifically, the image display unit includes a right-eye camera and a left-eye camera. The right-eye camera is a camera that is arranged at a position corresponding to the right eye of the user of the image display unit and can capture an outside scene in the front side direction of the image display unit. The left-eye camera is a camera that is arranged at a position corresponding to the left eye of the user of the image display unit and can capture an outside scene in the front side direction of the image display unit. The AR processing unit includes a target object included in the image captured by the right-eye camera (an object to be displayed in the vicinity of the virtual object) and a target object included in the image captured by the left-eye camera. It is also possible to determine the “target distance” in augmented reality processing using the amount of deviation between them and using the amount of deviation and the pixel parallax angle.

-Modification 4:
In the first embodiment, the AR processing unit may execute the augmented reality process only when a predetermined condition is satisfied. For example, after having a configuration capable of detecting the direction of the user's line of sight with respect to the image display unit, the AR processing unit satisfies at least one of the following conditions for the detected line of sight direction: Only in the case, the above augmented reality processing may be executed.
When the viewing angle is within a range of about 200 ° horizontal and about 125 ° vertical (for example, 75 ° downward and 50 ° upward).
-When it is within the range of about 30 ° in the horizontal direction and about 20 ° in the vertical direction, which is an effective visual field with excellent information receiving ability.
-When the gazing point is within the range of 60 ° to 90 ° horizontal and 45 ° to 70 ° vertical, which is a stable gazing field that appears quickly and stably.
-When the self-motion sensation (vection) induced by the image is within a range from about 20 ° where the induction of self-motion sensation begins to about 110 ° where the self-motion sense is saturated.

-Modification 5:
In the second embodiment, an example of augmented reality processing has been described. However, the procedure of the augmented reality process is merely an example, and various modifications can be made. For example, some steps may be omitted, and other steps may be added. Further, the order of the steps to be executed may be changed.

  The augmented reality process of the second embodiment has been described as an alternative process of the augmented reality process of the first embodiment. However, the augmented reality process of the second embodiment may be executed in parallel with the augmented reality process of the first embodiment.

  In step S302 of the augmented reality process of the second embodiment, three-dimensional information acquisition methods c1 to c3 are exemplified. However, the above method is merely an example, and the AR processing unit may acquire three-dimensional information using a method other than the methods c1 to c3. In addition, the AR processing unit may correct the three-dimensional information acquired using one method (for example, method c3) with the information acquired using another method (for example, method c1).

  In the augmented reality processing of the second embodiment, in order to further adjust the balance between the size of the real space and the size of the virtual object, it is projected as a virtual image in front of the user and the size of the area that can be captured by the camera. A process of matching (calibrating) the size of the region in a pseudo manner may be performed.

  The effects e1 to e3 in step S312 of the augmented reality process of the second embodiment are merely examples, and the effect added by the AR processing unit can be arbitrarily determined. For example, the AR processing unit may add the following effects instead of the effects e1 to e3 or together with the effects e1 to e3. Specifically, the AR processing unit adds an effect for imparting perspective to the virtual object according to the target distance at which the virtual object is placed. As an effect for imparting a sense of perspective, for example, a fog effect can be employed when the target distance is far, and a sharpness effect can be employed when the target distance is short.

Modification 6:
The present invention is not limited to the above-described embodiments, examples, and modifications, and can be realized with various configurations without departing from the spirit thereof. For example, the technical features in the embodiments, examples, and modifications corresponding to the technical features in each embodiment described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the above-described effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

10. Control unit (controller)
DESCRIPTION OF SYMBOLS 12 ... Lighting part 14 ... Touch pad 16 ... Cross key 18 ... Power switch 20 ... Image display part 21 ... Right holding part 22 ... Right display drive part 23 ... Left holding part 24 ... Left display drive part 26 ... Right optical image display part 28 ... Left optical image display unit 30 ... Earphone plug 32 ... Right earphone 34 ... Left earphone 40 ... Connection unit 42 ... Right cord 44 ... Left cord 46 ... Connecting member 48 ... Body code 51 ... Transmission unit 52 ... Transmission unit 53 ... Reception Unit 54: Receiver 61 ... Camera 62 ... Interpupillary distance measurement unit 110 ... Input information acquisition unit 100, 100a ... Head mounted display (head mounted display device)
120 ... Storage unit 122 ... Interocular distance (interocular distance storage unit)
124: Pixel parallax angle (pixel parallax angle storage unit)
130 ... Power supply 140 ... CPU
142, 142a ... AR processing unit (augmented reality processing unit)
DESCRIPTION OF SYMBOLS 160 ... Image processing part 170 ... Audio | voice processing part 180 ... Interface 190 ... Display control part 201 ... Right backlight control part 202 ... Left backlight control part 211 ... Right LCD control part 212 ... Left LCD control part 221 ... Right backlight 222 ... Left backlight 241 ... Right LCD
242 ... Left LCD
251 ... Right projection optical system 252 ... Left projection optical system 261 ... Right light guide plate 262 ... Left light guide plate PCLK ... Clock signal VSync ... Vertical sync signal HSync ... Horizontal sync signal Data ... Image data Data1 ... Right eye image data Data2 ... Left Image data for eyes OA ... External device PC ... Personal computer SC ... Outside view RE ... Right eye LE ... Left eye VI ... Virtual image ER ... End EL ... End AP ... End portion VR ... View VD ... View direction IV ... Virtual viewpoint OB ... virtual object OB1 ... virtual object OB2 ... virtual object OBS ... shadow of virtual object CM ... virtual camera CML ... virtual camera CMR ... virtual camera DI ... movement of virtual object VM ... mask object θa ... initial convergence angle (first convergence angle) )
θb ... target convergence angle θc ... convergence angle (second convergence angle θ)
θx ... Target parallax angle

SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms. One embodiment is a head-mounted display device, which allows a user to visually recognize a first virtual object and a second virtual object, image data representing the first virtual object, and the An augmented reality processing unit that generates image data representing a second virtual object and transmits the generated image data to the image display unit, wherein the augmented reality processing unit includes the first virtual object as the first virtual object. The right-eye image data and the left-eye image data are displayed so that the second virtual object is displayed at the convergence angle for the virtual object and the second virtual object is displayed at the convergence angle for the second virtual object. Produced as a head-mounted display device. In the head-mounted display device, the augmented reality processing unit calculates a convergence angle for the first virtual object using a preset first target distance, and for the second virtual object May be calculated using a preset second target distance. The head-mounted display device further includes a camera that captures the user's eyes, and the augmented reality processing unit calculates an interocular distance from an image of the user's eyes captured by the camera. The convergence angle for the first virtual object is calculated using the first target distance and the interocular distance, and the convergence angle for the second virtual object is calculated using the second target distance and the eye. You may calculate using distance between. In the head-mounted display device, the convergence angle for the first virtual object may be smaller than the convergence angle for the second virtual object. The head-mounted display device further includes a first convergence angle of a virtual image displayed based on the same right-eye image data and left-eye image data, and the right eye shifted to the left and right. An image parallax angle storage unit that stores a difference between the second convergence angle of the virtual image displayed based on the image data for the left eye and the image data for the left eye may be provided. The head-mounted display device further includes a camera that captures the user's eyes, and the augmented reality processing unit is configured to detect the user's line-of-sight direction from the image of the user's eyes captured by the camera. And a blur effect may be applied to either the first virtual object or the second virtual object in correspondence with the line-of-sight direction. In the head-mounted display device, the second virtual object may be an image obtained by enlarging or reducing the first virtual object. In the head-mounted display device, the augmented reality processing unit displays the right-eye image data and the left-eye image data so as to display the first virtual object at the first target distance. After the generation, the right eye image data and the left eye image data may be generated so that the first virtual object is displayed at a target distance different from the first target distance.

Claims (14)

A head-mounted display device that allows a user to visually recognize a virtual image and an outside scene,
An image display unit for allowing the user to visually recognize the virtual image;
Augmented reality, which is a virtual object that is an object for giving an augmented reality to the user, and that forms the virtual image representing the virtual object in which the user's visual discomfort is reduced A processing unit;
A head-mounted display device comprising: The head-mounted display device according to claim 1,
The augmented reality processing unit uses the three-dimensional information representing the environment around the user as a virtual three-dimensional object in a three-dimensional space, and harmonizes the virtual object with the environment, thereby the virtual object A head-mounted display device that alleviates the sense of discomfort with respect to the head. The head-mounted display device according to claim 2,
The augmented reality processing unit
Placing the virtual object in the three-dimensional information;
A visual effect corresponding to the environment is applied to at least one of the virtual object and the three-dimensional information, and then the virtual object is two-dimensionalized to alleviate the uncomfortable feeling with respect to the virtual object. Head mounted display device. The head-mounted display device according to claim 3,
The visual effect according to the environment is at least
Trimming a portion of the virtual object that is behind the three-dimensional object in the three-dimensional information;
Lighting the virtual object according to the environment;
A head including any one of adjustment of the behavior of the virtual object based on at least one of a coefficient of restitution and a coefficient of friction of the three-dimensional object in the three-dimensional information set according to the environment. Part-mounted display device. The head-mounted display device according to claim 3 or 4, further comprising:
An image acquisition unit that acquires an image of the user's visual field direction in a state in which the head-mounted display device is mounted;
The head-mounted display device, wherein the augmented reality processing unit estimates the environment by recognizing an image in the visual field direction acquired by the image acquisition unit. The head-mounted display device according to any one of claims 3 to 5,
The augmented reality processing unit is a head-mounted display device that considers a distance from the user's eye to the user's gaze point when two-dimensionalizing the virtual object. The head-mounted display device according to claim 1,
The augmented reality processing unit generates right-eye image data for the right eye representing the virtual object and left-eye image data for the left eye, so that the virtual object Add a visual effect to enable stereoscopic viewing to alleviate the uncomfortable feeling with respect to the virtual object,
The image display unit uses the right-eye image data and the left-eye image data to cause the left and right eyes of the user to visually recognize the different virtual images,
Furthermore, the first convergence angle of the virtual image displayed based on the same right-eye image data and the left-eye image data, the right-eye image data and the left-eye image shifted to the left and right A pixel parallax angle storage unit that stores a difference between the second convergence angle of the virtual image displayed based on the data,
The augmented reality processing unit, using the difference stored in the pixel parallax angle storage unit, the image data for the right eye and the image data for the left eye for fusing the virtual object to the outside scene A head-mounted display device that generates The head-mounted display device according to claim 7,
The augmented reality processing unit
Determining a target distance for the user to visually recognize the virtual object;
From the determined target distance, a target convergence angle that is a convergence angle at the target distance is calculated,
Using the calculated target convergence angle and the first convergence angle, calculate a target parallax angle that is a parallax angle at the target distance;
Using the target parallax angle and the difference stored in the pixel parallax angle storage unit, from the single image data representing the virtual object, the image data for the right eye and the image data for the left eye A head-mounted display device that generates The head-mounted display device according to claim 7,
The augmented reality processing unit
Determining a target distance for the user to visually recognize the virtual object;
Based on the determined target distance, two virtual cameras for acquiring a 2D projection image on the 3D model space are set,
From the determined target distance, a target convergence angle that is a convergence angle at the target distance is calculated,
Using the calculated target convergence angle and the first convergence angle, calculate a target parallax angle that is a parallax angle at the target distance;
Using the target parallax angle and the difference stored in the pixel parallax angle storage unit, the 2D projection image by one of the virtual cameras is scaled to generate the right eye image data, and the other A head-mounted display device that generates the left-eye image data by scaling a 2D projection image by a virtual camera. The head-mounted display device according to claim 9, further comprising:
An interocular distance storage unit that stores the interocular distance of the user;
The augmented reality processing unit, when setting the two virtual cameras, arranges a virtual viewpoint that is a virtual viewpoint on the 3D model space based on the determined target distance, and the arranged virtual viewpoint A head-mounted display device in which one virtual camera is arranged at a position away from the eye distance / 2 from the eye, and the other virtual camera is arranged at a position away from the arranged virtual viewpoint at an eye distance / 2. The head-mounted display device according to claim 10, further comprising:
An interpupillary distance measuring unit for measuring the interpupillary distance of the user;
A head-mounted display device in which a measurement result by the interpupillary distance measurement unit is stored in the interocular distance storage unit as the interocular distance. A method for controlling a head-mounted display device in which a user can visually recognize a virtual image and an outside scene,
Making the user visually recognize the virtual image;
Forming the virtual image representing the virtual object, which is a virtual object that is an object for giving augmented reality to the user, and in which the user's visual discomfort has been relaxed, in the step of visually recognizing; ,
A method for controlling a head-mounted display device. A method for controlling a head-mounted display device according to claim 12,
The generating step harmonizes the virtual object with the environment using three-dimensional information expressing the environment around the user as a virtual three-dimensional object in a three-dimensional space. A method for controlling a head-mounted display device that alleviates the uncomfortable feeling. A method for controlling a head-mounted display device according to claim 12,
The generating step generates right eye image data for the right eye representing the virtual object and left eye image data for the left eye, so that the virtual object is three-dimensional with respect to the virtual object. Add a visual effect to enable viewing to alleviate the sense of discomfort with the virtual object,
The visualizing step uses the right eye image data and the left eye image data to cause the left and right eyes of the user to visually recognize the different virtual images,
The generating step includes the first convergence angle of the virtual image displayed based on the same right-eye image data and the left-eye image data, the right-eye image data shifted to the left and right, and the The right-eye image data and the left-eye for fusing the virtual object to the outside scene using the difference between the second convergence angle of the virtual image displayed based on the left-eye image data Control method for a head-mounted display device that generates image data for a head.

Images