JP2015026286A - Display device, display system and control method of display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To eliminate an image that may damage realistic feeling to improve the realistic feeling.SOLUTION: A transparent head-mounted display device 1 for allowing a user to visually recognise an image as a virtual image comprises: an object detection section 73 detecting a coordinate position and a shape of a real object A in a real space; a display control section 74 causing an image object 26 having a set coordinate position in the real space to be displayed so that the image object is visually recognised according to a coordinate system in the real space; and a display section 30 displaying the image object 26. The display control section 74 controls the image object 26 in accordance with the coordinate position and shape of the real object A.

Description

  The present invention relates to a transmissive display device that allows a user to visually recognize an image as a virtual image, a display system, and a control method for the display device.

  Conventionally, as this type of display device, an optically transmissive head-mounted display device mounted on a user's head is known (see Patent Document 1). This display device is configured such that a user can visually recognize an actual scenery (outside scene) at the same time as visually recognizing an image as a virtual image.

JP 2012-163637 A

By the way, in this type of display device, for the purpose of fusing virtual reality with the real world, an image object in which a coordinate position in the real space is set is displayed so as to be visually recognized according to the coordinate system in the real space. Configuration (fitting) was considered. In other words, the image object is arranged in accordance with the coordinate system of the real space, and is made to recognize as if the image object actually exists in the real space being visually recognized by being superimposed on the real scenery. . According to such a configuration, since the image object is superimposed on the visually recognized real scene, a higher realism than that of the so-called augmented reality (AR) in which the captured image and the image object are combined on the image. Can be obtained.
However, there is a problem that the image object behaves ignoring the real object simply by displaying the image object so as to be visually recognized according to the coordinate system of the real space. For example, if an image object exists in the real space, the image object passes through the real object even in a case where the image object collides with the real object. Further, even when the image object is hidden behind the real object when viewed from the user, the image object can be recognized. Such a phenomenon gives a sense of incongruity to the user and causes a serious loss of realism.

  It is an object of the present invention to apply a display device, a display system, and a display device control method that can eliminate such a phenomenon and improve the real feeling.

  The display device of the present invention is a transmissive display device that allows a user to visually recognize an image as a virtual image, and includes an object detection unit that detects a coordinate position and a shape of a real object in the real space, and a coordinate position in the real space. And a display control unit that displays the image object so that the image object is visually recognized according to the coordinate system of the real space, and a display unit that displays the image object. It is characterized by being controlled according to the coordinate position and shape.

  In this case, the display control unit preferably controls the behavior of the image object according to the coordinate position and shape of the real object.

  In this case, when the image object is positioned behind the real object as viewed from the user based on the detected coordinate position of the real object and the set coordinate position of the image object, the display control unit It is preferable to change the image object so that the overlapping portion with the real object disappears.

  The display control unit preferably causes the image object to behave so as to avoid the real object in the coordinate system of the real space.

  Furthermore, it is preferable that the display control unit behaves as if the image object collided with the real object when the moved image object reaches the real object in the coordinate system of the real space.

  The display system of the present invention is a transmissive display system that allows a user to visually recognize a display image as a virtual image, and includes an object detection unit that detects a coordinate position and a shape of a real object in the real space, and a coordinate in the real space. A display control unit configured to display an image object with a position set so as to be visually recognized according to a coordinate system of a real space; and a display unit configured to display a display image. Control is performed according to the coordinate position and shape of the body.

  The display device control method of the present invention is a transmission type display device control method that allows a user to visually recognize a display image as a virtual image, and includes an object detection step of detecting a coordinate position and a shape of a real object in a real space; A display control step for displaying an image object in which a coordinate position in the real space is set so as to be visually recognized according to a coordinate system of the real space; and a display step for displaying the image object. Then, the image object is controlled according to the coordinate position and shape of the real object.

  According to these configurations, the image object can be made to behave in consideration of the coordinate position and shape of the real object, so that the phenomenon as described above can be eliminated and the realism can be improved. That is, in the case where the image object collides with the real object, the image object can behave as if it collided, and in the case where the image object is hidden behind the real object when viewed from the user, An image object can be changed as if it was hidden behind a real object. Further, by causing the image object to behave in consideration of the coordinate position and shape of the real object, it is possible to provide content in which the real object is entangled in the transmissive display device. For example, it is possible to provide a game or a navigation service involving an actual object using an image object.

  The display device preferably further includes a line-of-sight detection unit that detects a user's line of sight, and the display control unit displays the image object based on the detected line of sight.

  According to this configuration, it is possible to perform fitting of the image object with higher accuracy by displaying the image object in consideration of the user's line of sight.

  In addition, it is preferable that a distance detection unit that detects a distance from the user to the real object is further provided, and the display control unit displays the image object based on the detected distance.

  According to this configuration, by displaying the image object in consideration of the distance from the user to the real object, the fitting of the image object can be performed with higher accuracy.

  On the other hand, the display unit includes an image light output unit that outputs image light of an image, and a light guide unit that guides the output image light to the user's eyes, and the light guide unit transmits external light. The external light is preferably incident on the user's eye together with the image light.

  According to this configuration, by using a so-called virtual image projection type transmissive display device, the fitting of the image object can be performed with higher accuracy, and the realism can be improved.

It is explanatory drawing which shows the external appearance structure of the head mounted display apparatus which concerns on embodiment. It is a figure explaining an example of the virtual image and real scenery which a user can visually recognize. It is a block diagram explaining the control structure of a head mounting type display apparatus. It is a flowchart explaining the display operation | movement of the image object by a head mounted display apparatus. It is the transition figure which showed the 1st example of the behavior control of the image object by a display control part. It is the transition figure which showed the 2nd example of the behavior control of the image object by a display control part. It is the transition figure which showed the 3rd example of the behavior control of the image object by a display control part.

  Hereinafter, a display device, a display system, and a display device control method according to an embodiment of the present invention will be described with reference to the accompanying drawings. In this embodiment, a head-mounted display device to which the display device of the present invention is applied is exemplified. This head-mounted display device is a display device mounted on a user's head, and is also called a head mounted display (HMD). The head-mounted display device of the present embodiment is an optically transmissive head-mounted display device (transparent head-mounted display device) that allows a user to visually recognize an image as a virtual image and at the same time also visually recognize a real scene (outside scene). Display device, see-through type head-mounted display), and has a glasses-type (goggles-type) shape.

  As shown in FIG. 1, the head-mounted display device 1 includes a right holding unit 11, a right display driving unit 12, a left holding unit 13, a left display driving unit 14, a right optical image display unit 15, A left optical image display unit 16, an external camera 17, a right earphone 18 for the right ear, and a left earphone 19 for the left ear are provided.

  The right optical image display unit 15 and the left optical image display unit 16 are arranged at positions corresponding to the right and left eyes of the user when the head-mounted display device 1 is worn. One end of the right optical image display unit 15 and one end of the left optical image display unit 16 are connected at a position corresponding to the eyebrow of the user when the head-mounted display device 1 is mounted. The right holding unit 11 extends from the end ER that is the other end of the right optical image display unit 15. Similarly, the left holding unit 13 extends from the end EL which is the other end of the left optical image display unit 16.

  The right holding unit 11 is substantially perpendicular to the right optical image display unit 15 from the end ER of the right optical image display unit 15 to a position corresponding to the user's temporal region when the head-mounted display device 1 is mounted. It is a member provided by stretching so as to be formed. Similarly, the left holding unit 13 is connected to the left optical image display unit 16 from the end EL of the left optical image display unit 16 to a position corresponding to the user's temporal region when the head-mounted display device 1 is worn. It is a member provided by extending so as to form a substantially right angle. The right holding unit 11 and the left holding unit 13 hold the head-mounted display device 1 on the user's head like a temple (a vine or an ear hook) of glasses.

  The right display drive unit 12 is located on the inner side of the right holding unit 11, in other words, on the side facing the user's head when the head-mounted display device 1 is mounted, and at the end of the right optical image display unit 15. It is arranged on the part ER side. The left display driving unit 14 is disposed inside the left holding unit 13 and on the end EL side of the left optical image display unit 16.

  The external camera 17 captures an image of a real scene 23 in the direction of the user's field of view (in front of the user) when the head-mounted display device 1 is mounted. Although details will be described later, an image captured by the external camera 17 is used for recognition of the marker MK provided on the real object A, and in turn, recognition of the real object A.

  When the user wears the head-mounted display device 1, the right earphone 18 and the left earphone 19 are worn on the right and left ears, respectively, and perform various audio outputs to the right and left ears.

  Next, with reference to FIG. 2, an example of the virtual image 22 and the real scene 23 that can be visually recognized by the user wearing the head-mounted display device 1 will be described. As shown in FIG. 2, a virtual image 22 generated by the head-mounted display device 1 is displayed in the field of view 21 of the user wearing the head-mounted display device 1. Here, an image object 26 representing a maze navigator (guide) is displayed as the virtual image 22. In the user's field of view 21, the user can see the real scene 23 through the right optical image display unit 15 and the left optical image display unit 16. Here, as the real scenery 23, the walls A1, A2, A3, etc. of the maze that are the real objects A are visible. That is, the user is configured to be able to visually recognize the virtual image 22 and the real scene 23 simultaneously in the field of view 21. Reference numeral 27 denotes a virtual image display area in which the virtual image 22 is displayed in the field of view 21 (the virtual image 22 can be displayed).

  Next, the control configuration of the head-mounted display device 1 will be described with reference to FIG. As shown in FIG. 3, the head-mounted display device 1 includes a display unit 30 that allows a user to visually recognize a virtual image 22, an external camera 17, a right earphone 18, a left earphone 19, a sensor unit 31, It has a power supply unit 32, a wireless communication interface 33, and a control unit 34 for controlling these units.

  The sensor unit 31 detects various information related to the control of the head-mounted display device 1. Specifically, the sensor unit 31 includes a distance sensor (distance detection unit) 41, a GPS receiver 42, a nine-axis sensor 43, and a gaze detection sensor (gaze detection unit) 44. The distance sensor 41 detects the distance from the user (strictly speaking, the head-mounted display device 1) to the real object A by, for example, an infrared method or an ultrasonic method. The GPS receiver 42 receives the GPS signal and acquires position information indicating the position of the user (strictly speaking, the head-mounted display device 1). The 9-axis sensor 43 detects 3-axis acceleration, 3-axis angular velocity, and 3-axis geomagnetism. The line-of-sight detection sensor 44 detects the user's line of sight, for example, by an infrared method.

  The power supply unit 32 supplies power to each unit of the head-mounted display device 1. The wireless communication interface 33 is for performing communication with various external devices and external servers that are the sources of content data such as images (still images and moving images) and audio by wireless communication. Examples of the external device include a personal computer (PC), a mobile terminal (such as a mobile phone and a smartphone), and a game terminal. In the present embodiment, content data of a game or navigation service is acquired by the wireless communication interface 33, and display control of the image object 26 is performed based on the acquired content data.

  The display unit 30 includes a right display drive unit (image light output unit) 12, a left display drive unit (image light output unit) 14, a right light guide plate (light guide unit) 46 as the right optical image display unit 15, A left light guide plate (light guide portion) 47 as the left optical image display portion 16.

  The right display drive unit 12 includes a right backlight (BL) control unit 51 and a right backlight (BL) 52 that function as light sources, a right LCD (Liquid Crystal Display) control unit 53 and a right LCD 54 that function as display elements, Right projection optical system 55.

  The right backlight control unit 51 drives and controls the right backlight 52 based on the input control signal. The right backlight 52 is a light emitter such as an LED (Light Emitting Diode) or electroluminescence (EL). The right LCD control unit 53 drives and controls the right LCD 54 based on the input display data. The right LCD 54 is a transmissive liquid crystal panel in which a plurality of pixels are arranged in a matrix.

  The right projection optical system 55 projects (outputs) image light emitted from the right LCD 54, and is configured using, for example, a collimator lens. The right light guide plate 46 as the right optical image display unit 15 is formed of a light-transmitting resin material or the like, and reflects the image light output from the right projection optical system 55 along a predetermined optical path while being reflected by the user. Lead to the right eye. Further, the right light guide plate 46 transmits external light (light in real scenery) and makes the external light enter the user's eyes together with the image light.

  Similarly to the right display drive unit 12, the left display drive unit 14 is a left backlight (BL) control unit 61, a left backlight (BL) 62, a left LCD control unit 63, a left LCD 64, and left projection optics. System 65. Since the configuration and function of each element included in the left display driving unit 14 are the same as those included in the right display driving unit 12, the description thereof is omitted here. The left light guide plate 47 as the left optical image display unit 16 is formed of a light transmissive resin material or the like, and is used while reflecting the image light output from the left projection optical system 65 along a predetermined optical path. To the left eye. The left light guide plate 47 transmits external light and causes the external light to enter the user's eyes together with the image light.

  In this way, the image light guided to both eyes of the user of the head-mounted display device 1 forms an image on the retina, so that the user can enter the virtual image display area 27 as shown in FIG. The virtual image 22 (image object 26) is visually recognized. Here, the imaging of image light on the retina is referred to as “display” or “display”.

  The control unit 34 includes a CPU (Central Processing Unit) 67 and a storage unit 68. The storage unit 68 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and stores various computer programs. The CPU 67 reads out and executes a computer program from the storage unit 68, thereby operating as an operating system (OS) 71, an image analysis unit 72, an object detection unit 73, a display control unit 74, a display state control unit 75, and an audio processing unit 76. Function.

  The image analysis unit 72 acquires an image signal of an image captured by the external camera 17 and analyzes the acquired image signal. The image analysis unit 72 recognizes the object recognition marker MK (see FIG. 2) provided on the real object A by image analysis, and recognizes the real object A based on information obtained from the marker MK. That is, information on the position and orientation of the marker MK in the captured image, read information obtained by reading the marker MK, and information on the real object A stored in advance (the shape of the real object A and the arrangement position of the marker MK on the real object A) ) To recognize the position of the real object A in the field of view 21 and the shape of the real object A. In this embodiment, the marker MK is provided on the real object A, and the real object A is recognized by recognizing the marker MK. However, the feature position (feature point) of the real object A can be obtained without the marker. The structure which recognizes and recognizes the real object A may be sufficient. Further, the information on the real object A described above may be acquired from an external server by the wireless communication interface 33 every time image recognition is performed.

  The object detection unit 73 detects the coordinate position and shape of the real object A in the real space based on the recognition result of the real object A by the image analysis unit 72 and the detection result by the sensor unit 31. Specifically, the recognition result of the real object A by the image analysis unit 72, the information on the distance from the user to the real object A obtained from the distance sensor 41, and the user's head obtained by the 9-axis sensor 43 The relative coordinate position of the real object A with respect to the user in the real space and the shape of the real object A are detected based on the information on the orientation of the real object A.

  The display control unit 74 generates display data displayed by the display unit 30 and transmits the display data to the display unit 30 (the right LCD control unit 53 and the left LCD control unit 63). The display control unit 74 sets a coordinate position in the real space for the image object 26, and causes the image object 26 in which the coordinate position in the real space is set as display data to be visually recognized in accordance with the coordinate system of the real space. In this way, what is displayed (so-called fitting) is generated.

  Specifically, the display control unit 74 sets a relative coordinate position with respect to the real object A in the real space for the image object 26 based on the content data. The relative coordinate position of the image object 26, the detected coordinate position of the real object A, the information on the orientation of the user's head, the information on the user's line of sight obtained from the line-of-sight detection sensor 44, and Based on the above, the display position, display size, and orientation of the image object 26 in the virtual image display area 27 to be visually recognized according to the coordinate system of the real space (viewed according to the coordinate system of the real space) are specified. For example, when the image object 26 exists in the coordinate system of the real space, the display size is determined by the distance from the user to the image object 26. Therefore, the display size is specified by the distance from the user to the image object 26 calculated based on the above information. Based on the specified display position, display size, and orientation, the image object 26 is enlarged / reduced, rotated, and arranged to generate display data including the image object 26.

  When the image object 26 is displayed as a stereoscopic image, display data giving a parallax to the image object 26 is generated as display data for the right eye and display data for the left eye, and the display data for the right eye is converted to the right eye. The display data for the left eye is transmitted to the LCD control unit 53 to the left LCD control unit 63. In this case, the parallax amount (depth feeling) of the image object 26 is also set in accordance with the coordinate system of the real space. That is, as with the display size, the amount of parallax is determined by the distance from the user to the image object 26. Therefore, the distance from the user to the image object 26 is calculated based on the above information, and the image is based on this. The parallax amount of the object 26 is set.

  Further, the display control unit 74 controls the behavior of the image object 26 according to the detected coordinate position and shape of the real object A. That is, in the above description, it is described that the relative coordinate position in the real space is set based on the content data. However, strictly speaking, based on the content data and the detected coordinate position and shape of the real object A, The relative coordinate position is set. Further, the image object 26 is changed based on the detected coordinate position and shape of the real object A. As a result, the image object 26 behaves in consideration of the coordinate position and shape of the detected real object A.

  The display state control unit 75 controls the image display state on the display unit 30 by generating control signals for controlling the right display drive unit 12 and the left display drive unit 14 and transmitting them to the display unit 30. Specifically, the display state control unit 75 turns ON / OFF driving of the right LCD 54 by the right LCD control unit 53, ON / OFF driving of the right backlight 52 by the right backlight control unit 51, and left by a control signal. The right display driving unit 12 and the left display driving unit are individually controlled by controlling the driving of the left LCD 64 by the LCD control unit 63 and the ON / OFF of the driving of the left backlight 62 by the left backlight control unit 61 individually. The generation and emission of image light by each of 14 is controlled.

  The sound processing unit 76 transmits a sound signal to the right earphone 18 and the left earphone 19 to cause the right earphone 18 and the left earphone 19 to output sound.

  Next, the display operation of the image object 26 by the head-mounted display device 1 will be described with reference to FIG. This display operation is performed due to an operation start operation using a controller (not shown).

  As shown in FIG. 4, the head-mounted display device 1 first captures an actual scene 23 by the external camera 17 (S1). Then, the image analysis unit 72 recognizes the marker MK in the captured image of the external camera 17 and recognizes the real object A (S2). In the example of FIG. 2, the walls A1 and A2 that can be recognized by the marker MK are recognized from the captured image.

  When the real object A is recognized, the distance sensor 41 detects the distance from the user to the real object A (S3), and the 9-axis sensor 43 detects the orientation of the user's head (S4). Based on the recognition result of the real object A by the image analysis unit 72, information on the distance from the detected user to the real object A, and information on the detected head orientation, the object detection unit 73 The coordinate position (relative coordinate position with respect to the user) and shape of the body A are detected (S5).

  When the coordinate position and shape of the real object A are detected, the visual line detection sensor 44 detects the visual line of the user (S6). Then, the display control unit 74 estimates the user's visual field 21 (orientation) based on the detected line-of-sight information and the detected head-orientation information (S7).

  Thereafter, the display control unit 74 sets the coordinate position of the image object 26 in the real space (relative coordinate position with respect to the real object A) based on the content data and the detected coordinate position and shape of the real object A. (S8). Then, based on the estimated field of view 21, the set coordinate position of the image object 26 (relative coordinate position with respect to the real object A), and the detected coordinate position of the real object A (relative coordinate position with respect to the user). Then, the display position, display size, and orientation of the image object 26 are determined so as to be visually recognized according to the coordinate system of the real space (S9), and display data including the image object 26 is generated (S10). Thereafter, the generated display data is transmitted to the display unit 30 (S11). The “display control step” described in the claims includes steps S6 to S10.

  Then, the display unit 30 displays the display data including the image object 26 by the display unit 30 (projects image light onto the retina) (S12: display step). Thereby, this display operation is completed.

  Here, the behavior control of the image object 26 by the display control unit 74 will be described with reference to FIGS. As described above, the display control unit 74 controls the behavior of the image object 26 according to the detected coordinate position and shape of the real object A. FIG. 5 is a transition diagram showing a first example of behavior control of the image object 26 by the display control unit 74. In the example of FIG. 5, the display control unit 74 determines that the image object 26 is the real object A as viewed from the user based on the detected coordinate position of the real object A and the set coordinate position of the image object 26. When the image object 26 is located at the back (presumed to be), the image object 26 is changed so that the overlapping portion 26a of the image object 26 with the real object A disappears. Specifically, as shown in the figure, when the image object 26 is moved to the rear side of the wall A1 when viewed from the user, the image object 26 is overlapped with the wall A1 when viewed from the user. Change so that disappears. In this way, the image object 26 is displayed as if it was hidden behind the wall A1 when viewed from the user.

  FIG. 6 is a transition diagram showing a second example of behavior control of the image object 26 by the display control unit 74. In the example of FIG. 6, the display control unit 74 causes the image object 26 to behave so as to avoid the real object A in the coordinate system of the real space. Specifically, as shown in the figure, when the image object 26 is moved to the vicinity of the pillar A4 which is an obstacle, the trajectory of the image object 26 is changed and displayed as if the real object A was avoided. Let

  FIG. 7 is a transition diagram illustrating a third example of behavior control of the image object 26 by the display control unit 74. In the example of FIG. 7, the display control unit 74 behaves as if the image object 26 collided with the real object A when the moved image object 26 reaches the real object A in the coordinate system of the real space. Let Specifically, as shown in the figure, when the image object 26 is moved until reaching the wall A2, the trajectory of the image object 26 is changed so as to bounce off the wall A2, and the object A collides with the real object A. Is displayed. Further, when the image object 26 reaches the wall A1, the image object 26 is changed as if the image object 26 collided with the real object A (in the example of the figure, the image object 26 is reduced and deformed in the colliding direction).

  As described above, according to the configuration of the embodiment, the image object 26 can be made to behave in consideration of the coordinate position and the shape of the real object A, and thus the real feeling can be improved. In addition, by causing the image object 26 to behave in consideration of the coordinate position and shape of the real object A, it is possible to provide content tangled with the real object A in a transmissive display device. Therefore, it is possible to provide a game or navigation service involving the real object A using the image object 26. As content of the navigation service, for example, a guide for a maze is displayed as the image object 26, and the guide guides the route while avoiding the walls A1, A2, A3 by taking into account the walls A1, A2, A3 of the maze. It is assumed that something will be done. Further, as the game content, for example, a ball is displayed as the image object 26, and the thrown ball hits the walls A1, A2, A3 and bounces back.

  Further, by detecting the line of sight with the line-of-sight detection sensor 44 and displaying the image object 26 in consideration of the user's line of sight, fitting of the image object 26 (determination of the display position, display size, and orientation of the image object 26) is performed. ) Can be performed with higher accuracy.

  Further, by detecting the distance from the user to the real object A by the distance sensor 41 and displaying the image object 26 in consideration of this distance, the fitting of the image object 26 can be performed with higher accuracy.

  In the present embodiment, the coordinate position relative to the real object A is set as the coordinate position of the image object 26 in the real space, but the coordinate position of the image object 26 in the real space is set. A configuration in which a relative coordinate position with respect to the user is set, or a configuration in which an absolute position is set as the coordinate position of the image object 26 in the real space may be used.

  In the present embodiment, the coordinate position relative to the user is detected as the coordinate position of the real object A in the real space. However, the coordinate position of the real object A in the real space is absolute. It may be configured to detect the position.

  Further, in the present embodiment, the user's line of sight is detected, and the user's line of sight is detected based on the user's line of sight and the orientation of the user's head. A configuration in which the user's field of view is detected based on only the orientation of the user's head may be used.

  Furthermore, in the present embodiment, the object recognition marker MK may be an AR marker, a two-dimensional code (such as a QR code), or a barcode.

  In the present embodiment, the real scene is captured by the external camera 17 and the coordinate position and shape of the real object A are detected using image recognition on the captured image. The map data storing the coordinate positions and shapes of all the real objects A may be stored in advance, and the coordinate positions and shapes of the real objects A may be detected based on the map data.

  Further, in the present embodiment, in order to improve the accuracy of the fitting of the image object 26, the interocular distance, the right / left parallax angle, the convergence angle, the dominant eye, the parallax with the image, the relative speed of the user with respect to the actual object, and the like are detected. A configuration in which the image object 26 is fitted in consideration of these may be used.

  Furthermore, it is preferable to estimate the light source environment of the real scene 23 and adjust the color and brightness of the image object 26 according to the light source environment. The color and brightness of the image object 26 are preferably determined in consideration of the distance from the user to the real object A and changes in the user's visual environment (for example, dark adaptation and light adaptation).

  In the present embodiment, the distance from the user to the image object 26 in the coordinate system of the real space may be calculated and the focal length of the image object 26 may be changed.

  Furthermore, in this embodiment, the user operation in the content may be configured to be performed by a controller (not shown), or a part of the user's body (finger, hand, etc.) is recognized and the part of the body is recognized. It may be configured to accept behavior as a user operation. In addition, for example, the user's hand, finger, arm, or the like is recognized, and the display control unit 74 displays the image object 26 such as a bat, a racket, a playing card, or a pen as if it is held in the user's hand. Let The user may operate the image object 26 by moving a hand, finger, arm, or the like, and accept the operation as a user operation. Even in such a case, when the image object 26 is located behind the user's hand, finger, arm, or the like, which is the real object A when viewed from the user, the overlapping portion of the image object 26 with the real object A disappears. Then, the image object 26 is changed. Moreover, the structure which recognizes the real object A (For example, a bat, a racket, a playing card, a pen) which a user can handle, and receives the behavior of the said real object A as user operation may be sufficient. As a result, the structure of detecting a user's head movement (for example, direction) and a user's eyes | visual_axis and accepting this as user operation may be sufficient.

  Furthermore, the image object 26 to be displayed is not limited to the above-described one, and for example, an image object 26 such as an arrow or a mark may be displayed. Alternatively, a configuration may be used in which an object in the real space or a part of the human body (such as a fingertip or an arm) is captured and captured and displayed as the image object 26.

  In the present embodiment, the head-mounted display device 1 has a controller (not shown), and the controller includes an image analysis unit 72, an object detection unit 73, a display control unit 74, a display state control unit 75, and audio processing. The structure which mounts a part or all of the part 76 may be sufficient. In such a case, the controller has a configuration in which the controller is wiredly connected to the control unit 34 or the display unit 30 (the display unit 30, the external camera 17, the sensor unit 31, the right earphone 18, and the left earphone 19) by a cord or the like. Alternatively, it may be configured to be wirelessly connected to the control unit 34 or the display unit 30 by a wireless LAN, infrared communication, Bluetooth (registered trademark), or the like. Further, the function of the controller may be realized by a personal computer (PC), a personal digital assistant (PDA, mobile phone, watch-type mobile terminal, smart phone) or the like.

  Furthermore, in the present embodiment, the binocular head-mounted display device 1 (optically transmissive head-mounted display device) is exemplified as the display device, but the present invention is, for example, a monocular type or the like. The present invention can also be applied to other types of head mounted display devices. Moreover, in the said embodiment, although the spectacles type head mounted display apparatus 1 which covers the front of eyes is illustrated, it is not restricted to this, The type (front of eyes) which does not completely cover the front of eyes The present invention can also be applied to a head-mounted display device of a type that partially covers). Furthermore, the present invention is not limited to the head-mounted display device, but can be applied to other display devices such as a head-up display.

  Furthermore, in the present embodiment, the image light output unit includes a backlight control unit (right backlight control unit 51 and left backlight control unit 61), and a backlight (right backlight 52 and left backlight 62). The LCD control unit (the right LCD control unit 53 and the left LCD control unit 63) and the LCD (the right LCD 54 and the left LCD 64) are configured, but this mode is merely an example. The image light output unit may include a component for realizing another method together with these components or instead of these components.

  For example, the image light output unit may include an organic EL (Organic Electro-Luminescence) display and an organic EL control unit. Further, for example, the image light output unit may be configured to include LCOS (Liquid crystal on silicon, LCoS is a registered trademark), a digital micromirror device, or the like instead of the LCD.

  In the present embodiment, the present invention is applied to the so-called virtual image projection type head-mounted display device 1 (head-mounted display). For example, a laser retinal projection type (so-called retinal scanning type) head-mounted The present invention can also be applied to a type display device.

  Further, in the present embodiment, the image light output unit includes the projection optical system (the right projection optical system 55 and the left projection optical system 65) and is configured to project the image light emitted from the LCD. Instead of the projection optical system, the light output unit may include a scanning optical system configured by, for example, a MEMS mirror, and a signal light modulation unit (signal light forming unit) that emits signal light to the scanning optical system. good. In this case, the signal light formed and emitted by the signal light modulation unit is incident on a scanning optical system (MEMS mirror) that is a scanning unit. The scanning optical system emits signal light as scanning light toward a light guide having a half mirror layer, and the scanning light is scanned on the surface of the half mirror layer, thereby forming a virtual image by image light. The user can recognize the image by grasping the virtual image with the eyes.

  Furthermore, in the present embodiment, the present invention is applied to the head-mounted display device 1, but a display system in which the head-mounted display device 1 is connected to an external device or an external server. The present invention may be applied. In such a case, a part or all of the image analysis unit 72, the object detection unit 73, the display control unit 74, the display state control unit 75, and the sound processing unit 76 are mounted on the external device or the external server side.

  1: head-mounted display device, 26: image object, 26a: overlapping portion, 30: display unit, 41: distance sensor, 44: gaze detection sensor, 73: object detection unit, 74: display control unit, A: real thing body

Claims (10)

A transmissive display device that allows a user to visually recognize an image as a virtual image,
An object detection unit for detecting the coordinate position and shape of a real object in real space;
A display control unit for displaying an image object in which a coordinate position in the real space is set so as to be visually recognized according to a coordinate system in the real space;
A display unit for displaying the image object,
The display control unit controls the image object according to a coordinate position and a shape of the real object.   The display device according to claim 1, wherein the display control unit controls the behavior of the image object according to the coordinate position and shape of the real object.   The display control unit, when the image object is positioned behind the real object as viewed from the user, based on the detected coordinate position of the real object and the set coordinate position of the image object The display device according to claim 2, wherein the image object is changed so that a portion where the image object overlaps the real object disappears.   The display device according to claim 2, wherein the display control unit causes the image object to behave so as to avoid the real object in a coordinate system of a real space.   The display control unit causes the image object to behave as if it collided with the real object when the moved image object reaches the real object in a coordinate system of a real space. The display device according to claim 2. A line-of-sight detection unit for detecting the line of sight of the user,
The display device according to claim 1, wherein the display control unit displays the image object based on the detected line of sight. A distance detection unit for detecting a distance from the user to the real object;
The display device according to claim 1, wherein the display control unit displays the image object based on the detected distance. The display unit
An image light output unit for outputting image light of the image;
A light guide that guides the output image light to the user's eyes,
The display device according to claim 1, wherein the light guide unit transmits external light and causes the external light to enter the user's eyes together with the image light. A transmissive display system that allows a user to visually recognize a display image as a virtual image,
An object detection unit for detecting the coordinate position and shape of a real object in real space;
A display control unit for displaying an image object in which a coordinate position in the real space is set so as to be visually recognized according to a coordinate system in the real space;
A display unit for displaying the display image,
The display system, wherein the display control unit controls the image object according to a coordinate position and a shape of the real object. A control method for a transmissive display device that allows a user to visually recognize a display image as a virtual image,
An object detection step for detecting the coordinate position and shape of a real object in real space;
A display control step for displaying an image object in which a coordinate position in the real space is set so as to be visually recognized according to a coordinate system in the real space;
Performing a display step of displaying the image object;
In the display control step, the image object is controlled in accordance with the coordinate position and shape of the real object.

Images