JP2019008204A - Head-mounted display device - Google Patents

Abstract

To provide a head-mounted display device that enables a user to easily focus on virtual display even when the user sees the actual scene distant from the virtual display and subsequently moves their eyes to the virtual display in front of the user, and thereby can reduce a burden on the user.SOLUTION: A head-mounted display device 10 comprises: an image creating unit 40 that creates a first image for displaying a first virtual image and a second image as an eye-leading image; an image display unit 50 that displays the first and second images and outputs display light 3 of the first and second images; a projection see-through optical unit 70 that is arranged in front of the eyes of a user U, and includes a first portion 71 projecting the display light of the first and second images on the eyes of the user U and a second portion 72 allowing the user U to see through the actual scene; and a control unit 150 that controls the image creating unit 40.SELECTED DRAWING: Figure 1

Description

  The present invention relates to, for example, an optical see-through type head-mounted display (HMD) device that includes a display that allows a user to visually recognize a virtual image superimposed on a real scene.

  A head-mounted display (HMD) device, for example, uses an optically transmissive display to expand the range of information provided by displaying various information (virtual images) superimposed on the field of view visible to the naked eye. It is a display device capable of.

  An optically transmissive HMD device mounted on a user's head is described in Patent Document 1, for example. Patent Document 1 discusses a technique for superimposing a virtual image not only when a user is looking at an outside scene in front but also when looking at a book at hand.

JP2015-69362A

  For example, in an HMD device that displays a virtual image in accordance with the left and right eyes, when the user looks far away from the position of the virtual image and then tries to see the virtual image in front of it, the convergence angle of the left and right eyes Correspondingly, the present inventors have found that the distance from each eye to the virtual image is different, the viewpoint adjustment is delayed and the viewpoint does not match, and the virtual image may appear double or blurred. Recognized. In particular, in the HMD device, since a display for displaying a virtual image is arranged close to the user's eye, the virtual image display surface is set at a position relatively close to the user's eye, and the position close to the distance. When the viewpoint is moved (focused) to a virtual image on the surface of the virtual image in FIG. 3, the burden on the user's eyes increases, and the size of the displayed virtual image itself is reduced because the HMD device is small. This also tends to increase the burden on the user's eyes.

  One object of the present invention is to provide a head-mounted display that makes it easy to match the viewpoint and reduces the burden on the user even when the user moves the viewpoint (line of sight) to the virtual image in front of the user after looking farther than the virtual image. Is to provide a device.

  Other objects of the present invention will become apparent to those skilled in the art by referring to the aspects and best embodiments exemplified below and the accompanying drawings.

In the following, in order to easily understand the outline of the present invention, embodiments according to the present invention will be exemplified.
In the first aspect, a head mounted display (HMD) device comprises:
The first image for displaying the first virtual image and the second image as the viewpoint guidance image for displaying the second virtual image for assisting the user to move the viewpoint to the first virtual image And an image generation unit for generating
An image display unit that displays the generated first and second images and outputs display light of the first and second images;
A projection that is disposed in front of the user's eyes and includes a first portion that projects display light of the first and second images onto the user's eyes and a second portion that allows the user to see through the actual scene. Fluoroscopic optics,
A control unit for controlling the image generation unit;
Is provided.

  According to the first aspect, the user focuses on the second virtual image (virtual image of the viewpoint guidance image) that assists the viewpoint movement (line-of-sight movement), so that the first virtual image of the original display image is displayed. It is possible to quickly focus on the virtual image (in other words, the focus position of each eye can be accurately matched with the first virtual image position). Therefore, even when the user looks farther than the first virtual image and then moves the viewpoint (line of sight) to the first virtual image in front, it becomes easier to match the viewpoint, and quick and accurate focusing becomes possible. , The burden on the user can be reduced.

In a second aspect dependent on the first aspect, the head mounted display device comprises:
It may further include an operation unit that inputs a control signal to the control unit and that can be operated by the user.
The control unit may cause the image generation unit to generate the second image when a first input is made by the user operating the operation unit.

  According to the second aspect, when the user is in a situation where it is difficult to see the virtual image due to, for example, fatigue or the like, the second virtual image that assists the viewpoint movement by operating the operation unit (virtual image of the viewpoint guidance image) Can be displayed on the virtual image display surface. Therefore, convenience for the user is improved.

In a third aspect subordinate to the second aspect, the head mounted display device comprises:
The camera may further include a viewpoint detection unit that detects the viewpoint position of the user,
The control unit is configured such that the viewpoint of the user is a second distance shorter than the first distance from the actual scene where the distance from the user's eyes is the first distance by the viewpoint detection unit. When the movement to the first virtual image is detected, the image generation unit may generate the second image.

  As a result, the viewpoint detector detects that the user has moved the viewpoint (line of sight) to the first virtual image in front after viewing the real scene farther than the first virtual image, and assists the viewpoint movement. The second virtual image to be displayed (virtual image of the viewpoint guidance image) can be automatically displayed on the virtual image display surface. Therefore, the convenience for the user is improved, and the trouble of operating the operation unit of the user can be saved.

In a fourth aspect dependent on any one of the first to third,
The second virtual image is displayed in the vicinity of the first virtual image on the virtual image display surface of the first virtual image, and
The second virtual image may be a virtual image of a viewpoint guiding object as the viewpoint guiding image in which at least one of color, luminance, and position on the virtual image display surface changes with time.

  According to the fourth aspect, the viewpoint (line of sight) of the user using a virtual image of a still image that is easy to visually recognize, whose color and brightness change over time, or a virtual image of a moving image of a moving object. Therefore, the user can quickly shift his / her own eye to a state in which the first virtual image is in focus.

In a fifth aspect subordinate to the fourth aspect,
In the case where the position of the viewpoint guiding object changes on the virtual image display surface,
The front direction of the user is the first direction, the left-right direction that is orthogonal to the first direction and is along the line segment connecting the left and right eyes of the user is the second direction, When the vertical direction along the vertical line that is orthogonal to each of the first and second directions is the third direction,
The control unit is symmetrical with respect to the second virtual image so that the position in the second direction and the position in the third direction are both changed on the virtual image display surface. You may move along a part of circular arc.

  According to the fifth aspect, the ciliary muscle that adjusts the thickness of the crystalline lens of the human eye by moving the viewpoint both in the horizontal direction and in the vertical direction efficiently in a balanced manner in each direction. It is possible to guide the viewpoint quickly while relaxing.

  Those skilled in the art will readily understand that the illustrated embodiments according to the present invention can be further modified without departing from the spirit of the present invention.

FIG. 1A is a diagram illustrating an external configuration of an example of an eyeglass-type head mounted display (HMD) device according to the present invention, and FIG. 1B is a configuration example of a main part of the HMD device of FIG. FIG. FIG. 2 is a diagram for explaining the distance to the virtual image display surface, the convergence angle, and the like when the left and right eyes see a virtual image on the virtual image display surface. FIG. 3A is a diagram illustrating a configuration example for generating and outputting an image related to original display and a viewpoint guidance image, and FIG. 3B is an image related to original display on a virtual image display surface. FIG. 3C is a diagram showing an example of the display mode of each of the virtual image and the virtual image of the viewpoint-derived image, and FIG. 3C shows the display of the virtual image of the original display image and the virtual image of the viewpoint-guided image on the virtual image display surface. It is a figure which shows the other example of an aspect. FIG. 4 is a diagram illustrating an example of a system configuration (block configuration) of the eyeglass-type HMD device in FIG. 5A to 5C are superimposed on the actual scene when the glasses-type HMD device of FIG. 1A is used for driving assistance of a driver (user) driving the vehicle. It is a figure which shows the example of a virtual image. FIG. 6 is a flowchart illustrating an example of a procedure for displaying the viewpoint guidance image on the display device. FIG. 7 is a flowchart illustrating another example of the procedure for displaying the viewpoint guidance image on the display device.

  The best mode described below is used to easily understand the present invention. Accordingly, those skilled in the art should note that the present invention is not unduly limited by the embodiments described below.

  FIG. 1A is a diagram illustrating an external configuration of an example of an eyeglass-type head mounted display (HMD) device according to the present invention. As shown in FIG. 1A, an eyeglass-type HMD device (hereinafter simply referred to as an HMD device) 10 has an appearance similar to eyeglasses that are used by being worn on the head of a user U. Yes.

  The HMD device 10 includes a display that allows the user U to visually recognize a virtual image superimposed on a real scene. In this specification, the concept of a head mounted display (HMD) device is to be interpreted in a broad sense. For example, a main body that performs image processing or the like is fixed in a vehicle, and a display that displays a virtual image is placed near the face or eyes. It may be configured that both provide information exchange using wired communication or wireless communication. In addition, even if the display structure for displaying a virtual image is arranged in the vicinity of the eyes, it can be referred to as an HMD even if it is of a type that is worn not on the head but on the neck, shoulders, trunk, etc. The HMD device 10 may be a kind of display device capable of displaying augmented reality (AR).

  FIG. 1A is a view of the user's head (face) viewed from the right direction (X direction). In FIG. 1A, the right eye 1R of the user U (where R is “Right” is shown. “Left” is written as L. The projection fluoroscopic optical unit 70R arranged in front of the eyes (for example, arranged so as to cover the front of the eyes) and the projection fluoroscopy Located on the upper side (Z direction side) of the optical unit 70R, the frame unit 75R that supports the projection fluoroscopic optical unit 70R, and the operation unit 20 that is provided on a part of the frame unit 75R and that can be operated by the user U with the right hand. And having.

  The frame portion 75R is formed by connecting a front portion located on the upper side of the nose of the user U and a temple portion extending from the right end of the front portion toward the ear 5R of the user U. Composed. The above configuration is the configuration of the right half of the HMD device 10, and although not shown, the configuration of the left half is the same.

  FIG. 1B is a diagram illustrating a configuration example of a main part of the HMD device in FIG. In FIG. 1A, an outline of the right half display unit is shown. Although not shown, the left half has the same configuration. The system configuration including both the right side and the left side will be described later with reference to FIG. In the following description, it is assumed that the HMD device 10 is used by being mounted on the head of a vehicle driver (user U). Note that the application of the HMD device 10 is not limited to this (this will be described later).

  The HMD device 10 includes a first image for displaying the first virtual image and a viewpoint guidance image for displaying the second virtual image for assisting the user U to move the viewpoint to the first virtual image. The image generation unit 40 (40R, 40L) for generating the second image and the generated first and second images and the display light 3 (3R, 2R) for the first and second images are displayed. 3L), and an image display unit 50 (50R, 50L).

  Further, the HMD device 10 is arranged in front of the user U's eyes, and the half of the first and second images (in other words, an image to be originally displayed and a viewpoint guidance image) are displayed on the user U's eyes. A projection fluoroscopic optical unit 70 (70R, 70L) including a first portion 71 that projects display light 4 (4R, 4L) generated by being reflected by the mirror 65, and a second portion 72 that allows the user to see through the actual scene. , And light guide plates 60 (60R, 60L) configured to include the light guide plates 64a and 64b and the lens 62.

  Here, the half mirror 65 (indicated by hatching in FIG. 1B) is displayed in the X-axis direction (left and right as viewed from the user U) via the light guide 60. The light 3 (3R, 3L) is reflected and projected to the user's U eye 1 (1R, 1L), and the light from the actual scene in the Y direction (front as viewed from the user U) is transmitted and transmitted to the user. Can be communicated to U's eye 1 (1R, 1L).

  Therefore, the user U can see through the actual scene not only through the first portion 71 but also through the second portion 72 of the projection see-through optical unit 70 (70R, 70L). Further, the first portion 71 and the second portion 72 may overlap. In other words, for example, the common part may function as the first part 71 and also function as the second part 72.

  Further, in this specification, an optical member (a portion corresponding to a normal eyeglass lens) arranged in front of the user U's eye is referred to as a projection fluoroscopic optical unit 70 (70R, 70L) for convenience. The optical system including the projection see-through optical unit 70 (70R, 70L) and the rear half mirror 73R is an optical system arranged in the vicinity of the user's U eye 1 (1R, 1L). 73 (73R, 73L). However, this name is for convenience, and the eyepiece optical unit 73 (73R, 73L) can also be called a projection fluoroscopic optical unit in a broad sense.

  The HMD device 10 is further controlled by a control unit 150 (as shown in FIG. 4, the control unit 150 is provided in the control device 30) that controls the image generation unit 40 (40R, 40L), and the control unit 150. An image pickup device such as a CCD that inputs a signal and picks up an image of the operation unit 20 (including the viewpoint guidance operation unit 22) that can be operated by the user U and the eyes (right eye 1R, left eye 1L) of the user U. A configured pupil imaging camera (inner camera) 80 (80R, 80L) and a real scene imaging camera (outer camera) 90 (90R, 90L).

  Here, the above-mentioned “first virtual image” is a virtual image of the AR content that should be displayed originally, and in the example of FIG. 1B, a predetermined value is obtained from the right eye 1R (and the left eye 1L) of the user U. Of the vehicle speed display of “60 km / h” as the first virtual image 200 (I) on the virtual image display surface 190 separated by a distance of (not limited to a specific distance and may be assumed to vary as appropriate) A virtual image is shown. Note that (I) of reference numeral 200 (I) indicates a virtual image, and this is the same for other reference numerals.

  On the virtual image display surface 190, a virtual image of a “circular figure with movement” is provided as a second virtual image 300 (I) that assists the user U in moving the viewpoint to the first virtual image 200 (I). It is shown. The virtual image display surface 190 is a virtual surface that is assumed to be at a predetermined distance from the eyes of the user U when the real space around the vehicle is displayed in an orthogonal coordinate system. Any point can be treated as having the same distance from the eyes of the user U (equal distance).

  The image display unit 50 (50R, 50L) includes a display device (for example, a liquid crystal display device with a backlight), and displays an image on a display surface (described later) of the display device. The vehicle speed display “60 km / h” as the “first image” and the “circular figure with movement” (that is, the moving image of the circular figure) as the “second image” These are displayed on the display surface of the display device.

  Moreover, the light guide part 60 (60R, 60L) comprised including the light guide plates 64a and 64b and the lens 62 is, for example, the user U's of the frame part 75 (75R, 75L) in FIG. It is provided in a front portion (or a portion including a connecting portion with a temple portion extending toward the user 5's ear 5) located above the nose. Further, the light guide plates 64a and 64b may be, for example, cylindrical light guides or may be hologram light guide plates.

  Note that the control unit 150 may be divided into two parts, for example, one provided in the right temple part of the frame 75 in FIG. 1A and the other provided in the left temple part.

  Similarly, the image generation unit 40 (40R, 40L) and the image display unit 50 (50R, 50L) are arranged between the left and right temple portions of the frame portion 75 or the left and right temple portions and the front portion of the frame portion 75. You may provide in the vicinity of a connection part.

  The imaging signal (imaging data) of the pupil imaging camera (inner camera) 80 (80R, 80L) is sent to the control unit 150 to detect the movement of the viewpoint (line of sight) of the user U. In addition, an imaging signal (imaging data) of a real scene imaging camera (outside camera) 90R that captures a real scene such as in front of the vehicle is controlled to appropriately display a virtual image useful for driving the vehicle according to the running state of the vehicle. Sent to the unit 150.

  Reference is now made to FIG. FIG. 2 is a diagram for explaining the distance to the virtual image display surface, the convergence angle, and the like when the left and right eyes see a virtual image on the virtual image display surface. In FIG. 2, the distance between the left eye 1L and the right eye 1R of the user U is d1, and the distance between each eye (1L, 1R) and the projection fluoroscopic optical unit (70L, 70R) is f1. Further, when each eye 1L, 1R is looking at an infinite point, each line of sight of each eye 1L, 1R is in the direction of YL, YR, which are parallel segments.

  Here, with reference to the positions of the eyes 1L and 1R, a virtual image Q (I) (for example, “60 km / h in FIG. 1B) is displayed on the virtual image display surface 190 that is forward (Y-axis direction) by a distance L1. The vehicle speed display 200 (I)) is displayed, and the user U drives the vehicle while gazing at a distant scene, and sometimes moves the viewpoint to the virtual image Q (I) as necessary. Assume that necessary information is obtained.

  When the user U looks far away from the position of the virtual image Q (I) and then tries to move the viewpoint to the virtual image Q (I) in front of it, the convergence angles of the left eye 1L and the right eye 1R Since the distance from each eye to the virtual image Q (I) on the virtual image display surface 190 varies according to β, the focus adjustment of the left eye 1L and the right eye 1R is delayed, the viewpoint does not match, and the virtual image Q There may be a problem that (I) looks double or blurs. In particular, this inconvenience is likely to occur when the user U is tired from driving for a long time.

  In FIG. 2, θL is the gaze direction angle of the left eye, θR is the gaze direction angle of the right eye, and the convergence angle β is an angle obtained by adding θL and θR. In FIG. 2, a point XL is a point where a line segment connecting the left eye 1 </ b> L and the virtual image Q (I) intersects the left-eye (left side) projection fluoroscopic optical unit 70 </ b> L, and the point XR is the right A line segment connecting the eye 1R and the virtual image Q (I) is a point that intersects the right-eye (right side) projection fluoroscopic optical unit 70L.

  In order to reduce the burden on the user due to the above inconvenience, the HMD device 10 of the present embodiment assists the user U's viewpoint movement (line-of-sight movement) as shown in FIG. The virtual image (virtual image of the viewpoint guidance image) 300 (I) is displayed. Hereinafter, a configuration for displaying a virtual image of the viewpoint guidance image and an example of the viewpoint guidance image will be specifically described with reference to FIG.

  FIG. 3A is a diagram illustrating a configuration example for generating and outputting an image related to original display and a viewpoint guidance image, and FIG. 3B is an image related to original display on a virtual image display surface. FIG. 3C is a diagram showing an example of the display mode of each of the virtual image and the virtual image of the viewpoint-derived image, and FIG. 3C shows the display of the virtual image of the original display image and the virtual image of the viewpoint-guided image on the virtual image display surface. It is a figure which shows the other example of an aspect. In each drawing, the same reference numerals are given to the portions common to the above-described drawings (this is the same in the subsequent drawings).

  As shown in FIG. 3A, the storage unit (comprising ROM, RAM, EEPROM, etc.) 110 has virtual image display data (for example, vehicle speed indicating the current vehicle speed) for displaying the original virtual image. Data and display data such as arrows and warning marks for navigation) are accumulated. Note that the virtual image display data is display image data to be visually recognized by the user U, and may be data stored in the storage unit 110 in advance, or may be an external cloud using wireless communication. It may be data downloaded as appropriate from a server or the like.

  The image processing unit 151 appropriately reads out the data, performs image processing such as changing the size of the display image as necessary, and supplies the image-processed data to the image output unit S.

  The image output unit S is provided with the image generation unit 40 and the image display unit 50 shown in FIG. 1B (notation of R and L is omitted in the reference numerals). The image generation unit 40 includes a viewpoint guidance image generation unit 43 that generates a viewpoint guidance image. The image display unit 50 includes a liquid crystal display device (LCD) 55 with a backlight (BL) 53 and an LCD drive unit 51 that drives the display devices (53, 55) as display devices. Note that an organic EL display device, an inorganic EL display device, or the like may be used instead of the liquid crystal display device.

  Images (including original virtual image display images and viewpoint guidance images) are displayed on the display surface 57 of the liquid crystal display device (LCD) 55, and display light (reference numeral 3R in FIG. 1B) indicating those images is displayed. The display light output from the display surface 57 is guided to the eyes of the user U via the light guide unit 60 and the projection see-through optical unit 70 (with the configuration shown in FIG. 1B as an example). It is burned. As a result, as shown in FIG. 2, when the user views with the left eye 1L and the right eye 1R, the virtual image Q (I) (virtual image that should originally be displayed) is displayed on the virtual image display surface 190. The first virtual image, the virtual image of the viewpoint guidance image) is visually recognized.

  In addition, said structure is an example and is not limited to this. Various aspects can be considered for displaying the viewpoint guidance image.

  The virtual image (second virtual image) of the viewpoint guidance image guides the viewpoint to the virtual image (first virtual image) of the original display image without assisting the user's eyes (helps focusing). Therefore, as the viewpoint guidance image (second image), for example, a simple figure that is easy to see for the user U can be considered. In addition, it may be a simple icon, a color tone with a higher emphasis compared to the original display image (first image), a moving image displaying images continuously, etc. The icon etc. which the user U can move and operate to arbitrary positions using the operation part 20 may be sufficient.

  In addition, the viewpoint guidance image (second image) may be embedded in advance on the original display image (first image), and the user is further away from the virtual image of the first image. Alternatively, the viewpoint guidance image may be displayed only when the viewpoint detection unit or the like detects the timing at which the line of sight is moved to the first virtual image in front of the camera. Alternatively, the user U can display the virtual image of the viewpoint guidance image appropriately by operating the operation unit 20 (viewpoint guidance operation unit 22) when the user U feels that the viewpoint movement is a burden, The user's U brain wave, facial expression, etc. may be analyzed to detect that the user U feels a burden, and when the detection is made, a virtual image of the viewpoint guidance image may be displayed.

  In addition, when the user U is always looking far away, as in the case of using in a vehicle, and a situation in which the virtual image display in front is viewed only when necessary, the original display image is displayed. When displaying a virtual image (first virtual image), a virtual image (second virtual image) of the viewpoint guidance image may always be displayed.

  Next, refer to FIGS. 3B and 3C. In FIG. 1 (B), the vehicle speed display of “60 km / h” was shown as the first virtual image Q (I), but in FIGS. 3 (B) and 3 (C), as another example, for navigation The virtual image 203 (I) of the arrow with the tip bent to the right is shown. This virtual image 203 (I) is a virtual image of a navigation display indicating that the road ahead is curving to the right (or turning to the right).

  For example, when the user U is tired, the user looks at a distance and then moves the viewpoint (line of sight) to the nearby virtual image 203 (I). When the user U feels inconvenience, the user U operates the viewpoint guidance operation unit 22 of the operation unit 20 illustrated in FIG. 1B to display the virtual image 203 (I) of the viewpoint guidance image as the second virtual image. Can be displayed. In addition, when the user U has previously set the viewpoint guidance image automatic display mode in the HMD device 10 by operating the operation unit 20, the control unit 150 performs the above-described viewpoint (line of sight). ) Can be detected, and the virtual image 300 (I) (or 303 (I) or the like) of the viewpoint guidance image as the second virtual image can be automatically displayed.

  Here, the virtual image 300 (I) (or 303 (I) or the like) of the viewpoint guidance image as the second virtual image is a virtual image display in which the first virtual image 203 (I) in FIG. 3B is displayed. On the surface 190 (in other words, at the same distance (including a distance that is substantially the same) as the distance to the first virtual image with respect to the position of the user's eye), the first virtual image 203 (I) The second virtual images 300 (I) and 303 (I) are displayed at least one of color (may be a pattern or shape), luminance, and position on the virtual image display surface 190 over time. It can be a virtual image of a viewpoint guiding object (circular figure in the examples of FIGS. 3B and 3C) as a viewpoint guiding image that changes with time.

  In the example of FIG. 3B, the virtual image 300 (I) of the viewpoint guidance object (circular figure) is blinking, in other words, the luminance changes with time, and thus the user U's attention. It becomes the mode which is easy to pull. Therefore, the user U focuses on the second virtual image (virtual image of the viewpoint guidance image (viewpoint guidance object)) 300 (I) that assists the viewpoint movement (line-of-sight movement), so that it is a virtual image of the original display image. A certain first virtual image 203 (I) can be quickly focused.

  Therefore, even when the user U moves the viewpoint (line of sight) to the first virtual image 203 (I) in front of the user after looking farther than the first virtual image 203 (I), the viewpoint can be easily adjusted. The burden on the user U can be reduced. In other words, the user U can efficiently guide the user's viewpoint (line of sight) in a short time using a virtual image of a still image that is easy to view and whose color and brightness change over time. The eyes 1L and 1R can be quickly shifted to a state in which the first virtual image 203 (I) is in focus.

  Next, reference is made to FIG. In the example of FIG. 3C, a virtual image 303 (I) of an object (in other words, a moving image) that accompanies the movement of position is used as a viewpoint guiding object that is a viewpoint guiding image.

  In this case, the viewpoint of the user U is efficiently guided in a short time along the movement of the object on the virtual image display surface 190. Therefore, the user U can quickly shift his / her eyes 1L and 1R to a state in which the first virtual image 203 (I) is in focus.

  Here, attention is focused on the movement of the virtual image 303 (I) of the viewpoint guiding object in the example of FIG. In FIG. 3C, a partial arc with a bidirectional arrow (which is a part of a circle, the circle includes an ellipse, and is interpreted broadly in the sense of a “curve”) is depicted. This partial arc indicates the movement of the viewpoint guiding object. The viewpoint guiding object moves, for example, like a pendulum of a clock, such as moving from left to right (waving) and then returning to the center.

  In other words, the front direction of the user U is the first direction (Y direction), the right and left directions are perpendicular to the first direction and are along the line segment connecting the left and right eyes 1L and 1R of the user U. When the direction is the second direction (X direction) and the vertical direction along the vertical line, which is the direction orthogonal to each of the first and second directions, is the third direction (Z direction), the control unit 150, the position of the second virtual image 303 (I) on the virtual image display surface 190 in the second direction (X direction which is the left-right direction) and in the third direction (Z direction which is the vertical direction) are both In order to change, it is moved along a part of the arc that is line-symmetric in the second direction (the above-mentioned partial arc).

  Thereby, the ciliary muscle that adjusts the thickness of each lens of the left eye 1L, 1R of the user U by moving the viewpoint in both the left-right direction and the up-down direction is balanced in each direction, It is possible to guide the viewpoint quickly while relaxing efficiently.

  In addition, as described above, when the user U is in a situation where it is difficult to see a virtual image due to, for example, fatigue or the like, the second virtual image that assists the viewpoint movement by operating the viewpoint guiding operation unit 22 of the operation unit 20 (see FIG. (A virtual image of a viewpoint guidance image or a viewpoint guidance object) can be displayed on the virtual image display surface 190, and thus the user U can receive a viewpoint guidance by the viewpoint guidance image only when necessary. Will improve.

  Further, when a mode for automatically displaying a virtual image of the viewpoint guidance image is set on the HMD device 10 (automatic display mode of the viewpoint guidance image), the viewpoint position of the user U provided in the HMD device 10 is set. By the viewpoint detection unit (reference numeral 154 in FIG. 4) to detect, the viewpoint of the user U is a second distance shorter than the first distance from the actual scene whose distance from the user U's eyes is the first distance. The controller 150 displays the second virtual images 300 (I) and 303 (I) on the image generator 40 when it is detected that the first virtual image 203 (I) is moved. (Second image) can be displayed on the display surface 57 (see FIG. 3A) of the liquid crystal display device (LCD) 55.

  In this case, the viewpoint detection unit indicates that the user U has moved the viewpoint (line of sight) to the first virtual image 203 (I) in the foreground after viewing the real scene farther than the first virtual image 203 (I). (Reference numeral 154 in FIG. 4) detects and automatically displays the second virtual images (virtual images of the viewpoint guidance images) 300 (I) and 303 (I) for assisting the viewpoint movement on the virtual image display surface 190. Therefore, the convenience of the user U is improved, and the trouble of operating the operation unit 20 of the user U can be saved.

  Reference is now made to FIG. FIG. 4 is a diagram illustrating an example of a system configuration of the eyeglass-type HMD device in FIG. The control unit 30 includes the operation unit 22 and the control unit 150 illustrated in FIG. 1B and the storage unit 110 illustrated in FIG.

  In addition, the control device 30 includes various sensors 120, a communication unit (including a wired communication unit 132, a wireless communication unit 134, and a voice communication unit 136) 130, and a GPS unit 140.

  The control unit 150 includes an image processing unit 151, a display control unit 152, a real scene analysis unit 153, a viewpoint detection unit 154, a state information acquisition unit 155 such as a vehicle, a position information acquisition unit 156, and an imaging control. A unit 157 and an audio processing unit 158.

  The display mechanism 500 includes a right image output unit SR (including an image generation unit 40R and an image display unit 50R), a left image output unit SL (including an image generation unit 40L and an image display unit 50L), and a right-eye display unit. (Right side) projection fluoroscopic optical unit 70R and left eye (left side) projection fluoroscopic optical unit 70L. The display mechanism 500 can individually generate an image corresponding to each of the left eye 1L and the right eye 1R of the user U and project (project) it on each eye. Further, the display mechanism 500 can generate a parallax image (necessary when forming a three-dimensional virtual image) as necessary.

  Further, as cameras whose imaging operation is controlled by the imaging control unit 157, a right-eye (right-side) pupil imaging camera (inner camera) 80R, a left-eye (left-side) pupil imaging camera (inner camera) 80L, A right-eye (right side) actual scene imaging camera (outer camera) 90R and a left-eye (left side) actual scene imaging camera (outer camera) 90L are provided.

  In addition, the pupil imaging cameras 80L and 80R in FIG. 4 and the viewpoint detection unit 154 included in the control unit 150, for example, select a mode for automatically displaying a viewpoint guidance image (viewpoint guidance image automatic display mode). Based on the detected information, the image processing unit 151 and the display control unit 152 in the control unit 150 are operated to display the viewpoint guidance image.

  Reference is now made to FIG. 5A to 5C are superimposed on the actual scene when the glasses-type HMD device of FIG. 1A is used for driving assistance of a driver (user) driving the vehicle. It is a figure which shows the example of a virtual image. As shown in each of FIGS. 5A to 5C, the vehicle 600 includes a handle 170, and here, the operation unit 20 (including the viewpoint guidance operation unit 22) is provided inside the handle 170. ing. In the example of FIG. 1A described above, the operation unit 20 is provided in the HMD device 10 itself, but here, the operation unit 20 is provided in a part of the handle 170 in the vehicle 600.

  Here, when the user U operates the operation unit 20 (viewpoint guidance operation unit 22), a control signal or the like generated by the operation is transmitted to the HMD device 10 by wireless communication. The HMD device 10 includes the wireless communication unit 134 as illustrated in FIG. 4, and thus can receive the transmitted control signal and the like, whereby the control unit 150 views the image generation units 40R and 40L. Display a guidance image.

  In the example of FIG. 5A, the display of “60 km / h, 2000 rpm” indicating the vehicle speed and the engine speed is displayed as the first virtual image 201 (I), and the lower side (and vicinity) of the display. In addition, a virtual image 303 (I) of the viewpoint guidance image (viewpoint guidance object) accompanied by the movement shown in FIG. 3C is displayed.

  In the example of FIG. 5B, the first virtual image 203 (I) and the second virtual image (virtual image of the viewpoint guidance image) 303 (I) shown in FIG. 3C are displayed. As the second virtual image 205 (I), “P” indicating that there is a parking area on the right front is displayed.

  In the example of FIG. 5 (C), a warning message “please refuel” is displayed as the first virtual image 207 (I). A blinking second virtual image (virtual image of the viewpoint guidance image) 300 (I) is displayed.

  Note that the displayed second virtual images 303 (I) and 300 (I) may not be automatically displayed when a predetermined time elapses, for example. In this case, an operation for the user U to turn off the display becomes unnecessary (however, the present invention is not limited to this).

  Reference is now made to FIG. FIG. 6 is a flowchart showing an example of a procedure for displaying the viewpoint guidance image (second image) on the display device (display surface 57 of the liquid crystal display device 55 shown in FIG. 3A).

  The control unit 150 determines whether or not the user U has performed an operation for displaying the viewpoint guidance image (operation of the viewpoint guidance operation unit 22 provided in the operation unit 20) (step S10). The generation unit 40 (40R, 40L) is controlled to display the viewpoint guidance image (second image) on the display surface 57 of the liquid crystal display device 55, which is a display device (step S20). As a result, the second virtual image (virtual image of the viewpoint guidance image) 300 (I) and the like described above are displayed on the virtual image display surface 190.

  In step S10, in the case of N, the control unit 150 determines whether or not the viewpoint guide image automatic display mode is set (step S30). In the case of N, the control unit 150 returns to step S10. In the case, it is determined whether or not the viewpoint has been moved from a distance to the virtual image display (step S40). In step S40, if Y, the process proceeds to step S20. If N, the process proceeds to step S50.

  Step S50 is an optional determination step. Here, for example, based on the biological information of the user U (including the pulse rate, heart rate, brain wave, etc.), the control unit 150 determines that the user U Estimate whether or not the burden of viewing the virtual image display is felt. In step S50, if Y, the process proceeds to step S20. If N, the process returns to step S10.

  In the case of the flowchart of FIG. 6, when the automatic display mode of the viewpoint guidance image is set, the viewpoint guidance image is displayed when Y is set in any of steps S40 and S50.

  Reference is now made to FIG. FIG. 7 is a flowchart illustrating another example of the procedure for displaying the viewpoint guidance image on the display device. The control unit 150 determines whether or not the user U has performed an operation for displaying the viewpoint guidance image (operation of the viewpoint guidance operation unit 22 included in the operation unit 20) (step S60). The generation unit 40 (40R, 40L) is controlled to display the viewpoint guidance image (second image) on the display surface 57 of the liquid crystal display device 55, which is a display device (step S100). As a result, the second virtual image (virtual image of the viewpoint guidance image) 300 (I) and the like described above are displayed on the virtual image display surface 190.

  In step S60, in the case of N, the control unit 150 determines whether or not the viewpoint guide image automatic display mode is set (step S70). In the case of N, the control unit 150 returns to step S60. In the case, it is determined whether or not the viewpoint has been moved from a distance to the virtual image display (step S80). In step S80, if N, the process returns to step S60, and if Y, the process proceeds to step S90.

  Step S90 is an optional determination step. Here, for example, based on the biological information of the user U (including the pulse rate, heart rate, brain wave, etc.), the control unit 150 determines that the user U Estimate whether or not the burden of viewing the virtual image display is felt. In step S90, if Y, the process proceeds to step S100, and if N, the process returns to step S60.

  In the case of the flowchart of FIG. 7, if the automatic display mode of the viewpoint guidance image is set, the viewpoint guidance image is displayed when both are Y in steps S80 and S90. The control procedures in FIGS. 6 and 7 are examples, and the conditions for displaying the viewpoint guidance image can be changed as appropriate.

  When the display device according to the embodiment described above is applied to a head mounted display (HMD) device as shown in FIGS. 1A and 1B, the user U is made to visually recognize a virtual image superimposed on a real scene. It is possible to realize a new and useful optical transmission type head mounted display (HMD) device that is capable of providing a viewpoint guiding function.

In addition, as a result of intensive studies, the inventors of the present application have obtained the following recognition regarding the effectiveness in displaying a viewpoint induced image.
(1) The viewpoint guidance image is displayed more easily in response to the operation of the user U than when it is automatically displayed.
(2) The viewpoint guiding image is easier to guide the viewpoint when it is a moving image than when it is a still image.
(3) The viewpoint guidance image is easier to guide the viewpoint if it is a filled image than the line image.
(4) As the virtual image of the viewpoint induced image is displayed at a position closer to the virtual image of the original display image, the viewpoint is more easily guided to the virtual image of the original display image.
According to the above, the highest viewpoint guidance effect can be obtained by the method in which the position of the filled virtual image of the viewpoint guidance image moves according to the user's operation in the vicinity of the virtual image of the original display image.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the above-mentioned embodiment, A various deformation | transformation and application are possible. For example, in the example of FIG. 2, the distance from the position of both eyes of the user U to the virtual image display surface 190 is described as L1, but this distance is not fixed and can be appropriately changed.

  For example, warning information during operation is displayed on a virtual image display surface at a distance L0 (L0 <L1), and the user's heart rate information, engine speed information, etc. (information indicating the state of the user or the vehicle) The nature and importance of the information to be displayed such that the store information and the route information (normal AR information superimposed on the landscape) are displayed at the distance L3 (L3> L1). The distance to the virtual image display surface can be appropriately changed according to the degree (priority).

  In this case, the user U has an advantage that various kinds of information can be grasped more appropriately, but the burden on both eyes of the user U increases. Therefore, application of the present invention is effective. When the present invention is applied, when the distance to the virtual image display surface is switched, the virtual image of the viewpoint guidance image is also displayed on the virtual image display surface after the switching. The virtual image of the viewpoint guidance image is preferably displayed in the vicinity of the various types of information (information that should be displayed originally).

  In the above embodiment, the application of the HMD device 10 according to the present invention has been described mainly assuming vehicle driving assistance, but is not limited thereto. For example, the HMD device 10 can be used for assisting the progress of a treasure hunt game or the like that is performed outdoors. For example, when the user U wearing the HMD device 10 arrives at a predetermined point, a hint suggesting a treasure place is displayed. The user U repeats the real scene space and the virtual image space. Since it will be seen and the burden on the eye becomes large, the application of the present invention is effective.

  The HMD device 10 of the present invention can also be used for a user U to enjoy reading while walking outdoors. In this case, the information that should be displayed is type information. In the case where the user U reads through the real scene through the projection fluoroscopic optical unit and appropriately reads the type as a virtual image, the user U repeatedly sees the real scene space and the virtual image space. Thus, since the burden on the eyes is increased, the application of the present invention is effective. Thus, the HMD device of the present invention can be used for various applications.

  Further, in the above-described embodiments (for example, in FIGS. 3B and 3C), the viewpoint induced image (or the virtual image of the viewpoint induced image) and the original display image (or the virtual image of the original display image) are: Although they are drawn as separate images, they are not necessarily separate, for example, the original display image (first image, original image 203 (I) in FIG. 3B) However, when the image is changed to a form suitable for viewpoint guidance that can be distinguished from the original display image, and the image after the change is used for the viewpoint guidance of the user U, the “original display image is The “image after changing in a manner different from the original display image” can no longer be regarded as the “viewpoint induced image”, but the original “original display image.” For example, the original display image is , Change to a color that cannot be displayed normally, or different from normal If or blinking blinking pattern, which are those aimed at the viewpoint induction effect, the image can be treated as "viewpoint induced image".

  In addition, for example, the configuration including the storage unit 110, the image processing unit 151, and the image output unit S shown in FIGS. 1B, 3A, and 4 is a wearable computer (wearable). The HMD device as a computer device can be realized by software by operating it with a software program. In this case, the software displays “a computer displays a first image for displaying the first virtual image and a second virtual image that assists the user to move the viewpoint to the first virtual image. An image generation unit that generates a second image as a viewpoint guidance image, and the generated first and second images, and display light of the first and second images. An image display unit for outputting; a first portion that is disposed in front of the user's eyes and that projects display light of the first and second images to the user's eyes; and a first part that allows the user to see through the actual scene. It can be referred to as a program having a function of displaying a second virtual image that is operated as a projection fluoroscopic optical unit including two parts and a control unit that controls the image generation unit. By using this software program, for example, it is possible to easily give a virtual image display function of a viewpoint guidance image to a general-purpose optical transmission type HMD device.

  An operation unit that can be operated by voice may be used as the operation unit operated by the user. For example, when the user speaks “viewpoint guidance image on”, the switch may be turned on to display a virtual image of the viewpoint guidance image.

  The present invention is not limited to the above-described exemplary embodiments, and those skilled in the art will be able to easily modify the above-described exemplary embodiments to the extent included in the claims. .

  1 (1R, 1L) ... user's eyes (viewpoint), 3 (3R, 3L) ... display light, 5 ... user's ear, 10 ... HMD device, 20 ... operation , 22 ... viewpoint guidance operation unit, 30 ... control device, 40 (40R, 40L) ... image generation unit, 43 ... viewpoint guidance image generation unit, 50 (50R, 50L) ... Image display unit 51 ... LCD drive unit 53 ... Backlight 55 ... Liquid crystal display device (LCD) 60 (60R, 60L) ... Light guide unit 62 ... Lens 64 (64a, 64b) ... light guide plate, 65 ... half mirror (projection optical system), 70 ... projection see-through optical part, 71, 72 ... first and second parts of the projection see-through optical part 73 (73R, 73L) ... eyepiece optical unit including a half mirror and a projection fluoroscopic optical unit, 75 (75R, 7 L): Frame part of HMD device (including front part and temple (vine) part), 80 (80R, 80L) ... Pupil imaging camera (inner camera), 90 ... Real scene imaging camera (outer camera) , 110... Storage unit, 112... Virtual image display image data, 114... Viewpoint guidance image data, 150... Control unit, 190... Virtual image display surface, 200 (I), 203 ( I) ... virtual image of information (AR content) to be displayed, 300 (I), 303 (I) ... virtual image of viewpoint guidance image, U ... user (user), β ... Angle of convergence, θL, θR: Gaze direction angle.

Claims (5)

The first image for displaying the first virtual image and the second image as the viewpoint guidance image for displaying the second virtual image for assisting the user to move the viewpoint to the first virtual image And an image generation unit for generating
An image display unit that displays the generated first and second images and outputs display light of the first and second images;
A projection that is disposed in front of the user's eyes and includes a first portion that projects display light of the first and second images onto the user's eyes and a second portion that allows the user to see through the actual scene. Fluoroscopic optics,
A control unit for controlling the image generation unit;
A head-mounted display device comprising: An operation unit that inputs a control signal to the control unit and that can be operated by the user is further provided.
The said control part makes the said image generation part produce | generate the said 2nd image, when there exists the 1st input by the said user operating the said operation part. The head mounted display device described. A viewpoint detection unit for detecting the viewpoint position of the user;
The control unit is configured such that the viewpoint of the user is a second distance shorter than the first distance from the actual scene where the distance from the user's eyes is the first distance by the viewpoint detection unit. 2. The head mounted display device according to claim 1, wherein when the movement to the first virtual image is detected, the image generation unit generates the second image. 3. The second virtual image is displayed in the vicinity of the first virtual image on the virtual image display surface of the first virtual image, and
The second virtual image is a virtual image of a viewpoint guiding object as the viewpoint guiding image in which at least one of color, luminance, and position on the virtual image display surface changes with time. 4. The head mounted display device according to any one of 3 above. In the case where the position of the viewpoint guiding object changes on the virtual image display surface,
The front direction of the user is the first direction, the left-right direction that is orthogonal to the first direction and is along the line segment connecting the left and right eyes of the user is the second direction, When the vertical direction along the vertical line that is orthogonal to each of the first and second directions is the third direction,
The control unit is symmetrical with respect to the second virtual image so that the position in the second direction and the position in the third direction are both changed on the virtual image display surface. The head mounted display device according to claim 4, wherein the head mounted display device is moved along a part of an arc.

Images