JP2017049468A - Display device, control method for display device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a user who uses a display device to adjust a displayed image as appropriate so that the user can see an outside scenery and the displayed image.SOLUTION: An HMD 1 includes a virtual image display device 100 including a first display device 100A and a second display device 100B for displaying images for the right eye and the left eye of a user while the user can see the outside scenery. The HMD 1 includes a control device 300 that controls the display mode of the image by the virtual image display device 100 in accordance with the viewing distance of the user.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a display device, a display device control method, and a program.

  Conventionally, a head-mounted display device that displays an image of an object in real space is known (for example, see Patent Document 1). Patent Document 1 discloses a configuration in which focus adjustment is performed so that a position where a user perceives an object from an image displayed by a display device matches a position where the user views the object in real space.

Special table 2013-537784 gazette

By the way, when the user who uses the display device visually recognizes the image displayed on the display device and the outside scene, the appearance of the display image changes depending on which direction or how much distance the user gazes at. However, in this type of display device, there has been no example of adjusting the appearance of the display image.
The present invention has been made in view of the above circumstances, and an object thereof is to allow a user who uses a display device to appropriately adjust a display image when the user can visually recognize an outside scene and a display image.

To achieve the above object, the present invention provides a display unit that displays an image corresponding to each of a user's right eye and left eye so that an outside scene can be visually recognized, and an image display mode by the display unit, And a control unit that controls according to the gaze distance of the user.
ADVANTAGE OF THE INVENTION According to this invention, when a user visually recognizes an outside scene and an image, the display mode of an image can be adapted to the distance which a user gazes at. For this reason, a display image can be adjusted appropriately, for example, a user's load can be reduced and the visibility of a display image can be improved.

In the display device according to the present invention, the control unit changes a display position of the image and / or a display size of the image according to the gaze distance of the user. And
According to the present invention, the display position and display size of an image can be adapted to the distance that the user gazes at.

Further, the present invention is characterized in that, in the display device, the control unit controls an image display mode by the display unit based on a convergence angle of both eyes corresponding to the gaze distance of the user. .
ADVANTAGE OF THE INVENTION According to this invention, the display mode suitable for the distance which a user gazes is realizable by controlling the display mode of an image corresponding to the convergence angle of a user's both eyes.

Further, according to the present invention, in the display device, the control unit calculates a convergence angle of both eyes of the user, and the user visually recognizes an image displayed on the display unit corresponding to the calculated convergence angle. It is characterized in that the position to be changed is changed.
According to the present invention, since the display device obtains the convergence angle of both eyes of the user and changes the position at which the user visually recognizes the display image, the image can be displayed at a position suitable for the gaze distance of the user. For this reason, the visibility of a display image can be improved and a user can visually recognize the display image suitable for the distance which a user gazes in an outside scene.

In the display device according to the aspect of the invention, the control unit may change a display position where the display unit displays the image from a preset standard position corresponding to a convergence angle of both eyes of the user. Shifting.
ADVANTAGE OF THE INVENTION According to this invention, an image can be displayed on the position suitable for a user's gaze distance.

In the display device according to the present invention, the display device further includes a state detection unit that detects a state of both eyes of the user, and the control unit is based on a detection result of the state detection unit. And a display mode of the image by the display unit is controlled according to the calculated gaze distance.
According to the present invention, since the display device detects the state of both eyes of the user and calculates the gaze distance, the display mode of the image can be appropriately adjusted without the user performing a complicated operation.

Further, according to the present invention, in the display device, the state detection unit includes a first imaging unit provided corresponding to each of the right eye and the left eye of the user, and the control unit includes the first imaging unit. Calculating a gaze distance of the user by obtaining at least one of a distance between pupils of both eyes of the user, a pupil diameter, or a gaze time in a gaze direction based on a captured image of the user. And
According to the present invention, the gaze distance can be calculated with high accuracy from the state of both eyes of the user.

In the display device according to the aspect of the invention, the state detection unit includes a second imaging unit that captures an image of the gaze direction of the user, and the control unit is based on a captured image of the second imaging unit. The gaze distance of the user is calculated by obtaining a distance to the gaze object that the user gazes at.
According to the present invention, the gaze distance can be calculated with higher accuracy.

Further, in the display device according to the present invention, the display unit includes: an emission unit that emits image light; and an optical system that guides the image light emitted from the emission unit to each of the right eye and the left eye of the user. A changing unit that displaces at least a part of the light emitting unit or the optical system and changes the angle of image light incident on the right and left eyes of the user from the optical system. The control unit operates the changing unit according to the gaze distance of the user.
According to the present invention, the display mode of an image can be adjusted by displacing the member.

Further, according to the present invention, in the display device, the control unit controls a position of an image in a display range where the user visually recognizes an image displayed by the display unit according to a gaze distance of the user. It is characterized by.
ADVANTAGE OF THE INVENTION According to this invention, the display position of an image can be adjusted appropriately according to a user's gaze distance.

The display device may further include an image generation unit that generates an image to be displayed by the display unit based on image data, and the control unit is configured to generate the image generation unit according to a gaze distance of the user. A part of the image data is cut out to generate an adjustment image, and the adjustment image is displayed on the display unit.
According to the present invention, by cutting out an image, the adjustment range of the display position of the image can be expanded, and uncomfortable feeling when changing the display position of the image can be eliminated.

Further, according to the present invention, in the display device described above, when the gaze distance of the user is closer than a preset reference distance, the control unit uses the display unit to select either the right eye or the left eye of the user. An image corresponding to is displayed.
According to the present invention, even when the user's gaze distance is short, the burden on the user can be reduced.

In order to achieve the above object, the present invention provides a display device that displays an image corresponding to each of a user's right eye and left eye so that an outside scene can be visually recognized, and an emitting unit that emits image light. An optical system that guides the image light emitted by the emission unit to each of the right eye and the left eye of the user, and an emission direction of the emission unit or from the optical system to the right eye and the left eye of the user. A change unit that changes an incident angle of incident image light; an image generation unit that generates an image displayed by the display unit based on image data; and a gaze distance of the user is calculated and calculated. The process of operating the changing unit according to the gaze distance performed, and the image generation unit cut out a part of the image data according to the gaze distance of the user to generate an image for adjustment, The process of displaying the image for adjustment on the display unit Even without, characterized in that it comprises a control unit for executing either.
ADVANTAGE OF THE INVENTION According to this invention, when a user visually recognizes an outside scene and an image, the display mode of an image can be adapted to the distance which a user gazes at. For this reason, a display image can be adjusted appropriately, for example, a user's load can be reduced and the visibility of a display image can be improved.

Further, the present invention is characterized in that the display device includes an external imaging unit, and the control unit is capable of executing processing for displaying an image based on a captured image of the external imaging unit on the display unit. To do.
ADVANTAGE OF THE INVENTION According to this invention, the user can visually recognize the external scene which is the scenery of the outer side of a display part.

Further, in the display device according to the present invention, the display unit is configured such that the user can visually recognize an outside scene by external light transmitted through the display unit, and an image based on a captured image of the external imaging unit is It is characterized by being displayed so as to be visible simultaneously with the outside scene.
ADVANTAGE OF THE INVENTION According to this invention, when displaying an external scene and a display image so that visual recognition is possible, a display image can be adjusted appropriately, for example, a user's load can be reduced and the visibility of a display image can be improved. .

In the display device according to the aspect of the invention, when the control unit displays an image based on the captured image on the display unit, the control unit displays an image based on the captured image compared to external light transmitted through the display unit. A process for improving visibility is performed.
According to the present invention, the image displayed on the display unit can be clearly recognized, and the outside scene visually recognized by the external light and the display image can be easily distinguished.

In the display device according to the aspect of the invention, when the control unit displays an image based on the captured image on the display unit, the brightness or color of the image based on the captured image is changed, or the imaging By performing edge enhancement processing on the image based on the image, processing for improving the visibility of the image based on the captured image is performed.
ADVANTAGE OF THE INVENTION According to this invention, the visibility of the display image of a display part can be improved effectively.

Further, according to the present invention, in the display device, the control unit displays an image based on a captured image of the external imaging unit on the display unit corresponding to either the right eye or the left eye of the user. It is characterized by doing.
ADVANTAGE OF THE INVENTION According to this invention, a user can be made to visually recognize the image which a display part displays, without being influenced by a user's gaze distance.

According to the present invention, in the above display device, an image based on an image captured by the external imaging unit corresponds to each of the right eye and the left eye of the user in a form that does not have left and right parallax by the display unit. And displaying it.
ADVANTAGE OF THE INVENTION According to this invention, it is hard to receive to the influence of a user's gaze distance, and a display image can be visually recognized favorably.

Moreover, the present invention is characterized in that, in the display device, the control unit enlarges a captured image of the external imaging unit and displays the enlarged image on the display unit.
According to the present invention, an outside scene can be enlarged and displayed.

Moreover, the present invention is characterized in that, in the display device, the control unit displays an image based on a captured image of the external imaging unit when receiving an operation of the user.
According to the present invention, an outside scene can be displayed according to a user's operation.

In the display device according to the aspect of the invention, the control unit detects a gaze distance of the user, and when the detected gaze distance is shorter than a preset distance, the control unit is based on a captured image of the external imaging unit. An image is displayed.
According to the present invention, it is possible to display an outside scene corresponding to the gaze distance of the user.

Further, the present invention is characterized in that, in the display device, the external imaging unit images a range including the visual line direction of the user when the display unit is mounted.
ADVANTAGE OF THE INVENTION According to this invention, the image based on the captured image of the range containing a user's gaze direction can be displayed.

In order to achieve the above object, the display device control method of the present invention is a display device including a display unit that displays an image corresponding to each of the right eye and the left eye of a user so that an outside scene can be visually recognized. The display mode of the image by the display unit is controlled according to the gaze distance of the user.
ADVANTAGE OF THE INVENTION According to this invention, when a user visually recognizes an outside scene and an image, the display mode of an image can be adapted to the distance which a user gazes at. For this reason, a display image can be adjusted appropriately, for example, a user's load can be reduced and the visibility of a display image can be improved.

In order to achieve the above object, the present invention can be executed by a computer that controls a display device including a display unit that displays an image corresponding to each of the user's right eye and left eye so that the outside scene can be visually recognized. The computer causes the computer to function as a control unit that controls an image display mode by the display unit according to the gaze distance of the user.
By executing the program of the present invention by the computer, when the user visually recognizes the image displayed on the display device and the outside scene, the display mode of the image can be adapted to the distance that the user gazes at. For this reason, a display image can be adjusted appropriately, for example, a user's load can be reduced and the visibility of a display image can be improved.

The perspective view explaining the appearance of HMD of a 1st embodiment of the present invention. The perspective view which shows the internal structure which removed the exterior member from the virtual image display apparatus. (A) is a top view which shows the external appearance of a virtual image display apparatus, (B) is a front view, (C) is a side view. The perspective view which shows the external appearance of a virtual image display apparatus from another angle. Sectional drawing of the planar view of the main-body part which comprises a virtual image display apparatus. Sectional drawing explaining the arrangement | positioning relationship between an optical apparatus part and a light guide member. Explanatory drawing which shows a viewer's visual recognition distance and a convergence angle. The figure explaining the relationship between a convergence angle and display distance. The functional block diagram of each part of HMD. It is explanatory drawing which shows the adjustment method of the display mode of an image, (A) is a schematic diagram which shows a user's gaze distance and a convergence angle, (B) is a schematic diagram which shows the visual field of a user's both eyes. . It is explanatory drawing which shows the adjustment method of the display mode of an image, (A) shows the operation | movement which adjusts a display position by mechanical operation, (B) and (C) show the operation | movement which adjusts a display position by image processing. . The flowchart which shows operation | movement of HMD. The flowchart which shows a reference data production | generation process. The flowchart which shows a convergence angle detection process. 5 is a flowchart showing image adjustment processing. The flowchart which shows operation | movement of HMD in 2nd Embodiment. The flowchart which shows a microscope display process. The figure which shows the example of a display in a microscope display process. The flowchart which shows a telescope display process. 10 is a flowchart illustrating image adjustment processing according to the second embodiment. The figure which shows the structure of HMD in 3rd Embodiment.

[First Embodiment]
Hereinafter, an embodiment of a virtual image display device according to the present invention will be described in detail with reference to FIG.

  FIG. 1 is an explanatory diagram showing an external configuration of an HMD (Head Mounted Display) 1 as a display device according to a first embodiment to which the present invention is applied. The HMD 1 includes a virtual image display device 100 (display unit) that allows a user to visually recognize a virtual image while being mounted on the user's head, and a control device 300 that controls the virtual image display device 100. The control device 300 also functions as a controller for the user to operate the HMD 1.

  The HMD 1 of the present embodiment is an optically transmissive display device that allows a user to visually recognize a virtual image with the virtual image display device 100 and at the same time to see the outside scene through the virtual image display device 100. In the present specification, a virtual image visually recognized by the user with the HMD 1 is referred to as a “display image” for convenience. The virtual image display device 100 emitting image light generated based on image data is also referred to as “displaying an image”.

The virtual image display device 100 is connected to the control device 300 by a connection cable 41. The connection cable 41 includes a power cable (not shown) that supplies power from the control device 300 to the virtual image display device 100, and a data communication cable (not shown) that transmits and receives various data between the virtual image display device 100 and the control device 300. Built in.
Further, the connection cable 41 and the audio cable 46 branched off are connected to the control device 300. The audio cable 46 is connected to the right earphone 32 and the left earphone 34 and the microphone 63. The right earphone 32 and the left earphone 34 output sound based on the sound signal output from the sound processing unit 190 (FIG. 9).

The microphone 63 collects sound and outputs the sound signal to the sound processing unit 190 (FIG. 9) described later. For example, the microphone 63 may be a monaural microphone or a stereo microphone, may be a directional microphone, or may be an omnidirectional microphone.
Here, when the virtual image display device 100 has a power source such as a battery, the virtual image display device 100 and the control device 300 can be connected by wireless communication.

First, the configuration of the virtual image display device 100 will be described.
As shown in FIG. 1, the virtual image display device 100 constitutes a head mounted display having an appearance like glasses. The virtual image display device 100 allows an observer or a user wearing the virtual image display device 100 to visually recognize image light (video light) based on a virtual image, and allows the observer to view an external image (outside scene) through a see-through or Can be observed. The virtual image display device 100 includes first and second optical members 101a and 101b that cover the front of an observer (user or user) in a transparent manner, a frame portion 102 that supports both optical members 101a and 101b, and a frame portion 102. The first and second image forming main body portions 105a and 105b are added to a portion extending from the left and right ends to a rear vine portion (temple) 104. Here, the first display device 100A in which the first optical member 101a on the left side and the first image forming main body portion 105a in the drawing are combined is a portion that forms a virtual image for the right eye, and can be used alone as a virtual image display device. Function. Further, the second display device 100B in which the second optical member 101b on the right side in the drawing and the second image forming main body portion 105b are combined is a portion that forms a virtual image for the left eye, and functions alone as a virtual image display device. To do.

  FIG. 2 shows an internal structure of the virtual image display device 100. By comparing FIG. 2 with FIG. 1, the appearance and the inside of the virtual image display device 100 are compared. For example, the first and second image forming main body portions 105 a and 105 b are respectively configured by the projection lens 30 housed in the lens barrel portion 39 and the image display device 80 including the video display element 82.

  As shown in each drawing from FIG. 1 to FIG. 4, the frame 102 provided in the virtual image display device 100 is disposed on the back side along the frame 107 disposed on the upper end side and the frame 107. The resin part 108 is provided. The virtual image display device 100 has a configuration without a frame-like portion on the lower side. The frame 107 constituting the frame portion 102 is an elongated plate-like member bent in a U-shape, and includes a front portion 107a extending in the left and right lateral directions, which are corresponding directions of the eyes for the observer, and the observer. A pair of side surface portions 107b and 107c extending in the depth direction corresponding to the front and rear directions. Both side surface portions 107b and 107c are not strictly parallel, but have a small angle so as to spread slightly toward the tip. The frame 107, that is, the front surface portion 107a and the side surface portions 107b and 107c are metal integrated parts formed of aluminum die casting or other various metal materials. The resin portion 108 is disposed along the frame 107, and by fitting with the frame 107, in cooperation with the resin portion 108, for example, various cables for image formation can be stored. In the frame 107, the width in the depth direction of the front portion 107a and the resin portion 108 is approximately the same as the thickness or width of the light guide device 20 corresponding to the first and second optical members 101a and 101b. The first optical member 101a and the first image forming main body portion 105a are aligned on the left side of the frame 107, specifically, the portion from the left end portion to the side surface portion 107b in the front portion 107a, and the first image forming main body portion 105a is directly aligned by, for example, screwing. It is supported by being fixed. In addition, the second optical member 101b and the second image forming main body 105b are aligned on the right side of the frame 107, specifically, the portion from the right end portion to the side surface portion 107c toward the front portion 107a, for example, by screwing. It is supported by being directly fixed by. The first optical member 101a and the first image forming main body 105a are aligned with each other by fitting or the like, and the second optical member 101b and the second image forming main body 105b are aligned with each other by fitting or the like. .

  The frame 107 and the resin portion 108 constituting the frame portion 102 not only support the first and second image forming main body portions 105a and 105b, but also cooperate with a cover-like exterior member 105d that covers them. It has a role of protecting the inside of the second image forming main body portions 105a and 105b. The frame 107 is separated from the first and second optical members 101a and 101b connected to the first and second image forming main body portions 105a and 105b, or the upper portion excluding the base side of the light guide device 20, or It touches loosely. For this reason, even if there is a difference in thermal expansion coefficient between the central light guide device 20 and the frame portion 102 including the frame 107, the light guide device 20 is allowed to expand within the frame portion 102, and the light guide It is possible to prevent the device 20 from being distorted, deformed, or damaged.

  A nose receiving portion 40 is provided along with the frame portion 102. The nose receiving portion 40 has a role of supporting the frame portion 102 by coming into contact with the observer's nose. That is, the frame portion 102 is disposed in front of the observer's face by the nose support portion 40 supported by the nose and the pair of temple portions 104 supported by the ears. The nose receiving portion 40 is fixed by screwing at a front portion 107 a of one frame 107 constituting the frame portion 102. In addition, the external appearance shown with reference to FIG. 1 as mentioned above is an example. For example, the configuration of parts that are not directly involved as an optical mechanism, such as a mechanism that is fixed by screwing, can be changed as appropriate.

  As shown in FIG. 2 and the like, the first display device 100A is considered to include a projection see-through device 70 (optical system) that is an optical system for projection and an image display device 80 (emission unit) that forms image light. be able to. The image display device 80 corresponds to a projection unit that projects image light onto the projection fluoroscopic device 70. The second display device 100B is configured to be bilaterally symmetric with the first display device 100A. The arrangement of the members included in the second display device 100B and the members inside the second display device 100B are the same as those of the first display device 100A. Therefore, hereinafter, the first display device 100A will be described and replaced with the description of the second display device 100B.

  The projection see-through device 70 has a role of projecting the image formed by the image display device 80 as a virtual image onto the observer's eyes. The projection see-through device 70 includes the first optical member 101a or the light guide device 20 and the projection lens 30 for image formation. The first optical member 101a or the light guide device 20 includes a light guide member 10 for light guide and see-through and a light transmitting member 50 for see-through. The first image forming main body portion 105 a includes the image display device 80 and the projection lens 30. As will be described in detail later, the projection lens 30 constituted by a plurality of lenses is housed in the lens barrel portion 39 and constitutes the optical device portion 130 together with the lens barrel portion 39. The optical device unit 130, that is, the projection lens 30 is fixed at the end 39 t of the lens barrel unit 39 in a state of being accurately positioned with respect to the light guide device 20 by fitting.

  The image display device 80 includes a backlight (right backlight 221 in FIG. 9) as an illumination device 81 that emits illumination light to the image display element 82 in addition to the image display element (image element) 82 that is a transmissive spatial light modulator. Left backlight 222). In addition, the image display device 80 includes drive control units (the right drive control unit 211 and the left drive control unit 212 in FIG. 9) that control the operation of the video display element 82 and the like. The video display element 82 is housed in a video element case 86, and is assembled to the lens barrel portion 39 that houses the projection lens 30 for image formation via the video element case 86. In other words, the lens barrel portion 39 is a connecting member that connects the image display element 82, the projection lens 30, the light guide device 20, and the like.

  As described above, the light guide device 20 is a block-shaped member including the light guide member 10 for light guide and see-through and the light transmitting member 50 for see-through. The light guide member 10 is a part of the prism type light guide device 20 and is an integral member, but is divided into a first light guide portion 11 on the light exit side and a second light guide portion 12 on the light incident side. Can be captured. The light transmission member 50 is a member (auxiliary optical block) that assists the see-through function of the light guide member 10, and is fixed integrally with the light guide member 10 to form one light guide device 20. Of the light guide device 20 having the above-described configuration, the leading end 12j located on the light source or the light incident side (base side) is fitted to the end 39t of the lens barrel 39, so that the projection lens 30 has high accuracy. Well positioned and fixed.

  Here, for example, as shown in FIG. 3A to FIG. 3C, FIG. 4, etc., among the optical surfaces constituting the light guide device 20, a first exposed surface 20 a that is a front side (outside) exposed surface. The second exposed surface 20b, which is the exposed surface on the back side (inner side), is a portion that is exposed to the outside and affects the function of the see-through. As shown in the figure, the first exposed surface 20a is composed of a third surface S13 of the optical surfaces of the light guide member 10 and a third transmission surface S53 which is an optical surface of the light transmission member 50, and the second surface. The exposed surface 20 b includes a third surface S <b> 11 and a fourth surface S <b> 14 among the optical surfaces of the light guide member 10, and a first transmission surface S <b> 51 among the optical surfaces of the light transmission member 50.

  Hereinafter, the image display device 80 and the projection lens 30 constituting the first image forming main body 105a (see FIG. 1) will be described in detail with reference to FIG.

  In addition to the video display element 82 described above, the image display apparatus 80 includes an illumination apparatus 81 that emits illumination light to the video display element 82, and a drive control unit 84 that controls operations of the illumination apparatus 81 and the video display element 82. And have.

  The illumination device 81 of the image display device 80 includes a light source that generates light including three colors of red, green, and blue, and a backlight light guide unit that diffuses light from the light source into a light beam having a rectangular cross section. . The video display element (video element) 82 is formed of, for example, a liquid crystal display device and includes a plurality of pixels. The video display element (video element) 82 should be a display target of a moving image or the like by spatially modulating the illumination light from the illumination device 81. Form image light. Although not shown, the drive control unit 84 supplies an image signal to a light source driving circuit that supplies electric power to the illumination device 81 and emits illumination light with a stable luminance, and a video display element (video element) 82. Alternatively, a driving signal is output to form a liquid crystal driving circuit that forms color video light or image light that is the source of a moving image or still image as a transmittance pattern. Note that the liquid crystal driving circuit can have an image processing function, but an external control circuit can also have an image processing function.

  The projection lens 30 is a projection optical system that includes lenses 31 to 33 as three optical elements along the incident side optical axis as a component, and a lens barrel portion that houses these lenses (optical elements) 31 to 33. 39. The lenses 31 to 33 are aspheric lenses including both a non-axisymmetric aspheric surface (non-axisymmetric aspheric surface) and an axisymmetric aspheric surface (axisymmetric aspheric surface), and are light guide members that are relay optical systems. An intermediate image corresponding to the display image of the video display element 82 is formed inside the light guide member 10 in cooperation with a part of the light guide member 10. In addition, among each lens (optical element) 31-33, the lens surface 31a which is the light emission surface of the first lens 31 is a non-axisymmetric aspheric surface, and the lens surfaces other than the lens surface 31a are as follows. It is an axisymmetric aspherical surface.

  Hereinafter, the light guide device 20 and the like will be described in detail. As described above, the light guide device 20 includes the light guide member 10 and the light transmission member 50. Among these, as for the light guide member 10, the center side (front side) part near a nose is extended linearly in planar view. Of the light guide member 10, the first light guide portion 11 disposed on the center side close to the nose, that is, on the light exit side, includes a first surface S 11, a second surface S 12, and a surface having an optical function. The second light guide portion 12 having the third surface S13 and disposed on the peripheral side away from the nose, that is, on the light incident side, has a fourth surface S14 and a fifth surface as a surface having an optical function. S15. Among these, the first surface S11 and the fourth surface S14 are continuously adjacent, and the third surface S13 and the fifth surface S15 are continuously adjacent. In addition, the second surface S12 is disposed between the first surface S11 and the third surface S13, and the fourth surface S14 and the fifth surface S15 are adjacent to each other at a large angle. Furthermore, here, the first surface S11 and the third surface S13 which are arranged to face each other are substantially parallel to each other. On the other hand, other surfaces having an optical function, that is, the second surface S12, the fourth surface S14, and the fifth surface S15 are non-axisymmetric curved surfaces (free curved surfaces).

  Here, among the plurality of surfaces constituting the light guide member 10, for the surfaces S14 and S15 other than the surfaces from the first surface S11 to the third surface S13, at least one free-form surface has a sign of curvature depending on the direction. It includes at least one different point. As a result, the light guide member 10 can be miniaturized while precisely guiding the image light.

  In the light guide device 20, the light guide member 10 is joined to the light transmission member 50 via the adhesive layer CC, and is constituted by the joint surface of the light guide member 10 and the light transmission member 50 and the adhesive layer CC. The part to be formed is referred to as a joint CN. The light guide device 20 is formed by coating the base material to be the light guide member 10 and the light transmitting member 50 at the joint portion CN and then coating the joined base material by a dipping process. Yes. That is, the hard coat layer 27 of the light guide member 10 is provided on the entire light guide device 20 together with the light transmissive member 50.

  The main body 10s of the light guide member 10 is formed of a resin material exhibiting high light transmittance in the visible range, and is molded by, for example, injecting and solidifying a thermoplastic resin in a mold. In addition, as a material of the main body 10s, for example, a cycloolefin polymer or the like can be used. Although the main body 10s is an integrally formed product, the light guide member 10 can be functionally divided into the first light guide portion 11 and the second light guide portion 12 as described above. The first light guide portion 11 allows the image light GL to be guided and emitted, and allows the external light HL to be seen through. The second light guide portion 12 allows the image light GL to be incident and guided.

  In the first light guide portion 11, the first surface S11 functions as a refracting surface that emits the image light GL to the outside of the first light guide portion 11, and also functions as a total reflection surface that totally reflects the image light GL on the inner surface side. It is an optical surface. The first surface S11 is disposed in front of the eye EY and has a planar shape as described above. The first surface S11 is a surface formed by the hard coat layer 27 applied to the surface of the main body 10s.

  The second surface S12 is the surface of the main body 10s, is an optical surface with the half mirror layer 15 attached to the surface, and is a non-axisymmetric aspheric surface. The half mirror layer 15 is a light-transmissive reflective film (that is, a semi-transmissive reflective film). The half mirror layer (semi-transmissive reflective film) 15 is formed not on the entire second surface S12 but on a partial region (not shown) in which the second surface S12 is narrowed mainly in the vertical direction along the Y axis. . The half mirror layer 15 is formed by forming a metal reflective film or a dielectric multilayer film on the partial area PA in the lower ground of the main body 10s. The reflectance of the half mirror layer 15 with respect to the video light GL is set to be 10% or more and 50% or less in the assumed incident angle range of the video light GL from the viewpoint of facilitating observation of the external light HL by see-through. The reflectance of the half mirror layer 15 of the specific embodiment with respect to the video light GL is set to 20%, for example, and the transmittance with respect to the video light GL is set to 80%, for example.

  The third surface S13 is an optical surface that functions as a total reflection surface that totally reflects the video light GL on the inner surface side. The third surface S13 is arranged substantially in front of the eye EY, and has a planar shape like the first surface S11. The first surface S11 and the third surface S13 are parallel to each other. Due to the smooth surface, when the external light HL is viewed through the first surface S11 and the third surface S13, the diopter is 0, and in particular, no zooming occurs. Yes. The third surface S13 is a surface formed by the hard coat layer 27 applied to the surface of the main body 10s.

  In the second light guide portion 12, the fourth surface S14 is an optical surface that functions as a total reflection surface that totally reflects the video light GL on the inner surface side, and is an axisymmetric aspheric surface. The fourth surface S <b> 14 also functions as a refracting surface that allows the image light GL to enter the second light guide portion 12. That is, the fourth surface S <b> 14 serves both as a light incident surface on which the image light GL is incident on the light guide member 10 from the outside and a reflection surface on which the image light GL is propagated inside the light guide member 10. The fourth surface (light incident surface) S14 is a surface formed by the hard coat layer 27 applied to the surface of the main body 10s.

  In the second light guide portion 12, the fifth surface S15 is an optical surface formed by forming a light reflecting film RM formed of an inorganic material on the surface of the main body 10s. The fifth surface S15 is an axisymmetric aspheric surface that functions as a reflecting surface.

  As described above, the light transmission member 50 is integrally fixed to the light guide member 10 to form one light guide device 20, and is a member (auxiliary optical block) that assists the see-through function of the light guide member 10. . The light transmission member 50 includes a first transmission surface S51, a second transmission surface S52, and a third transmission surface S53 as side surfaces having an optical function. Here, the second transmission surface S52 is disposed between the first transmission surface S51 and the third transmission surface S53. The first transmission surface S51 is on a surface obtained by extending the first surface S11 of the light guide member 10, and the second transmission surface S52 is joined and integrated with the second surface S12 by the adhesive layer CC. It is a curved surface, and the third transmission surface S53 is on a surface obtained by extending the third surface S13 of the light guide member 10. Among these, since 2nd transmissive surface S52 and 2nd surface S12 of the light guide member 10 are integrated by joining via the thin contact bonding layer CC, they have the shape of the substantially same curvature.

  The light transmissive member (auxiliary optical block) 50 exhibits high light transmittance in the visible range, and the main body portion of the light transmissive member 50 is a thermoplastic resin material having substantially the same refractive index as the main body 10 s of the light guide member 10. Is formed. The light transmitting member 50 is similar to the light guiding member 10 in that a hard coat layer 27 is applied to the surface of the main body portion. That is, the first transmission surface S51 and the third transmission surface S53 are surfaces formed by the hard coat layer 27 provided on the surface of the main body portion.

  In the present embodiment, inside the light guide member 10, the image light from the image display element 82 is transmitted five times from the first surface S11 to the fifth surface S15 including the second surface S12 that is a non-axisymmetric aspheric surface. The light is guided by reflection. In addition, the light guide device 20 that covers the front of the eye as a whole includes the third surface S13 and the third transmission surface S53 as the first exposure surface 20a, and the first surface S11 and the first transmission surface S51 that are parallel to these are the second surfaces. The half mirror layer 15 is incorporated along the second surface S12 including the exposed surface 20b. As a result, both the display of the image light GL and the see-through for visually recognizing the external light HL can be achieved, and the light guide member 10 can correct the aberration of the image light GL.

  Hereinafter, the optical path of the image light GL and the like in the virtual image display device 100 will be specifically described with reference to FIG. The image light GL emitted from the image display element (image element) 82 passes through the lenses 31 to 33 constituting the projection lens 30 and is converged while being given the desired astigmatism. 10 is incident on the fourth surface S14 having positive refractive power. This astigmatism is canceled while passing through each surface of the light guide member 10, and image light is finally emitted toward the observer's eye in a state corresponding to the initial display. Is done.

  Here, the arrangement of the optical device unit 130 including the projection lens 30 with respect to the light guide member 10 will be described. The first reference optical axis AX1 on the optical device unit 130 or the projection lens 30 side is inclined by 5 to 45 ° with respect to the second reference optical axis AX2 on the light emission side of the light guide member 10. Here, the second reference optical axis AX2 corresponds to a line-of-sight reference axis extending in the Z direction corresponding to the front direction of the observer's face. In other words, as shown in FIG. 6, the first reference optical axis AX1 on the projection lens 30 side forms an obtuse angle α with respect to the horizontal reference axis AX3 parallel to the X direction corresponding to the alignment direction of the eyes EY. Yes. In this case, the horizontal reference axis AX3 is orthogonal to the second reference optical axis (line-of-sight reference axis) AX2, and passes through the intersection between the first reference optical axis AX1 and the fourth surface S14. Under such a premise, the obtuse angle α formed by the first reference optical axis AX1 and the horizontal reference axis AX3 is 95 ° to 135 °. Thereby, the fitting property when the first image forming main body portion 105a including the projection lens 30 and the light guide device 20 including the light guide member 10 are arranged along the temporal region from the face can be improved. In particular, the device can be made thinner and lighter. In addition, by setting the obtuse angle α formed by the first reference optical axis AX1 and the horizontal reference axis AX3 to 95 ° or more, it is possible to prevent the connecting portion between the projection lens 30 and the light guide member 10 from being angular and protruding from the face. . On the other hand, by setting the obtuse angle α formed by the two axes AX1 and AX3 to 135 ° or less, it is possible to prevent the tip of the projection lens 30 and the portion corresponding to the image display element 82 from protruding laterally or laterally of the face. Note that the obtuse angle α formed by the two axes AX1 and AX3 is set to 105 ° or less from the viewpoint of more reliably preventing a portion corresponding to the video display element 82 from protruding from the face. That is, it is more preferable that the obtuse angle α formed by the first reference optical axis AX1 and the horizontal reference axis AX3 is 95 ° to 105 °.

  Note that the first reference optical axis AX1 and the horizontal reference axis AX3 are on a reference plane HM corresponding to the paper surface of FIG. 6 and extending along the XZ plane. The reference plane HM is a reference plane in optical design, but is slightly downward facing forward as shown in FIG. This is because it is natural for the observer to make the eye EY slightly downward, and the downward line of sight takes less burden on the eye EY.

  The light guide member 10 is slightly outwardly inclined with respect to the rotation angle about the Y axis perpendicular to the reference plane HM relative to the horizontal reference axis AX3. Specifically, the first surface S11 or the third surface S13 of the light guide member 10 forms an inclination angle β with respect to the horizontal reference axis AX3. As a result, as shown in FIG. 3A and the like, the first and second optical members 101a and 101b are arranged so as to slightly protrude outward on the tip side or the center side sandwiched between both. . Thereby, the fitting property at the time of arrange | positioning the 1st and 2nd optical members 101a and 101b along a face can be improved.

  Returning to FIG. 5, the optical path of the image light GL will be described. The image light GL that has entered the fourth surface S <b> 14 of the light guide member 10 and passed through the fourth surface S <b> 14 advances while converging and passes through the second light guide portion 12. In this case, the light is reflected by the fifth surface S15 having a relatively weak positive refractive power, and is incident on the fourth surface S14 again from the inside and reflected.

  The image light GL reflected by the fourth surface S14 of the second light guide portion 12 is incident on the third surface S13 having substantially no refractive power in the first light guide portion 11 and is totally reflected, substantially. Is incident on the first surface S11 having no refractive power and is totally reflected. The image light GL forms an intermediate image in the light guide member 10 that is a relay optical system before and after passing through the third surface S13. The image plane II of the intermediate image corresponds to the image plane OI of the video display element 82.

  The image light GL totally reflected by the first surface S11 is incident on the second surface S12. In particular, the image light GL incident on the half mirror layer 15 is partially transmitted through the half mirror layer 15 while partially transmitting. And is incident again on the first surface S11 and passes therethrough. The half mirror layer 15 acts as a layer having a relatively strong positive refractive power with respect to the image light GL reflected here. The first surface S11 acts as having no refractive power with respect to the video light GL passing through the first surface S11.

  The video light GL that has passed through the first surface S11 enters the pupil of the observer's eye EY or an equivalent position thereof as a substantially parallel light beam. That is, the observer observes an image formed on the video display element (video element) 82 by the video light GL as a virtual image.

  On the other hand, among the external light HL, the light incident on the + X side from the second surface S12 of the light guide member 10 passes through the third surface S13 and the first surface S11 of the first light guide portion 11, but this At this time, since the third surface S13 and the first surface S11 are substantially parallel to each other (that is, substantially planar optical surfaces that are substantially parallel to each other), almost no aberration or the like occurs. That is, the observer observes an external field image without distortion through the light guide member 10. Similarly, of the external light HL, the light incident on the −X side with respect to the second surface S12 of the light guide member 10, that is, the light incident on the light transmission member 50 is the third transmission surface S53 provided thereon. When passing through the first transmission surface S51, the third transmission surface S53 and the first transmission surface S51 are substantially parallel to each other, so that no aberration or the like occurs. That is, the observer observes an external field image without distortion through the light transmission member 50. Further, among the external light HL, the light incident on the light transmissive member 50 corresponding to the second surface S12 of the light guide member 10 is transmitted through the third transmissive surface S53 and the first surface S11. Since the surface S53 and the first surface S11 are substantially parallel to each other, aberrations and the like hardly occur. That is, the observer observes an external image with little distortion through the light transmission member 50. The second surface S12 of the light guide member 10 and the second transmission surface S52 of the light transmission member 50 both have substantially the same curved surface shape, have substantially the same refractive index, and the gap between them is substantially the same. The adhesive layer CC is filled with the same refractive index. That is, the second surface S12 of the light guide member 10 and the second transmission surface S52 of the light transmission member 50 do not act as a refractive surface with respect to the external light HL.

  However, since the external light HL incident on the half mirror layer 15 is partially reflected while partially transmitting through the half mirror layer 15, the external light HL from the direction corresponding to the half mirror layer 15 is The transmittance of the half mirror layer 15 is weakened. On the other hand, since the video light GL is incident from the direction corresponding to the half mirror layer 15, the observer can view the outside world together with the image formed on the video display element (video element) 82 in the direction of the half mirror layer 15. You will observe the image.

  Of the image light GL propagated in the light guide member 10 and incident on the second surface S <b> 12, the image light GL not reflected by the half mirror layer 15 enters the light transmissive member 50, but is provided in the light transmissive member 50. Returning to the light guide member 10 is prevented by an anti-reflection portion (not shown). That is, the image light GL that has passed through the second surface S12 is prevented from returning to the optical path and becoming stray light. In addition, the external light HL incident from the light transmitting member 50 side and reflected by the half mirror layer 15 is returned to the light transmitting member 50. However, by providing the light transmitting member 50 with an antireflection portion, the light guiding member 10 is provided. It is prevented from being injected. That is, it is possible to prevent the external light HL reflected by the half mirror layer 15 from returning to the optical path and becoming stray light.

  Further, a right optical system driving unit 85 (changing unit) is disposed around the video display element 82. The right optical system driving unit 85 includes, for example, an actuator (not shown), and swings the optical device unit 130 including the video display element 82 and the projection lens 30. As a result, the angle of the optical device section 130 with respect to the light guide member 10 changes in the directions of arrows SA1 and SA2 in the drawing. When the optical device unit 130 is tilted in the SA1 direction, the angle of the image light GL that enters the light guide member 10 from the projection lens 30 changes. That is, when the first reference optical axis AX1 is tilted to the SA1 side, the image light GL incident on the eye EY moves in the direction of the arrow SA2 in the figure due to the change in the reflection angle of the image light GL inside the light guide member 10. To do. The first display device 100A of FIG. 5 corresponds to the right eye of the user, and the video light GL appears to move to the right side for the user as the video light GL moves in the SA2 direction.

  In contrast, when the right optical system driving unit 85 swings the optical device unit 130 in the SB1 direction, the angle of the first reference optical axis AX1 of the video light GL with respect to the light guide member 10 changes to the SB1 side. As a result, the reflection angle of the video light GL at the light guide member 10 changes, and the optical axis (second reference optical axis AX2) of the video light GL incident on the eye EY changes to the arrow SB2 side. For this reason, the video light GL appears to move to the left side to the right eye of the user.

  As described above, in the first display device 100A, the right optical system driving unit 85 moves the optical device unit 130, whereby the image light GL incident on the right eye EY of the user can be moved in the left-right direction.

  Further, the second display device 100B is provided with a left optical system drive unit 87 (FIG. 9) that moves the optical device unit. The left optical system driving unit 87 (changing unit) includes an actuator (not shown) or the like as a power source in the same manner as the right optical system driving unit 85 described above. The inclination of the image light GL with respect to 10 is changed. By the operation of the left optical system driving unit 87, the inclination of the image light GL incident on the left eye of the user can be changed, and the image light GL can be moved in the left-right direction.

FIG. 7 is an explanatory diagram showing the viewing distance and the convergence angle of the observer, and shows the viewing distances different from those in FIGS. 7 (A) and 7 (B). FIG. 7C is a diagram showing the relationship between the convergence angle β, the inter-pupil distance PD, and the viewing distance d. FIG. 8 is a chart for explaining the relationship between the convergence angle β and the display distance.
Comparing FIG. 7A and FIG. 7B, the example of FIG. 7A shows a state in which the object OB is relatively placed near the two eyes EYa and EYb (distance d1) for visual recognition. FIG. 7B shows a state in which the object OB is placed relatively far from both eyes EYa and EYb (distance d2; d2> d1) for visual recognition. When visually recognizing an object OB at a close position as shown in FIG. 7A, since both eyes EYa and EYb are inward, the value of the convergence angle β, which is an angle formed by the lines of sight of the left and right eyes EYa and EYb, becomes large. . On the other hand, as shown in FIG. 7B, when the object OB is placed relatively far from both eyes EYa and EYb and viewed, the value of the convergence angle β decreases.

  When the image is viewed with binocular vision as in the virtual image display device 100 of the present embodiment, the convergence angle, which is the angle formed between the display target and the left and right eyes, plays an important role in grasping the sense of distance of the image light. . The observer determines the convergence angle β due to the left and right parallax in the displayed image. If the convergence angle β increases, the observer feels that the image is near, and if the convergence angle β decreases, the observer feels that the image is far away. . For example, as shown in FIG. 7C, assuming that the inter-pupil distance PD (the distance between the right eye and the left eye) of the observer is 65 mm, one point at the center of the image, that is, ahead of the principal ray direction. The relationship between the convergence angle β related to the center point CE of the virtual image visually recognized as a certain point and the display distance d, which is the distance from the assumed position of the virtual image by the image light to the positions of the eyes EYa and EYb (the assumed eye position EY), It is represented by a graph as shown in FIG. That is, in FIG. 8, the horizontal axis represents the display distance d (m), and the vertical axis represents the convergence angle β (°). In this case, for example, if the value of the convergence angle β is 1.0 °, the display distance d is about 4 m (or about 4 m), and if the value of the convergence angle β is 5.0 °, the display distance d. Is about several tens of centimeters.

The convergence angle β that determines the distance feeling of the observer is determined by the parallax (convergence angle β) set in the transmissive display device 100 as described above. This corresponds to the angle formed by the chief ray when the virtual image display device 100 displays an image, and the vergence angle of the eye of the observer viewing this image. That is, by adjusting the display position of the image, it is possible to affect the convergence angle when the observer visually recognizes the image. For this reason, the same viewing distance as that of the real object can be felt by adjusting the display position of the image in accordance with the convergence angle when the observer views the real object.
For example, if a display suitable for a state in which the convergence angle β is any value within a range of 1.0 ° to 5.0 ° is performed, the observer PE visually recognizes a nearby real object (object) OB. It is suitable for the case. In this example, a virtual image IM, which is an image that is visually recognized as being at an assumed display position by video light from the transmissive display device 100 in a state where the observer PE has the object OB in hand, You can work while watching from time to time at almost the same position as OB. That is, it is possible to display an image that looks the same as the object OB at a close position. In addition, when the object OB observer located at a position away from the observer PE views the image while viewing it, various displays such as adjusting the value of the convergence angle β to be larger than 5.0 ° are possible. . That is, in many cases of use at work sites that are diverse, by appropriately setting the parallax, that is, the value of the convergence angle β in any of the above ranges according to the use, an appropriate sense of distance can be obtained. An observer can experience the world of augmented reality (AR).

In addition, for example, when it is assumed that video light in a state of viewing an image at a relatively long distance such as watching a movie is viewed, the display may be performed in a usage mode (for example, about d = 25 m) that increases the display distance. it can. In this case, it is conceivable that the convergence angle β value is, for example, about 0.15 °. In the case of such an aspect, it can be said that it is suitable for use in which a video is viewed in a relaxed state.
In the present embodiment, as described above, the pair of left and right display devices 100A and 100B are tilted and arranged to tilt the principal ray of the left and right video light, thereby providing parallax for the left and right images. . Thereby, according to the set angle of parallax, for example, as shown in FIG. 5A or the like, the virtual image IM can be set at substantially the same position as the real object MA at the viewer's hand. Furthermore, it can be visually recognized so as to overlap with the same position, and can cope with augmented reality (AR). In this case, the chief rays PRa and PRb are inclined according to the parallax, so that the state of the vergence angle of the observer's eyes naturally fits the intended display position. Further, in the pair of left and right display devices 100A and 100B, the focal length is adjusted, and the image display position of the visually recognized virtual image IM is adjusted to the display distance d corresponding to the value of the angle θ indicating the parallax, thereby causing congestion. The so-called convergence angle and adjustment contradiction, where the angle and adjustment do not match, are also avoided or suppressed.

  Further, as shown in FIG. 8, the convergence angle β approaches 0 as the viewing distance of the observer becomes longer. For example, when the viewing distance is 15 m, the convergence angle is 0.3 °. That is, when the convergence angle β is 0.3 ° or less, the viewing distance is felt to be 15 m or more. In this case, the observer hardly perceives any further change in the viewing distance, and therefore, when the convergence angle β is 0.3 ° or less, the convergence angle may be regarded as almost zero. In the present embodiment, the control unit 140 can display an image corresponding to each of the right eye and the left eye of the observer in a manner that does not have left and right parallaxes by the virtual image display device 100. The convergence angle is almost zero. Specifically, it can be said to be 0.3 ° or less and 0.3 ° or less and 0.0 ° or more.

  As shown in FIGS. 1 and 3B, the virtual image display device 100 includes outer cameras 35 and 36 (second imaging unit and external imaging unit) that capture the user's (observer) line-of-sight direction. The outer camera 35 is embedded in the first image forming main body 105a located on the right side of the virtual image display device 100, that is, at the end on the first display device 100A side, and is exposed at the exterior member 105d. The outer camera 36 is embedded in the second image forming main body 105b located at the left side of the virtual image display device 100, that is, at the end on the second display device 100B side, and is exposed at the exterior member 105d.

  The outer cameras 35 and 36 are arranged on the exposed surface on the front side (outside) of the virtual image display device 100, face the front direction of the user's face, and view the outside scene that the user sees through the light guide member 10. Take an image. The imaging range, that is, the angle of view of the outer cameras 35 and 36 only needs to capture at least a part of the outside scene in the user's viewing direction with the virtual image display device 100 mounted. In addition, it is more preferable that the angle of view of the outer cameras 35 and 36 is set so that the entire field of view visible to the user through the light guide member 10 can be photographed. In other words, it is preferable that the imaging range, that is, the angle of view of the outer cameras 35 and 36 is a direction including the user's line-of-sight direction.

  The outer cameras 35 and 36 are connected to a control unit 140 (FIG. 9) built in the control device 300, execute imaging in accordance with the control of the control unit 140, and output captured image data. Since the outer camera 35 and the outer camera 36 are spaced apart from each other in the frame 102, parallax occurs between the captured image of the outer camera 35 and the captured image of the outer camera 36. Therefore, the outer cameras 35 and 36 are used as stereo cameras, and information in the depth direction can be acquired based on the captured image of the outer camera 35 and the captured image of the outer camera 36.

  As shown in FIG. 4, the virtual image display device 100 has four inner cameras 37 a, 37 b, 38 a, and 38 b (first imaging unit) on the exposed side facing the user's eye, that is, the back side (inner side). Prepare. The inner cameras 37a and 37b are provided in the first display device 100A corresponding to the right eye of the user, and the inner cameras 38a and 38b are provided in the second display device 100B corresponding to the left eye of the user. The inner cameras 37 a and 38 a are disposed in the vicinity of the nose receiving portion 40, and the inner cameras 37 b and 38 b are disposed at the end of the frame portion 102.

  The inner camera 37a images the right eye of the user from the upper left side, and the inner camera 37b images the right eye of the user from the upper right side. The inner camera 38a images the user's left eye from the upper right side, and the inner camera 38b images the user's left eye from the upper left side.

  Each of the inner cameras 37a, 37b, 38a, and 38b is connected to a control unit 140 (FIG. 9) of the control device 300, which will be described later, executes imaging under the control of the control unit 140, and outputs captured image data. Since the inner cameras 37a, 37b, 38a, and 38b image the user's eyeballs, the eye movements can be detected and measured using the captured images, and the user's line-of-sight direction and convergence angle can be obtained.

  As a technique for detecting eye movement by imaging an eyeball and measuring a line-of-sight direction, for example, there is a method of measuring a line-of-sight direction by detecting a reflected image reflected from the captured image to the eyeball. Also, for example, there is a method for obtaining the gaze direction by detecting the pupil center of the eyeball from the captured image. Further, a light emitting element (not shown) that irradiates the eyeball with infrared light is provided on the exposed surface on the back side of the virtual image display device 100, and reflected light on the eyeball surface is detected by the inner cameras 37a, 37b, 38a, and 38b. There is also a method for obtaining the gaze direction.

In this embodiment, as an example, the control unit 140 executes the following processes (1) to (3) for each of the right and left eyes of the user from the captured images of the inner cameras 37a, 37b, 38a, and 38b. .
(1) Processing for detecting the pupil center and obtaining the interpupillary distance.
(2) Processing for detecting the iris pattern appearing in the pupil and determining the pupil diameter.
(3) Processing for detecting the center of the pupil and determining the gaze direction.

Next, the configuration of the control device 300 will be described.
The control device 300 is a device for controlling the HMD 1. As shown in FIG. 1, the control device 300 includes a power switch 301 for turning on / off the power of the HMD 1, a track pad 302 and a key switch unit 303 operated by a user, in a substantially box-shaped housing. The control device 300 also includes an LED (Light Emitting Diode) indicator 304 indicating the operating state of the HMD 1, and an up / down key 305 for adjusting the volume output from the right earphone 32 and the left earphone 34.

The power switch 301 is a slide-type switch. By operating the power switch 301, not only the power of the control device 300 but also the power of the virtual image display device 100 is turned on / off.
The track pad 302 detects a user's contact operation on the operation surface of the track pad 302 and outputs operation data indicating the detected operation position. As the track pad 302, various track pads such as an electrostatic type, a pressure detection type, and an optical type can be adopted. Further, a multi-touch method that can simultaneously detect touch operations at a plurality of points may be employed.

FIG. 9 is a functional block diagram of each part constituting the HMD 1.
As shown in FIG. 9, the HMD 1 is connected to the external device OA via the interface 114. The interface 114 connects various external devices OA that are content supply sources to the control device 300. As the interface 114, for example, an interface corresponding to a wired connection such as a USB interface, a micro USB interface, or a memory card interface can be used.
The external device OA is used as an image supply device that supplies an image to the HMD 1, and for example, a personal computer (PC), a mobile phone terminal, a game terminal, or the like is used.

  The control device 300 includes a control unit 140, an operation unit 135, an input information acquisition unit 110, a storage unit 120, a transmission unit (Tx) 51, and a transmission unit (Tx) 52.

  The operation unit 135 detects an operation by the user. The operation unit 135 includes the power switch 301, the track pad 302, the key switch unit 303, and the up / down key 305 shown in FIG. The input information acquisition unit 110 acquires an operation signal or operation data output from the operation unit 135 in response to an operation input by the user.

  The storage unit 120 is a nonvolatile storage device, and stores various computer programs. The storage unit 120 may store image data to be displayed on the virtual image display device 100 of the HMD 1. The control unit 140 may generate display data displayed by the virtual image display device 100 by executing a program stored in the storage unit 120.

  A three-axis sensor 113, a GPS 115, and a communication unit 117 are connected to the control unit 140. The triaxial sensor 113 is a triaxial acceleration sensor, and the control unit 140 can acquire the detection value of the triaxial sensor 113. The GPS 115 includes an antenna (not shown), receives a GPS (Global Positioning System) signal, and obtains the current position of the control device 300. The GPS 115 outputs the current position and the current time obtained based on the GPS signal to the control unit 140. The GPS 115 may have a function of acquiring the current time based on information included in the GPS signal and correcting the time counted by the control unit 140 of the control device 300.

  The communication unit 117 executes wireless data communication conforming to a wireless communication standard such as a wireless LAN (WiFi (registered trademark)) or Miracast (registered trademark). The communication unit 117 can also perform wireless data communication conforming to a short-range wireless communication standard such as Bluetooth (registered trademark), Bluetooth Low Energy, RFID, or Felica (registered trademark).

When the external device OA is wirelessly connected to the communication unit 117, the control unit 140 acquires content data from the communication unit 117 and performs control for displaying an image on the virtual image display device 100. On the other hand, when the external device OA is wired to the interface 114, the control unit 140 acquires content data from the interface 114 and performs control for displaying an image on the virtual image display device 100. Therefore, the communication unit 117 and the interface 114 are hereinafter collectively referred to as a data acquisition unit DA.
The data acquisition unit DA acquires content data from the external device OA. The data acquisition unit DA acquires image data displayed by the HMD 1 from the external device OA.

  The control unit 140 includes a CPU, a ROM, a RAM (all not shown), and the like. The control unit 140 controls each unit of the HMD 1 by reading and executing a program stored in the storage unit 120 or the ROM. In addition, the control unit 140 executes the above-described program to perform an operating system (OS) 150, an image processing unit 160, a display control unit 170, an imaging control unit 181, an eye image analysis unit 182, an external image analysis unit 183, and change control. Functions as a unit 184, a drive control unit 185, and a sound processing unit 190.

  The image processing unit 160 acquires an image signal included in the content. The image processing unit 160 transmits a synchronization signal such as a vertical synchronization signal VSync and a horizontal synchronization signal HSync for displaying an image included in the content, a clock signal PCLK, and digital image data (Data in the figure) by the transmission unit 51. To do. Further, the image processing unit 160 may execute image processing such as resolution conversion processing, various tone correction processing such as adjustment of luminance and saturation, and keystone correction processing on the image data as necessary. Good. The transmission units 51 and 52 function as a transceiver for serial transmission between the control device 300 and the virtual image display device 100.

The display control unit 170 transmits a control signal to the image display device 80 included in the virtual image display device 100. The display control unit 170 transmits control signals to the right backlight control unit 201 and the left backlight control unit 202 that control lighting of the right backlight 221 and the left backlight 222 provided in the image display device 80. In addition, the display control unit 170 transmits control signals to the right drive control unit 211 and the left drive control unit 212 that drive the video display element 82.
The right backlight control unit 201 and the left backlight control unit 202 control lighting on / off and light emission luminance with respect to each of the right backlight 221 and the left backlight 222 in accordance with a control signal received from the display control unit 170. I do. The right backlight 221 and the left backlight 222 are light emitters such as LEDs and electroluminescence (EL), and may be configured using a laser light source or a lamp.

  Further, the right drive control unit 211 and the left drive control unit 212 respectively switch driving of the video display element 82 on / off in accordance with a control signal received from the display control unit 170. Therefore, the display control unit 170 controls the backlight on / off and the image display by the video display element 82 in the first display device 100A and the second display device 100B.

The audio processing unit 190 acquires an audio signal included in the content, amplifies the acquired audio signal, and outputs the amplified audio signal to the right earphone 32 and the left earphone 34 through the audio cable 46.
In addition, the sound processing unit 190 converts the sound collected by the microphone 63 into digital data. The voice processing unit 190 performs, for example, speaker recognition processing and voice recognition processing by extracting and modeling features from digital voice data. In the speaker recognition process, a human voice is detected from the sound collected by the microphone 63, the detected human voice is identified for each person, and the person who is speaking for each voice is specified. In the voice recognition process, for example, text conversion is performed to convert voice collected by the microphone 63 into text.

As described above, the virtual image display device 100 includes the projection fluoroscopy device 70 and the image display device 80 in each of the first display device 100A and the second display device 100B, and the image display device 80 is connected to the control unit via the interface 25. 140.
The outer cameras 35 and 36, the inner cameras 37a, 37b, 38a and 38b, the right optical system driving unit 85, and the left optical system driving unit 87 included in the virtual image display device 100 are connected to the control unit 140 via the interface 25. Is done.

The interface 25 is composed of a connection cable 41 and a connector for connecting the connection cable 41, and may be composed of a wireless communication line instead of the connection cable 41. The interface 25 outputs the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the image data Data transmitted by the transmission unit 51 to the corresponding reception units (Rx) 53 and 54. Further, the interface 25 outputs a control signal transmitted from the display control unit 170 to the receiving units 53 and 54 and the image display device 80.
The interface 25 transmits and receives control signals for the control unit 140 to control the outer cameras 35 and 36 and the inner cameras 37a, 37b, 38a, and 38b, and captured image data of these cameras.

Although not shown, the virtual image display device 100 may include a vibration sensor or a motion sensor (motion sensor). The vibration sensor (not shown) is configured using, for example, an acceleration sensor, and detects vibration caused by this operation when the user performs an operation (knock operation) to strike the frame 102. As the motion sensor, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, or the like can be used. For example, a nine-axis sensor that functions as an acceleration sensor (three axes), an angular velocity sensor (three axes), and a geomagnetic sensor (three axes) may be mounted. In this case, the control unit 140 can detect the movement of the virtual image display device 100, that is, the movement and direction of the head of the user wearing the virtual image display device 100. These vibration sensor and motion sensor are connected to the control device 300 via the interface 25.
The receiving units 53 and 54 function as a receiver for serial transmission between the control device 300 and the virtual image display device 100.

The HMD 1 has a function of adjusting the display mode of an image displayed on the virtual image display device 100 in accordance with the line-of-sight direction (gaze direction) of the user wearing the virtual image display device 100.
FIGS. 10 and 11 are explanatory views showing a method for adjusting the display mode of an image. The adjustment of the display mode of an image will be described with reference to FIGS. 10 and 11.

FIG. 10A is a schematic diagram showing the gaze distance and convergence angle of the user.
When the user views the object OB located in the outside scene, that is, in real space through the light guide member 10, the convergence angle CA of the left and right eyes varies depending on the distance (gaze distance) between the object OB and the eye EY. To do. The example on the left of FIG. 10A shows a state where the object OB is far away, and the example on the right side of the figure shows a state where the object OB is closer.

  As a human eye movement, a so-called eyeball convergence movement in which the line of sight moves toward the center of both eyes when one object OB is watched with both eyes is known. As shown in the figure, when it is considered that the line of sight of the user's right eye (EY (R) in the figure) and left eye (EY (L) in the figure) intersects with the object OB. The intersecting angle CA corresponds to the convergence angle. The convergence angle CA becomes smaller as the object to be watched is farther away, and becomes larger as it is closer. In other words, as the object OB is closer, the line of sight of the user's eyes is closer to the center.

  A position where the line of sight of the right eye EY of the user intersects the light guide member 10 is defined as a position 10a, and a position where the line of sight of the left eye EY intersects the light guide member 10 is defined as a position 10b. The distance between the position 10a and the position 10b varies depending on the distance from the eye EY to the object OB, similar to the magnitude of the convergence angle CA described above. That is, the distance between the position 10a and the position 10b has a correlation with the convergence angle CA. Specifically, the larger the convergence angle CA, the smaller the distance between the position 10a and the position 10b, and the smaller the convergence angle CA, the position 10a. -The distance between the positions 10b is large.

  Further, the distance between the position 10a and the position 10b is affected by the distance between the pupils of both eyes of the user. The interpupillary distance can be obtained based on the captured images of the inner cameras 37a, 37b, 38a, and 38b.

When the user views the outside scene through the light guide member 10, the HMD 1 displays an image that is visible while overlapping the outside scene. As an aspect of this display, an image (hereinafter referred to as an AR image) that provides information about an outside scene that is a real space and that has a so-called AR (Augmented Reality) effect can be displayed. The AR image is displayed so that information on an object existing in the real space overlaps with the object or is aligned with the object. By viewing the AR image, the user can simultaneously view the object and various information related to the object. The display of the AR image is characterized in that it is displayed in accordance with the position where the object can be seen by the user.
When the AR display is performed, the user's line of sight gazes at the object OB existing in the real space.

  When performing AR display, the HMD 1 can adjust the display size, display color, and the like of the AR image or a display object (image, graphic, character, etc.) included in the AR image in order to harmonize the AR image with the outside scene. In addition, the display object displayed so as to overlap with the object OB is switched between a display mode in which the object is displayed in front of the object OB and a display mode in which the display object is displayed so as to be viewed behind the object OB. May be. Alternatively, the AR image may be subjected to image processing, and displayed in a display mode in which the display object is visually recognized as having a shadow.

  Moreover, as another aspect which HMD1 displays when a user permeate | transmits the light guide member 10 and visually recognizes an external scene, it is not restricted to the information regarding the target object which a user gazes, and images containing various images, characters, etc. May be displayed. In this case, the process for adjusting the position where the object is visually recognized by the user is not performed, or an image is displayed at the end of the user's field of view so as not to prevent the object from being visually recognized. This type of display is performed, for example, when the user visually recognizes the outside scene while viewing the contents of the document. Even when this display is performed, the user's line of sight pays attention to the object OB existing in the real space, and when viewing the display image of the HMD 1, the line of sight moves away from the object OB and the display image is watched.

In any of the above examples, the user looks at the object OB in the real space and looks at the display image of the HMD 1 as necessary. The line of sight when the user gazes at the object OB passes through the positions 10 a and 10 b in the light guide member 10. If the position at which the HMD 1 displays an image by the virtual image display device 100 is a position that matches the positions 10a and 10b, the user increases the line of sight when viewing the object OB and when viewing the display image of the HMD 1. There is no need to move it.
Therefore, the HMD 1 controls the display mode according to the distance (gaze distance) to the object OB of the outside scene that the user gazes so that the user can view the display image without moving the line of sight greatly. . In this embodiment, as an example of display mode control, a case where a user views (views) a display image, that is, a display position is described.

FIG. 10B is a diagram schematically showing the visual field of both eyes of the user. The left side of the figure shows the field of view VR (L) of the left eye EY, and the right side of the figure shows the field of view VR (R) of the right eye EY.
The user's visual field VR (L) and visual field VR (R) include display images PL and PR generated by the image display device 80 and projected by the projection fluoroscopic device 70. Since the projection fluoroscopic device 70 causes the user to perceive (view) a virtual image by causing the image light GL to enter the eye EY, the physical display surface for displaying the display images PL and PR in front of the eye EY is Absent. However, for the user, the image light GL visually recognizes an image that occupies a part of the visual fields VR (L) and VR (R), and therefore the image light GL schematically forms the display images PL and PR here. It will be explained as a thing.

  The display images PL and PR are the maximum areas in which the virtual image display device 100 can display images, and can be referred to as displayable areas. An image P1 in the figure is an image that is actually displayed on the display images PL and PR that are displayable ranges by the virtual image display device 100. In the example of FIG. 10B, the same image P1 is visually recognized by both eyes of the user.

  When the image P1 in the visual field VR (L) is close to the position 10b and the image P1 in the visual field VR (R) is close to the position 10a, the user can visually recognize the object OB and the image P1 without greatly moving the line of sight. . In this case, since the movement of the user's line of sight in the depth direction (perspective direction) is small, the user is less likely to be burdened with the eyes related to the focus adjustment in the perspective direction. Therefore, when the object OB in the real space and the display image of the virtual image display device 100 are viewed alternately, for example, the load on the user's visual acuity can be effectively reduced.

Hereinafter, display control executed by the HMD 1 will be specifically described.
FIG. 11A shows an operation of adjusting the display position by a mechanical operation, and FIGS. 11B and 11C show an operation of adjusting the display position by image processing.

  The control unit 140 can change the angle of the optical device unit 130 with respect to the light guide member 10 in the first display device 100A and the second display device 100B by the right optical system drive unit 85 and the left optical system drive unit 87 described above. . The position of the image light GL incident on the right eye EY from the light guide member 10 is changed by the operation of the right optical system driving unit 85, so that the display image PR, that is, the displayable region in the visual field VR (R) moves. Similarly, the position of the image light GL incident on the left eye EY from the light guide member 10 is changed by the operation of the left optical system driving unit 87, so that the display image PL, that is, the displayable area in the visual field VR (L) moves. . When the display images PL and PR move, the position at which the user visually recognizes the image P1 can be changed without changing the position of the image P1 in each of the display images PL and PR. Further, as shown in FIG. 11A, since the display images PL and PR are adjusted according to the distance between the position 10a and the position 10b (FIG. 10A), the display image PL in the visual field VR (L) is adjusted. The moving direction and the moving direction of the display image PR in the visual field VR (R) are opposite. Therefore, the direction in which the right optical system driving unit 85 moves the optical device unit 130 and the direction in which the left optical system driving unit 87 moves the optical device unit 130 are opposite to each other.

  The control unit 140 can change the position where the image P1 is displayed on the display image PL and the position where the image P1 is displayed on the display image PR by image processing. This image processing can be executed independently of the control for moving the right optical system driving unit 85 and the left optical system driving unit 87, and can be realized by software, for example.

When moving the position of the image P1 in the display images PL and PR, the control unit 140 makes the display size of the image P1 smaller than the display images PL and PR. Here, the control unit 140 may reduce the image P1, but may cut out a part of the image P1 to generate a display image P2 and control the display position of the image P2. The process of cutting out the image P2 may be performed by the image processing unit 160 (image generation unit).
For example, as shown in FIG. 11B, a portion near the center of the image P1 is cut out as an image P2, and the image P2 is displayed on each of the display images PL and PR as shown in FIG. 11C. The display position of the image P2 in the display images PL and PR is adjusted according to the distance between the position 10a and the position 10b (FIG. 10A). Accordingly, the moving direction of the image P2 in the display image PL is opposite to the moving direction of the image P2 in the display image PR.

  Thus, the control unit 140 uses the right optical system driving unit 85 and the left optical system driving unit 87 to move the components of the projection fluoroscopic device 70 and the image display device 80, and display by image processing. Each position adjustment can be performed. In addition, the display position can be adjusted more effectively by executing a combination of this mechanical adjustment and adjustment by image processing.

Further, in the initial state where the control unit 140 does not control the display mode (display position) shown in FIGS. 11A to 11C, the display images PL and PR are adjusted so as to be suitable for the value of the predetermined convergence angle CA. Is set. The convergence angle CA corresponding to this initial position (default position) is, for example, in the range of 1 ° (degrees) or more and 5 ° or less, and more specifically, the distance from the user's eye EY to the object OB. It is a suitable position for the case of 4 m.
In another example, it is preferable that the line-of-sight direction is 0.2 ° to 2 ° and on the inner side as compared with the case where the line of sight of both eyes of the user faces directly in front (the distance to the object OB is infinity). In this case, 65 mm can be adopted as a standard value of the distance between the pupils (PD) of the user.

  In order to perform the above adjustment, the control unit 140 includes an imaging control unit 181, an eye image analysis unit 182, an external image analysis unit 183, a change control unit 184, and a drive control unit 185.

The imaging control unit 181 controls the outer cameras 35 and 36 and the inner cameras 37a, 37b, 38a, and 38b to execute imaging, and acquires captured image data.
The eye image analysis unit 182 analyzes the captured image data of the inner cameras 37a and 37b, and generates data regarding the state of the right eye EY of the user wearing the virtual image display device 100. The eye image analysis unit 182 analyzes the captured image data of the inner cameras 38a and 38b, and generates data related to the state of the user's left eye EY.

The outer image analysis unit 183 detects the object OB present in the direction in which the user is gazing by analyzing the captured image data of the outer cameras 35 and 36, and obtains the distance to the object OB. The outer cameras 35 and 36 function as stereo cameras spaced apart from each other as described above. The outer image analysis unit 183 detects the parallax between the captured image of the outer camera 35 and the captured image of the outer camera 36, and based on the detected parallax and a predetermined interval between the outer cameras 35 and 36, Calculation processing can be executed to determine the distance to the object OB. The distance obtained by the outer image analysis unit 183 may be a distance from the outer cameras 35 and 36 to the object OB. Moreover, you may convert the calculated | required distance into the distance from a user's eye EY to the target object OB using a known parameter.
Further, the outer image analysis unit 183 generates a convergence angle CA based on the data regarding the state of both eyes of the user generated by the eye image analysis unit 182 and the distance to the object OB obtained by the outer image analysis unit 183. Is calculated.

The change control unit 184 adjusts the display positions of the images on the first display device 100A and the second display device 100B according to the convergence angle CA calculated by the outer image analysis unit 183. The change control unit 184 obtains an adjustment amount in each of the first display device 100A and the second display device 100B. Here, the change control unit 184 can individually obtain the adjustment amount of the mechanical adjustment using the right optical system driving unit 85 and the left optical system driving unit 87 and the adjustment amount by the image processing. Further, the change control unit 184 can obtain the adjustment amount of the mechanical adjustment and the adjustment amount of the image processing when the mechanical adjustment and the image processing are combined.
Further, the change control unit 184 may obtain the adjustment amount due to the movement of the right optical system driving unit 85 and the left optical system driving unit 87, and after the adjustment is performed by the drive control unit 185, the convergence angle CA may be obtained again. Good. In this case, the change control unit 184 obtains the adjustment amount of the image processing based on the convergence angle CA obtained by the outer image analysis unit 183 after the mechanical adjustment.

  The adjustment amount obtained by the change control unit 184 can be a shift amount for shifting the display position from a standard position that is a preset image display position. The standard position is determined on the assumption that, for example, a user having a standard pupillary distance looks at the outside scene at a standard gaze distance, that is, a convergence angle, and is stored in the storage unit 120 by including it in the setting data 121. Is done. The interpupillary distance (interocular distance) can be set to, for example, 65 mm because 60 to 65 mm is a standard value. The standard gaze distance can be determined in accordance with the specifications of the virtual image display device 100. For example, it may be set to 4 m for the specifications used for general work, and 1 m for the specifications used for the work at a close position (such as the hand). The shift amount can be determined as the number of pixels in the video display element 82 or the drive amount driven by the drive control unit 185. For example, when the gaze distance is close and the convergence angle is large, the display position may be shifted inward (nose side) from the standard position.

The drive control unit 185 controls and drives the right optical system drive unit 85 and the left optical system drive unit 87 based on the adjustment amount of the mechanical adjustment obtained by the change control unit 184.
The right optical system driving unit 85 can swing the optical device unit 130 stepwise, for example. The same applies to the left optical system driving unit 87. In this case, the drive control unit 185 controls the number of stages for moving the right optical system drive unit 85 and the left optical system drive unit 87. When the right optical system driving unit 85 and the left optical system driving unit 87 can tilt the optical device unit 130 without any step, the change control unit 184 includes the right optical system driving unit 85 and the left optical system driving unit 87. Control the amount of movement.

  An arithmetic expression, a function, a matrix, a parameter, and the like used for arithmetic processing executed by the eye image analysis unit 182, the external image analysis unit 183, and the change control unit 184 of the control unit 140 are stored as setting data 121 in the storage unit 120. . Further, the reference value of the adjustable range and the set value of the target accuracy in the processing described later are stored in the storage unit 120 as the adjustment reference data 123.

  Further, the adjustment by the control unit 140 is performed with a goal that the convergence angle CA that matches the display positions of the images P1 and P2 is ± 1 ° with respect to the convergence angle CA calculated based on the user's line of sight. It is preferable. Here, the display positions of the images P1 and P2 are, for example, the center positions of the images P1 and P2.

In FIG. 10A, when the object OB is at the center of both eyes of the user, that is, an example in which the user visually recognizes the object OB so that the object OB is positioned at the center of both eyes. Show. When either the right eye or the left eye of the user is clearly the dominant eye, the position of the object OB is a position shifted from the center position of the right eye EY and the left eye EY to the dominant eye side. Become.
In this case, the outer image analysis unit 183 can detect the object OB biased toward the dominant eye, calculate the distance to the object OB with reference to the dominant eye, and calculate the convergence angle CA. It is good also as a structure. In addition, the setting data 121 may include an arithmetic expression, a function, a matrix, a parameter, and the like suitable for arithmetic processing based on the dominant eye. Further, the dominant eye may be designated by the user's operation, and a guidance message for requesting the dominant eye may be displayed on the virtual image display device 100. Moreover, you may perform the process which determines a dominant eye by the display of the image of the virtual image display apparatus 100, and operation in the control apparatus 300. FIG.

  In this case, the control unit 140 can also adjust the display position of the image according to the user's dominant eye. For example, assuming that the dominant eye is the right eye, the image display device 80 corresponding to the right eye EY displays an image at a position shifted by 20 pixels from the reference position, and the image display device 80 corresponding to the left eye EY. An image is displayed at a position shifted by 15 pixels from the reference position. As described above, when the display mode is controlled in accordance with the convergence angle, the image is displayed in a different display mode for each of the dominant eye and the non-dominant eye, thereby providing a more natural appearance. Can display images.

Furthermore, when detecting the convergence angle, the positions 10a and 10b shown in FIG. 10A may not be the target positions with respect to the center of both eyes due to the influence of the dominant eye. In other words, the object OB may be positioned on the dominant eye side instead of the front of the user. In this case, in the process of detecting the convergence angle CA, the control unit 140 may obtain the convergence angle with reference to the dominant eye side position (for example, the position 10a). Specifically, assuming the target position 10b ′ via the center of both eyes with respect to the position 10a and the position 10a, the convergence angle may be obtained based on the distance between the position 10a and the position 10b ′. Further, the convergence angle may be obtained by performing the same process on the dominant eye side position (for example, the position 10b). Furthermore, the final convergence angle may be calculated by obtaining an average value of the two convergence angles obtained in this way.
The dominant eye is not limited to the method of detecting as described above. When the object OB is located at the right side of the user's field of vision, the right eye is used as the dominant eye and the detection of the convergence angle and the control of the display mode are performed. You may perform the process of. When the object OB is at a position close to the left side of the user's field of vision, the left eye may be the dominant eye and the convergence angle detection and display mode control may be performed. That is, the setting related to the dominant eye may be changed according to the relative position between the object and the user. In addition, when the user makes a setting related to the dominant eye, this setting may be prioritized.

FIG. 12 is a flowchart showing the operation of the HMD 1 and shows processing relating to adjustment of the display position of the image.
The control unit 140 generates reference data serving as a reference for detecting the user's gaze direction (gaze direction) and gaze distance (step ST1). The reference data generated in step ST1 is stored in the storage unit 120 as reference data 122.
The control unit 140 executes a convergence angle detection process for detecting a convergence angle of both eyes of the user after starting display of an image (including video) or at the start of display (step ST2). Subsequently, the control unit 140 executes image adjustment processing for adjusting the image display position on the virtual image display device 100 based on the convergence angle detected in step ST2 (step ST3). Thereafter, the control unit 140 determines whether or not to end the display of the image (step ST4), and returns to step ST2 while continuing the display (step ST4; No). When the display is to be ended (step ST4; Yes), this process is ended.

The reference data 122 generated by the control unit 140 in step ST <b> 1 may be executed when the HMD 1 is started up or when the user uses the HMD 1 for the first time. It is good also as a structure which can be performed.
Hereinafter, each process shown in FIG. 12 will be described in detail.

FIG. 13 is a flowchart showing the reference data generation process shown in step ST1 of FIG.
The control unit 140 displays an image for guiding the line of sight in the horizontal direction and the vertical direction on the virtual image display device 100 (step ST11). The line of sight is moved to the user according to this guidance (step ST12). The control unit 140 acquires captured image data obtained by causing the inner cameras 37a, 37b, 38a, and 38b to perform imaging during or after the movement, and analyzes the acquired captured image data, thereby moving the eye movement and the direction of the eyeball. The data regarding is acquired (step ST13). The control unit 140 determines whether or not the processing of steps ST11 to ST13 has been completed for all the preset directions (step ST14). If there is an unprocessed direction (step ST14; No), the control unit 140 determines whether or not the processing has been completed. Return to.

In step ST11, for example, characters that guide the user to direct the line of sight by designating 8 directions of up, down, right, left, upper right, lower right, upper left, lower left, and 9 directions including the front, Display an image. The controller 140 guides the user so that the line of sight is directed to the specified direction to the maximum extent in the eight directions except for the front.
In step ST13, the control unit 140 extracts an image of the user's eyeball from the captured images of the inner cameras 37a, 37b, 38a, and 38b, and (1) the interpupillary distance, (2) the pupil diameter or the amount of change in the pupil diameter. (3) gaze direction is detected, and detection data is generated. The pupil diameter can be obtained by detecting an iris image of the eyeball. In general, since the iris state reflects the movement of the ciliary muscle that changes the crystalline lens, the change in the pupil diameter or the data relating to the pupil diameter can be used as data relating to the gaze distance.

  Operation | movement of step ST11-ST13 should just be performed about each of the said 9 directions in the surface facing a user's eyeball. That is, in step ST11, a character or image for guiding one of the nine directions to designate a line of sight is displayed, and in step ST13, detection data regarding the designated direction is acquired. If this operation is executed nine times, for example, detection data can be acquired with the line of sight directed in nine directions. Further, in step ST11, the nine directions in which the line of sight is directed and the order in which the line of sight is directed are designated, and thereafter, the movement of the eyeball is continuously imaged by the inner cameras 37a, 37b, 38a, 38b, and the detection data in the nine directions is obtained in step ST13 You may collect all at once.

  When processing has been performed for all of the horizontal and vertical directions (step ST14; Yes), the control unit 140 displays an image for guiding the line of sight in the depth (perspective) direction on the virtual image display device 100 (step ST15). Then, the user moves his / her line of sight according to the guidance (step ST16), acquires captured image data obtained by the inner cameras 37a, 37b, 38a, and 38b performing imaging during or after the movement, and acquires the captured image data. By analyzing, the data regarding the eye movement and the direction of the eye is acquired (step ST17). In step ST15, for example, guidance is given so that the line of sight is directed in two directions: the closest position and the farthest position. You may acquire the detection data regarding the nearest position and the farthest position by performing the process of steps ST15-ST17 twice.

  In step ST15, the user may be guided to see a preset distance in the outside scene that is visible to the user, that is, in the real space. In this case, an object located at the preset distance in the real space is designated. Alternatively, guidance may be given so as to watch the object. In this case, the user may place an object to be used for performing the reference data generation process at a specified distance in advance.

  The control unit 140 generates reference data 122 used for detecting the gaze direction based on the detection data acquired in steps ST13 and ST17, and stores the reference data 122 in the storage unit 120 (step ST18). In this example, the reference data 122 includes data relating to eye movement and / or direction when the user turns his / her line of sight in a total of 11 directions including the 9 directions and the two perspective directions. Thereby, the eye direction is imaged by the inner cameras 37a, 37b, 38a, 38b, and the gaze direction can be specified by comparing the image of the eyeball in the captured image with the state of the eyeball when the nine directions are captured. Further, the gaze distance can be obtained from the direction of the eyeball.

FIG. 14 is a flowchart showing the convergence angle detection process in step ST2 of FIG.
The control unit 140 obtains captured image data by executing imaging with the inner cameras 37a, 37b, 38a, and 38b, and obtains data relating to the state of the eyeball by analyzing the captured image data (step ST31). The data acquired in step ST31 is, for example, data obtained by the same processing as in steps ST13 and ST17, and is preferably data that can be compared with the reference data 122.

The control unit 140 refers to the reference data 122 and calculates the gaze direction and gaze distance of the user based on the data acquired in step ST31 (step ST32).
The control unit 140 executes imaging with the outer cameras 35 and 36, acquires captured image data, and detects an object OB that appears in the captured image (step ST33). The control unit 140 detects an object that can be identified as an object (including a part of the object) in the captured image and that matches the gaze direction and gaze distance obtained in step ST32 as the object OB.

  The controller 140 calculates the distance to the object detected in step ST33 (step ST34). Since the outer cameras 35 and 36 constitute a stereo camera, the distance from the outer cameras 35 and 36 to the target can be obtained based on the parallax between the captured image of the outer camera 35 and the captured image of the outer camera 36. Here, the control unit 140 may convert the obtained distance into a distance from the user's eyes to the object.

  The controller 140 calculates a convergence angle based on the gaze direction calculated in step ST32 and the distance calculated in step ST34 (step ST35). In step ST35, since the gaze distance is obtained from the captured images of the outer cameras 35 and 36, there is an advantage that the gaze distance can be obtained with high accuracy by simpler calculation processing. Note that the operation of step ST34 may be omitted, and the convergence angle may be calculated in step ST35 using the gaze direction and gaze distance calculated in step ST32.

  Note that the specific method for detecting the convergence angle is not limited to the method using a stereo camera as described in FIG. 14, and for example, performing a plurality of times of imaging with a time difference using a monocular camera, The convergence angle may be detected using a plurality of captured images obtained by the plurality of times of imaging. Alternatively, the convergence angle may be detected by imaging the user's eyes (one eye or both eyes) and detecting a change in pupil diameter or pupil diameter. In general, it is known that the diameter of a human pupil expands in a short time when an object to be watched is visually recognized in an outside scene (real space). For this reason, you may utilize the function which detects the change of a pupil diameter, or a pupil diameter in the process which calculates | requires a user's gaze direction. In addition, the control unit 140 may use, for example, a look-up table (conversion table) in the process of obtaining the gaze distance from the captured image. In this case, the control unit 140 can obtain the gaze distance by referring to the lookup table based on the detection data obtained from the camera or the captured image. Moreover, you may combine the distance between eyes (interocular distance) as detection data. The lookup table may be stored in the storage unit 120 in advance, for example.

FIG. 15 is a flowchart showing the image adjustment processing in step ST3 of FIG.
The control unit 140 determines whether or not the convergence angle obtained by the operation of FIG. 14 is within the adjustable range (step ST51). In the HMD 1, there is a limit value of the convergence angle that can be adjusted by mechanical adjustment using the right optical system driving unit 85 and the left optical system driving unit 87 and adjustment by which the image processing unit 160 performs image processing. An adjustable range corresponding to is preset. When determining that the value of the convergence angle is within the adjustable range (step ST51; Yes), the control unit 140 determines whether mechanical adjustment is required (step ST52).

  The right optical system driving unit 85 and the left optical system driving unit 87 have a configuration in which, for example, adjustment is performed in stages, and the display position of the image can be moved greatly by swinging the optical device unit 130. For this reason, when the image display position is moved largely according to the convergence angle, it is efficient to perform mechanical adjustment and use only image processing when the image display position adjustment amount is small. In step ST52, whether to perform mechanical adjustment is determined using a preset threshold value.

When determining that the mechanical adjustment is not required (step ST52; No), the control unit 140 proceeds to step ST56 described later.
If it is determined that mechanical adjustment is required (step ST52; Yes), the control unit 140 determines adjustment amounts of the right optical system driving unit 85 and the left optical system driving unit 87 (step ST53). The control unit 140 operates the right optical system driving unit 85 and the left optical system driving unit 87 according to the adjustment amount determined in step ST53 (step ST54), and determines whether or not the target accuracy is satisfied after the adjustment (step ST55). ).

  As described above, in the HMD 1, the accuracy of image position adjustment in accordance with the convergence angle is set to ± 1 ° for the convergence angle, for example. When the convergence angle detected by the convergence angle detection process of FIG. 14 and the convergence angle that matches the display position of the adjusted image satisfy the set accuracy (step ST55; Yes), the control unit 140 performs this process. finish.

If the adjustment accuracy is not satisfied (step ST55; No), the control unit 140 determines an adjustment amount for image quality adjustment (step ST56) and executes the adjustment (step ST57). Control unit 140 determines whether or not the target accuracy is satisfied by adjustment (step ST58). The target accuracy determined in step ST58 is the same as that in step ST55, for example. When the target accuracy is not satisfied (step ST58; No), the control unit 140 returns to step ST56 and repeats the image processing. Note that an upper limit may be set for the number of executions of steps ST56 to ST57, and when the upper limit is reached, the process may return to step ST51.
Moreover, when it determines with satisfy | filling target accuracy (step ST58; Yes), the control part 140 complete | finishes this process.

  When determining whether or not the target accuracy is satisfied in step ST55, the control unit 140 can determine whether or not the adjustment satisfies the target accuracy when the adjustment amount of the mechanical adjustment is determined. For this reason, you may perform determination of step ST55 by step ST54.

Further, when it is determined that the convergence angle obtained by the operation of FIG. 14 is not within the adjustable range (step ST51; No), the control unit 140 moves to a display mode (display form) corresponding to the convergence angle outside the adjustment range. Are switched (step ST59).
In step ST59, for example, the control unit 140 stops the display of one of the first display device 100A and the second display device 100B, and switches to a state in which only one is displayed. When a single image is viewed with only one eye, it is not affected by the convergence angle. Therefore, even if the display position of the image is not adjusted to a position suitable for the convergence angle, there is no burden on the visual function of the user.

  In step ST59, the control unit 140 may perform VR (Virtual Reality) display. In the VR display, the captured images of the outer cameras 35 and 36 are displayed in the maximum size in the displayable areas of the first display device 100A and the second display device 100B. Here, the brightness of the image display device 80 may be increased, and the intensity of the external light transmitted through the light guide member 10 may be relatively decreased to reduce the visibility of the outside scene. In VR display, a region including the target detected in step ST33 of FIG. 14 may be cut out from the captured images of the outer cameras 35 and 36 and displayed. In this case, since the user visually recognizes the object by looking at the display image, the user is not easily affected by the difference in the convergence angle between the outside scene and the display image. For this reason, the load on the user's visual function can be reduced.

  Furthermore, when performing VR display, the control unit 140 converts the image displayed on the first display device 100A and the image displayed on the second display device 100B according to the gaze distance calculated in step ST32 or ST34 of FIG. You may give parallax. That is, two images having parallax may be generated from the captured images of the outer cameras 35 and 36, and these two images may be displayed on the first display device 100A and the second display device 100B, respectively. These two images constitute a 3D (stereoscopic) image. Therefore, the user can see the object in VR by the stereoscopic image.

  In the operations shown in FIGS. 12 to 15, arithmetic expressions, functions, matrices, parameters, and the like used for the arithmetic processing are all stored in advance as setting data 121 in the storage unit 120. In addition, the reference value and threshold value relating to the adjustable range that is the determination criterion in step ST51 of FIG. 15, the reference value and threshold value that determine the target accuracy, and the like are stored in the storage unit 120 as the adjustment reference data 123. The setting data 121 and the adjustment reference data 123 may be set or updated by operating the control device 300, or may be input from an external device (not shown) connected via the communication unit 117.

  As described above, the HMD 1 according to the embodiment to which the present invention is applied has the virtual image display device 100 that displays an image corresponding to each of the right eye and the left eye of the user so that the outside scene can be visually recognized. The HMD 1 includes a control unit 140 that controls the display mode of the image by the virtual image display device 100 according to the gaze distance of the user. For this reason, when a user visually recognizes an outside scene and an image, the display mode of an image can be adapted to the distance which a user gazes at. For this reason, a display image can be adjusted appropriately, for example, a user's load can be reduced and the visibility of a display image can be improved.

  The control unit 140 controls the display mode of the image by the virtual image display device 100 based on the convergence angle of both eyes corresponding to the gaze distance of the user. For this reason, the display mode suitable for the distance which a user gazes is realizable by controlling the display mode of an image according to the convergence angle of a user's both eyes.

  Further, the control unit 140 obtains the convergence angle of both eyes of the user, and changes the position at which the user visually recognizes the image displayed by the virtual image display device 100 in accordance with the obtained convergence angle. Therefore, an image can be displayed at a position suitable for the user's gaze distance, the visibility of the display image can be improved, and the user can visually recognize a display image suitable for the distance that the user gazes in the outside scene. Can do.

  Further, the HMD 1 includes outer cameras 35 and 36 and inner cameras 37a, 37b, 38a, and 38b as state detection units that detect the state of both eyes of the user. The control unit 140 calculates the gaze distance of the user based on the detection result of the state detection unit, and controls the display mode of the image by the virtual image display device 100 according to the calculated gaze distance. Since the HMD 1 detects the state of both eyes of the user and calculates the gaze distance, the display mode of the image can be adjusted appropriately without the user performing a complicated operation.

  The HMD 1 includes inner cameras 37a, 37b, 38a, and 38b that are provided corresponding to the right eye and the left eye of the user, respectively, as a state detection unit. The control unit 140 is used by obtaining at least one of the distance between the pupils of both eyes of the user, the pupil diameter, and the gaze time in the gaze direction based on the captured images of the inner cameras 37a, 37b, 38a, and 38b. The gaze distance of the person is calculated. Thereby, the gaze distance can be calculated with high accuracy from the state of both eyes of the user.

  Moreover, HMD1 is provided with the outer cameras 35 and 36 which image a user's gaze direction as a state detection part. The control unit 140 calculates the gaze distance of the user by calculating the gaze distance of the user by obtaining the distance to the gaze target object that the user gazes on the basis of the captured images of the outer cameras 35 and 36. It can be calculated.

  The virtual image display device 100 includes an image display device 80 that emits image light, and a projection fluoroscopy device 70 that guides the image light emitted from the image display device 80 to each of the right eye and the left eye of the user. The virtual image display device 100 displaces at least a part of the members constituting the image display device 80 or the projection fluoroscopy device 70, and changes the angle of image light incident on the right and left eyes of the user from the projection fluoroscopy device 70. The right optical system drive unit 85 and the left optical system drive unit 87 are provided. Since the control unit 140 operates the right optical system driving unit 85 and the left optical system driving unit 87 according to the gaze distance of the user, the display mode of the image can be adjusted by a mechanical operation.

  Further, the control unit 140 controls the position of the image in the display range where the user visually recognizes the image displayed by the virtual image display device 100 according to the gaze distance of the user. Thereby, the display position of an image can be appropriately adjusted according to a user's gaze distance.

  The HMD 1 also includes an image processing unit 160 that generates an image to be displayed by the virtual image display device 100 based on the image data. In accordance with the gaze distance of the user, the control unit 140 causes the image processing unit 160 to cut out part of the image data to generate an adjustment image, and causes the virtual image display device 100 to display the adjustment image. Thereby, by cutting out the image, the adjustment range of the display position of the image can be enlarged, and the uncomfortable feeling when changing the display position of the image can be eliminated.

  In addition, the control unit 140 controls the virtual image display device 100 to display either the right eye or the left eye of the user when the gaze distance of the user is closer than a preset reference distance and the convergence angle is not within the adjustable range. Display the corresponding image. Thereby, even if it is a case where a user's gaze distance is near, a user's burden can be eased.

[Second Embodiment]
FIG. 16 is a flowchart showing the operation of the HMD 1 in the second embodiment to which the present invention is applied.
In the second embodiment, the HMD 1 has the same configuration as that of the first embodiment. Therefore, the same reference numerals are assigned to the common components, and illustration and description thereof are omitted.

  The HMD 1 of the second embodiment executes a captured image display mode in which captured images of the outer cameras 35 and 36 are displayed by the virtual image display device 100. The HMD 1 can execute three operation modes as a captured image display mode: one is a microscope mode (near region enlargement mode), one is a telescope mode (far region enlargement mode), and one is a proximity display Mode. The captured image display mode is a kind of video see-through display.

The operation of FIG. 16 is executed instead of the operation shown in FIG. In FIG. 16, the same step numbers are assigned to the same operations (processes) as in FIG.
The control unit 140 of the HMD 1 generates reference data serving as a reference for detecting the user's gaze direction (gaze direction) and gaze distance (step ST1). The reference data generated in step ST1 is stored in the storage unit 120 as reference data 122.
The control unit 140 executes a convergence angle detection process for detecting a convergence angle of both eyes of the user after starting display of an image (including video) or at the start of display (step ST2).

Here, control section 140 determines whether or not zoom (enlargement) instruction has been received by input information acquisition section 110 (step ST101). The zoom instruction is, for example, an instruction that the user operates the track pad 302 or the key switch unit 303 to enlarge and display the field of view.
When a zoom instruction is accepted (step ST101; Yes), the control unit 140 determines whether or not the instructed zoom magnification (enlargement ratio) is equal to or greater than a preset value (step ST102). If the zoom magnification is greater than or equal to the set value (step ST102; Yes), the control unit 140 switches the operation of the HMD1 to the microscope mode (step ST103) and executes the microscope display process (step ST104). Details of the microscope display process executed by the HMD 1 in the microscope mode will be described later.

  If the zoom magnification is smaller than the set value (step ST102; No), the control unit 140 switches the operation of the HMD1 to the telescope mode (step ST106) and executes a telescope display process (step ST107). Details of the telescope display process executed by the HMD 1 in the telescope mode will be described later.

When the zoom instruction has not been received (step ST101; No), the control unit 140 determines whether or not the input information acquisition unit 110 has received an operation mode switching instruction (step ST109). The operation mode switching instruction is input by, for example, an operation of the operation unit 111 and is received by the input information acquisition unit 110.
When receiving an instruction to switch to the microscope mode in step ST109, the control unit 140 proceeds to step ST103. When receiving an instruction to switch to the telescope mode, the control unit 140 proceeds to step ST106. In addition, when an instruction to switch to the normal operation mode is received, and when switching to the telescope mode or the microscope mode is not performed, the control unit 140 executes an image adjustment process (step ST110).

  After executing the microscope display process in step ST104, the telescope display process in step ST107, or the image adjustment process in step ST110, the control unit 140 determines whether or not to end the image display (step ST4). . While the display is continued (step ST4; No), the process returns to step ST2. When the display is to be ended (step ST4; Yes), this process is ended.

  FIG. 17 is a flowchart showing in detail the microscope display process in step ST104 (FIG. 16). FIG. 18 is a diagram illustrating a display example in the microscope display process. FIG. 18A schematically shows an image of a real space (real image) that is viewed through the virtual image display device 100, and FIG. 18B shows an example of processing for generating a display image from the captured image. FIG. 18C shows an example of AR display.

  The control unit 140 sets the imaging direction of the outer cameras 35 and 35 when switched to the microscope mode in step ST103 as the imaging direction for imaging by the following operation (step ST121), and starts the stabilization process (step ST121). ST122). The control unit 140 starts imaging of the outer cameras 35 and 36, acquires captured image data of the outer cameras 35 and 36, and starts processing for generating a display image from the captured images (step ST123).

  In step ST123, the control unit 140 trims (cuts out) a part of the captured image, for example, to generate a display image. In the microscope display process, the image may be captured by only one of the outer camera 35 and the outer camera 36. In this case, a display image can be generated by trimming a part of the image captured by the outer camera 35 or the outer camera 36. When imaging is performed by both the outer camera 35 and the outer camera 36, the control unit 140 trims the captured images of the outer camera 35 and the outer camera 36 to acquire two images, and combines these two images. To generate a display image. For example, a stereoscopic display image composed of two display images having parallax may be generated from two images. Further, for example, one display image obtained by combining two images may be generated.

  As shown in FIG. 18 (A), in a state where the user focuses on the object OB, the captured images P11 of the outer cameras 35 and 36 center on the object OB as shown in FIG. 18 (B). This is a captured image. The captured image P11 is a captured image of one of the outer cameras 35 and 36. The control unit 140 cuts out a trimming range A corresponding to a predetermined range from the center of the line of sight from the captured image P11 to generate a display image. The position and range or size of the trimming range A in the captured image P11 can be set and changed as appropriate, but the initial value (default) is a range of a predetermined size located at the center of the captured image P11.

  When the virtual image display device 100 moves, the angle of view of the outer cameras 35 and 36 moves, and the captured image moves. If the position and range to be trimmed in the captured image are fixed in step ST123, the display image moves in accordance with the movement of the captured image. Depending on the movement of the virtual image display device 100, there is a possibility that a so-called video sickness is aroused to the user. Therefore, the HMD 1 starts the stabilization process in step ST122. The stabilization process is a process of moving the position and / or range for trimming the captured image in step ST123 so as to cancel (cancel) the motion of the captured image. For example, the movement can be canceled by changing the position of the trimming range A cut out from the captured image P11 in FIG. 18B in the direction opposite to the direction in which the angle of view of the outer cameras 35 and 36 moves. Further, when the outer cameras 35 and 36 move in the depth direction of the field of view, the size of the object to be imaged and the captured image changes. In this case, the control unit 140 can cancel the movement by changing the size of the trimming range A.

  The control unit 140 can detect and specify the movement of the captured image by a process of comparing a plurality of captured images sequentially input from the outer cameras 35 and 36. In the configuration in which the virtual image display device 100 includes an inertia sensor (motion sensor) (not shown) such as an acceleration sensor or an angular velocity sensor, the control unit 140 adjusts the angle of view of the outer cameras 35 and 36 based on the detection value of the inertia sensor. Can identify movement. The control unit 140 stabilizes the display image by changing the trimming position of the captured image in accordance with the movement of the angle of view of the outer cameras 35 and 36 and changing the trimming range as necessary.

  The stabilization process may not be a process for canceling all changes in the angle of view of the outer cameras 35 and 36. That is, since it is only necessary to prevent or reduce video sickness, a method of canceling a motion exceeding a predetermined speed among the motions of the angle of view can be employed. In this case, the low-speed movement below the predetermined speed is not canceled, and the display image changes corresponding to the movement of the angle of view. Thereby, for example, when the user intends to change the angle of view of the outer cameras 35 and 36, display according to the user's intention can be performed.

When starting display of the display image generated from the captured image, the control unit 140 performs display indicating the correspondence between the display range (display range) and the real space (step ST124). For example, as illustrated in FIG. 18C, the control unit 140 displays the index image P12 in an AR display so that the user can see the image superimposed on the outside scene of the real space visually recognized by the external light. The index image P12 indicates a range included in the display image displayed by the control unit 140, and corresponds to a range in which the control unit 140 trims the captured image. The index image P12 in the example of FIG. 18C has a ring shape, but it is sufficient that the range can be presented to the user, and the shape and display color are not limited. In addition, it is preferable that the index image P11 does not hinder the visual recognition of the real space. For example, the index image P11 is displayed with a luminance that allows the object OB in the real space to be visually recognized. In this case, the index image P11 is clearly seen and the object OB in real space can be visually recognized together with the index image P11.
In the example of FIG. 18C, the index image P12 is displayed so as to overlap the real object (object) OB in the real space visually recognized by the external light, and the user has a range where the index image P12 overlaps the outer camera 35. , 36 can be grasped based on the captured images. The display position and display size of the AR display can be determined by the outside image analysis unit 183 analyzing the images captured by the outside cameras 35 and 36.

The control unit 140 displays a display image obtained by enlarging the captured image with the initial value magnification before displaying the image with the designated zoom magnification (step ST125). The magnification of the initial value can be, for example, a magnification that is visually recognized by the same size as the outside scene that is visible to the user through the first display device 100A and the second display device 100B. In this case, since the object OB in the outside scene in the real space and the object OB in the display image appear to be almost the same size and do not expand or contract, the user can increase the magnification in this case to 1. Good.
Assume that the user visually recognizes an object located in the field of view by the external light transmitted through the first display device 100A and the second display device 100B. An image of the object displayed by the control unit 140 based on a captured image obtained by capturing the range including the object with the outer cameras 35 and 36 and an image (real image) of the object visually recognized by the external light are presented to the user. The case where they appear to be approximately the same size can be referred to as the above one.

Next, the control unit 140 changes the magnification of the display image displayed by the virtual image display device 100 (step ST126). The controller 140 changes the zoom value from the initial zoom value in step ST125 to the target magnification. This change can be, for example, a stepwise change. For example, a change rate is set in advance for a change in magnification, and the magnification can be changed with the set change rate as one step. The target magnification is preset as a magnification corresponding to the instruction received in step ST101 (FIG. 16), a magnification specified when the microscope mode is specified in step ST109, or a target magnification in the microscope mode. Magnification.
When executing step ST126 for the first time, the control unit 140 changes the magnification corresponding to one step from the magnification of the initial value of step ST125.

Control unit 140 changes the display parameter related to visibility in response to the change in magnification (step ST127). The display parameters related to visibility are a parameter for specifying the display luminance in the video display element 82, a parameter for specifying the display color, and a parameter for specifying the presence / absence of edge enhancement processing. The process of changing the display brightness in the video display element 82 is a process in which the image processing unit 160 changes the brightness of the image data of the display image and / or a process of changing the light emission brightness of the right backlight 221 and the left backlight 222. Can be realized.
Further, the display color can be changed by processing image data by the image processing unit 160. The edge enhancement process can be executed by the image processing unit 160. The above parameters can be included in the setting data 121 stored in the storage unit 120, for example. The above parameters are set in advance corresponding to the absolute value of the magnification or the ratio of the current magnification to the target magnification. The higher the brightness of the display image is, the higher the visibility of the display image is compared to an outside scene in real space that is visually recognized by external light. As for the color tone of the display image, the higher the saturation, the higher the visibility of the display image. In addition, when the edge enhancement process is executed, the visibility of the display image becomes higher than when the edge enhancement process is not executed. The parameter is set so that the higher the magnification of the display image, the higher the visibility of the display image. In step ST127, the control unit 140 sets or changes the parameter corresponding to the magnification changed in step ST126.

  Further, the display parameter related to visibility is not limited to the configuration included in the setting data 121. That is, the control unit 140 may acquire, generate, or calculate a display parameter by other methods as well as a method of acquiring the display parameter related to visibility from the setting data 121.

  For example, the control unit 140 may detect the brightness and color tone of the outside scene from the captured images of the outer cameras 35 and 36, the brightness of the captured image, the chromaticity distribution, and the like, and acquire the display parameter based on the detection result. In this case, a LUT (LookUp Table) for obtaining a display parameter from the detection result, an arithmetic expression, or the like may be stored in the storage unit 120 in advance. The control unit 140 may detect the brightness of the outside scene based on the captured image of the outside scene, determine the weather of the outside scene, and brightness, tone, and color obtained from the captured images of the outside cameras 35 and 36. The degree distribution may be compared with a preset threshold value to generate a comparison result. Further, in addition to the external cameras 35 and 36, an image sensor (not shown) or an infrared sensor may be provided in the virtual image display device 100 to detect the brightness, color tone, and the like in a range including the external environment, particularly the user's line-of-sight direction. . Furthermore, by comparing the detection results regarding the captured images of the outer cameras 35 and 36 with the detection results of the inner cameras 37a, 37b, 38a, and 38b, the brightness and color tone of the outside scene, the brightness of the captured image, the chromaticity distribution, and the like can be obtained. It may be detected.

  The control unit 140 determines whether or not the target magnification has been reached as a result of changing the magnification in step ST126 (step ST128). If the target magnification has not been reached (step ST128; No), the process proceeds to step ST126 to change the magnification. By repeatedly executing steps ST126 to ST128 in this manner, the display magnification in the virtual image display device 100 gradually changes until the target magnification is reached. The degree of change of the magnification can be arbitrarily set. For example, the magnification may be changed in units of a preset magnification such as 0.5 times, or the magnification may be changed once based on the difference between the initial value and the target value. The amount of change may be determined.

When the target magnification is reached (step ST128; Yes), the control unit 140 determines whether or not the target magnification has been changed (step ST129). For example, when the input information acquisition unit 110 receives a user operation to change the target magnification, the control unit 140 determines that the target magnification has been changed (step ST129; Yes). In this case, based on the changed target magnification, control unit 140 returns to step ST126 and starts a process of changing the display magnification.
When the target magnification is not changed (step ST129; No), the control unit 140 determines whether or not to end the image display (step ST130). While the display is continued (step ST130; No), the process returns to step ST129. When the display is to be ended (step ST130; Yes), this process is ended.

In the microscope display process of FIG. 17, the direction specified in step ST121 out of the range or direction included in the user's field of view can be enlarged and displayed as if it were a microscope or a magnifying glass.
The microscope mode is executed when a zoom magnification greater than or equal to the set value is designated in step ST102 (FIG. 16), and when a transition to the microscope mode is instructed in step ST109. Therefore, when the user wants to greatly enlarge a part of the field of view, the user can easily switch to the microscope mode by operating the operation unit 111 and can view the field of view with a greatly enlarged view.

  In the microscope display process, the image displayed in steps ST125 to ST126 can be displayed as a stereoscopic image having parallax. In the control unit 140, the image processing unit 160 converts the parallax between an image displayed on the first display device 100A corresponding to the right eye of the user and an image displayed on the second display device 100B corresponding to the left eye. Cause it to occur. In this case, in step ST127, the control unit 140 may change and set a parameter that specifies the parallax between the left and right images. In this case, an image generated based on the captured images of the outer cameras 35 and 36 is displayed as an image having parallax, and an object for AR display in the real space is displayed as parallax, for example, as an index image P12 in FIG. You may display as a plane image which does not have. Not only the index image P12 but also, for example, an image having no parallax as a guidance object based on text or an image in the AR content displayed by the control unit 140 for the purpose of guidance, instruction, and other information transmission to the user. May be displayed.

  In addition, when an image having parallax is displayed before shifting to the microscope mode, the control unit 140 may switch to a display mode having no left and right parallax when shifting to the microscope mode. In this case, the control unit 140 displays an image generated based on the captured images of the outer cameras 35 and 36 and an object such as text or an image included in the AR content as an image having no parallax.

  The control unit 140 starts the stabilization process in step ST122. In the stabilization process, when the imaging direction of the outer cameras 35 and 36 is changed, the position where the display image is cut out is moved in a direction to cancel or compensate for the change in the captured image. For this reason, even if the imaging direction of the outer cameras 35 and 36 changes, the display image generated from the captured image does not vary greatly. In addition, when the movement of the outer cameras 35 and 36 in the imaging direction, that is, when the change in the captured image is large, the display image cut out from the captured image also changes. Therefore, the motion sickness can be suppressed by following the change in the imaging direction of the outer cameras 35 and 36 and suppressing a sudden change in the display image.

In the microscope display process, the control unit 140 displays a display image generated from the captured images of the outer cameras 35 and 36 on either the first display device 100A or the second display device 100B, and on the other side. It is good also as a structure which does not display. This display method is called single-eye display. For example, when the display magnification exceeds the threshold set in advance by changing the display magnification in step ST126, the control unit 140 may switch to the one-eye display. In addition, when it is determined that the target magnification is a magnification exceeding the threshold value, switching to the one-eye display may be performed. Alternatively, a parameter for enhancing the visibility of the display image is set in step ST127, and when the value of this parameter exceeds a preset threshold value, one-eye display may be performed. During the execution of the one-eye display, the user can visually recognize the real space with the eyes on the side not displaying the image. Thereby, the visibility of the real space is ensured with the eye of one b, so that the visibility of the display image based on the captured image can be greatly enhanced with respect to the other eye.
Further, during the one-eye display, the AR display may be performed by superimposing on the real space on the side of the first display device 100A and the second display device 100B that does not display the display image based on the captured image.

FIG. 19 is a flowchart showing in detail the telescope display process of step ST107 (FIG. 16).
The telescope display process shown in FIG. 19 includes the same process as the microscope display process shown in FIG. Therefore, the same step number is assigned to the same process.

  The control unit 140 sets the imaging direction of the outer cameras 35 and 35 when switched to the telescope mode in step ST106 as the imaging direction for imaging by the following operation (step ST121), and starts the stabilization process (step ST121). ST122). The control unit 140 starts imaging of the outer cameras 35 and 36, acquires captured image data of the outer cameras 35 and 36, and starts processing for generating a display image from the captured images (step ST123). The stabilization process is the same as the process described with reference to FIG.

  The control unit 140 does not perform the display in step ST124, but displays a display image obtained by enlarging the captured image at the initial value magnification (step ST125), and changes the display magnification (step ST126). The controller 140 changes the zoom value from the initial zoom value in step ST125 to the target magnification. This change can be, for example, a stepwise change. For example, a change rate is set in advance for a change in magnification, and the magnification can be changed with the set change rate as one step. For example, when executing step ST126 for the first time, the control unit 140 changes the magnification corresponding to one step from the magnification of the initial value in step ST125. The subsequent change in magnification is as described with reference to FIG.

Control unit 140 changes the display parameter related to visibility in response to the change in magnification (step ST127). The display parameters related to visibility are as described above for step ST127.
The telescope display process is suitable when the user enlarges and looks far away like a binocular or a telescope. In the microscope display process, it is assumed that a position close to the virtual image display device 100 is enlarged and displayed, whereas in the telescope display process, it is preferable to assume that the user looks far away. For this reason, it is preferable to maintain a state in which the user can visually recognize the real space with the external light transmitted through the virtual image display device 100. Accordingly, in step ST127 in the telescope display process of FIG. 19, the visibility of the real space is set when the visibility of the display image displayed based on the captured images of the outer cameras 35 and 36 is higher than the visibility of the real space. The parameter is set so as to secure

  Specifically, values different from the microscope display process described with reference to FIG. 17 are set in advance for the parameters set in step ST127. Further, the upper limit value or the lower limit value of the parameter set by control unit 140 in step ST127 is set to a value different from that in the microscope display process. Thereby, in the telescope display processing of FIG. 19, a field of view in real space that can be visually recognized by the transmitted light of the virtual image display device 100 is secured.

  Thereafter, the control unit 140 determines whether or not the target magnification has been reached as a result of changing the magnification in step ST126 (step ST128). If the target magnification has not been reached (step ST128; No), step Moving to ST126, the magnification is changed. The control unit 140 repeatedly executes steps ST126 to ST128, and gradually changes the display magnification in the virtual image display device 100 to the target magnification.

  When the target magnification is reached (step ST128; Yes), the control unit 140 determines whether or not the target magnification has been changed (step ST129). When it is determined that the target magnification has been changed (step ST129; Yes), the control unit 140 returns to step ST126 and starts a process of changing the display magnification based on the changed target magnification. When the target magnification is not changed (step ST129; No), the control unit 140 determines whether or not to end the image display (step ST130). While the display is continued (step ST130; No), the process returns to step ST129. When the display is to be ended (step ST130; Yes), this process is ended.

  The telescope display process of FIG. 19 is executed when the transition to the telescope mode is instructed in step ST109. Therefore, when the user wants to greatly enlarge a part of the field of view, the user can easily switch to the microscope mode by operating the operation unit 111 and can view the field of view with a greatly enlarged view. Further, the enlarged display range in the telescope display process is the center of the field of view when the shift is instructed in step ST109 or a part of the captured image corresponding to the user's line-of-sight direction. In step ST124, the initial value is displayed. Display starts at the display magnification of. Therefore, the visual field does not change abruptly compared to before the telescope display process is executed, and the user can naturally see the enlarged image.

  In the telescope display process, the image displayed in steps ST125 to ST126 can be displayed as a stereoscopic image having parallax. In the control unit 140, the image processing unit 160 converts the parallax between an image displayed on the first display device 100A corresponding to the right eye of the user and an image displayed on the second display device 100B corresponding to the left eye. Cause it to occur. In this case, in step ST127, the control unit 140 may change and set a parameter that specifies the parallax between the left and right images. In addition, the object may be AR displayed in the real space by the telescope display process. For example, text or an image indicating the display magnification may be displayed as an AR. The AR display may be displayed as a planar image having no parallax.

  Further, one-eye display may be performed in the telescope mode. The timing or conditions for performing single-eye display can be the same as those described in the microscope display process.

FIG. 20 is a flowchart showing in detail the image adjustment processing in step ST110 (FIG. 16).
The image adjustment process in FIG. 20 includes the same process as the image adjustment process (FIG. 15) described in the first embodiment, and thus the same step number is assigned to the same process.

The control unit 140 determines whether or not the convergence angle obtained in step ST2 (FIG. 16) is within the adjustable range (step ST51). If it is determined that the value of the convergence angle is within the adjustable range (step ST51; Yes), it is determined whether mechanical adjustment is required (step ST52). When it determines with mechanical adjustment not being required (step ST52; No), the control part 140 transfers to step ST56.
If it is determined that mechanical adjustment is required (step ST52; Yes), the control unit 140 determines adjustment amounts of the right optical system driving unit 85 and the left optical system driving unit 87 (step ST53). The control unit 140 operates the right optical system driving unit 85 and the left optical system driving unit 87 according to the adjustment amount determined in step ST53 (step ST54), and determines whether or not the target accuracy is satisfied after the adjustment (step ST55). ). When the convergence angle detected by the convergence angle detection process in step ST2 and the convergence angle that matches the display position of the adjusted image satisfy the set accuracy (step ST55; Yes), the control unit 140 performs this process. finish.

If the adjustment accuracy is not satisfied (step ST55; No), the control unit 140 determines an adjustment amount for image quality adjustment (step ST56) and executes the adjustment (step ST57). Control unit 140 determines whether or not the target accuracy is satisfied by adjustment (step ST58). The target accuracy determined in step ST58 is the same as that in step ST55, for example. When the target accuracy is not satisfied (step ST58; No), the control unit 140 returns to step ST56 and repeats the image processing. Note that an upper limit may be set for the number of executions of steps ST56 to ST57, and when the upper limit is reached, the process may return to step ST51.
Moreover, when it determines with satisfy | filling target accuracy (step ST58; Yes), the control part 140 complete | finishes this process.

  On the other hand, when it is determined that the convergence angle obtained in the convergence angle detection process in step ST2 (FIG. 16) is not within the adjustable range (step ST51; No), the control unit 140 sets the convergence angle to a preset upper limit. It is determined whether or not the angle is larger (step ST161). If larger than the upper limit angle (step ST161; Yes), the control unit 140 switches to a display mode (display mode) corresponding to the convergence angle outside the adjustment range (step ST59). In step ST59, for example, the control unit 140 stops the display of one of the first display device 100A and the second display device 100B and switches to a state in which only one is displayed. When a single image is viewed with only one eye, it is not affected by the convergence angle. Therefore, even if the display position of the image is not adjusted to a position suitable for the convergence angle, there is no burden on the visual function of the user.

  If it is determined that the convergence angle is equal to or smaller than the upper limit angle (step ST161; No), the control unit 140 switches the operation mode to the proximity display mode and starts the proximity display process (step ST162). The proximity display mode is an operation mode in which a position close to the virtual image display device 100 is displayed in an easy-to-see manner using the captured images of the outer cameras 35 and 36, and the user's line of sight is at a position close to the virtual image display device 100. Run if you are facing.

  The control unit 140 sets the imaging direction of the outer cameras 35 and 35 (step ST163), and starts the stabilization process (step ST164). The control unit 140 starts imaging of the outer cameras 35 and 36, acquires captured image data of the outer cameras 35 and 36, and starts processing to generate a display image from the captured images (step ST165).

  In step ST165, for example, the control unit 140 trims (cuts out) a part of the captured image to generate a display image. Here, the control unit 140 may generate a display image using a captured image of only one of the outer camera 35 and the outer camera 36. In this case, a display image can be generated by trimming a part of the image captured by the outer camera 35 or the outer camera 36. When using the captured images of both the outer camera 35 and the outer camera 36, the control unit 140 trims the captured images of the outer camera 35 and the outer camera 36 to obtain two images, and these two images Are combined to generate a display image. For example, a stereoscopic display image composed of two display images having parallax may be generated from two images. Further, for example, one display image obtained by combining two images may be generated.

  The stabilization process is the same process as step ST122 (FIG. 17). The controller 140 changes the trimming position and range in the captured image so as to cancel or compensate for the motion of the captured image caused by the movement of the angle of view of the outer cameras 35 and 36.

  Control unit 140 starts displaying the display image generated in step ST165 (step ST166). In step ST166, the control unit 140 sets display parameters related to the visibility of the image to be displayed, and performs display adjustment. The parameters to be adjusted are, for example, a parameter that designates display luminance in the video display element 82, a parameter that designates a display color, and a parameter that designates whether or not edge enhancement processing is performed. The process of changing the display brightness in the video display element 82 is a process in which the image processing unit 160 changes the brightness of the image data of the display image and / or a process of changing the light emission brightness of the right backlight 221 and the left backlight 222. Can be realized. Further, the display color can be changed by processing image data by the image processing unit 160. The edge enhancement process can be executed by the image processing unit 160. The above parameters can be included in the setting data 121 stored in the storage unit 120, for example. The above parameters are set in advance corresponding to the absolute value of the magnification or the ratio of the current magnification to the target magnification. The higher the brightness of the display image is, the higher the visibility of the display image is compared to an outside scene in real space that is visually recognized by external light. As for the color tone of the display image, the higher the saturation, the higher the visibility of the display image. In addition, when the edge enhancement process is executed, the visibility of the display image becomes higher than when the edge enhancement process is not executed. The parameter is set so that the higher the magnification of the display image, the higher the visibility of the display image. In step ST166, the control unit 140 sets or changes the parameter corresponding to the convergence angle.

  The position and range where the control unit 140 trims the captured image is determined according to the convergence angle detected in step ST2 (FIG. 16). The user can use the outer cameras 35 and 36 to see a position that is so close that it is not easy to see with the naked eye. Further, according to the convergence angle of the user, the control unit 140 may enlarge and display an image cut out from the captured images of the outer cameras 35 and 36. The magnification in this case may be determined, for example, corresponding to the convergence angle.

  As described above, the HMD 1 according to the second embodiment to which the present invention is applied includes the outer cameras 35 and 36 that capture the outside of the virtual image display device 100, and the control unit 140 is based on the captured images of the outer cameras 35 and 36. A captured image display mode in which an image is displayed on the virtual image display device 100 can be executed. Thereby, the user can visually recognize the outside scene which is the scenery outside the display unit. The captured image display mode corresponds to the above-described microscope mode (near region enlargement mode), telescope mode (far region enlargement mode), and proximity display mode.

  During execution of the captured image display mode, the control unit 140 performs a process of increasing the visibility of the display image displayed based on the captured image, as compared to the real space visually recognized by the external light transmitted through the virtual image display device 100. Thereby, the outside scene visually recognized by the user from the display image can be clearly viewed with higher visibility than the real space visually recognized by the transmitted light of the virtual image display device 100. In addition, since the user can easily distinguish between the real space visually recognized by the transmitted light and the outside scene displayed based on the captured image, the convenience of the user can be improved.

  While executing the captured image display mode, the control unit 140 changes the luminance or color of the display image based on the captured image of the outer camera 35 or 36, or edge enhancement for the image based on the captured image of the outer camera 35 or 36. Process. With these processes, the control unit 140 performs a process of increasing the visibility of the display image as compared to the transmitted light, so that the visibility of the outside scene of the display image can be effectively enhanced.

In the captured image display mode, the control unit 140 causes the first display device 100A and the second display device 100B to transfer an image based on the captured images of the outer cameras 35 and 36 to either the right eye or the left eye of the user. You may display correspondingly. In this case, the display image can be visually recognized by the user without being affected by the user's gaze distance.
In addition, in the captured image display mode, the control unit 140 displays an image based on the captured images of the outer cameras 35 and 36 corresponding to each of the right eye and the left eye of the user in a manner having no left and right parallax. May be. In this case, the user can visually recognize the display image without stereoscopic viewing. For this reason, it is hard to be influenced by a user's gaze distance, and the visibility of a display image can be improved.

The control unit 140 may enlarge and display a display image generated based on the captured images of the outer cameras 35 and 36 in the captured image display mode. Further, the control unit 140 may execute the captured image display mode when the input information acquisition unit 110 receives a user operation.
In addition, the control unit 140 detects the user's gaze distance, and when the detected gaze distance is shorter than a preset distance (for example, when the convergence angle is larger than the set value), the captured image display mode is set. May be executed.

[Third Embodiment]
FIG. 21 is a diagram illustrating the configuration of the HMD in the third embodiment.
In the third embodiment, as illustrated in FIG. 21, the parallax setting unit (for example, refer to the frame unit 102 in FIG. 1) that sets parallax includes an angle variable unit. With this configuration, the crossing angle between the assembly reference direction Asa of the first optical member 101a that is the light guide device 20 and the assembly reference direction ASb of the second optical member 101b can be adjusted. That is, the angle of the second optical member 101b with respect to the first optical member 101a can be changed. Thereby, as shown in the drawing, the value of the angle θ formed by the principal rays PRa and PRb that define the left and right parallax can be changed.

  In the angle change, not only the optical members 101a and 101b (light guide device 20) but also the first and second display devices 100A and 100B are moved integrally. Various configurations are conceivable for the angle variable unit. For example, a piezo element or a micro adjuster that changes the posture of the first and second display devices 100A and 100B symmetrically in a pair of left and right configurations is applied. It is conceivable to operate it precisely. Further, a hinge with a stopper (for example, a hinge shown in FIG. 11) whose angle can be changed continuously or stepwise to the central portion CP of the frame portion 102 (see FIG. 1), which is a parallax setting unit, and can be fixed at a desired angle. It is also conceivable to provide an angle adjusting member for precisely operating a component such as (located at the same position as HN). When changing the postures of the first and second display devices 100A and 100B to a desired state, it is typically considered to use a deformable movable element such as the piezo element or the hinge as the angle variable unit. . In addition, as long as it is a movable element that can be deformed and can be precisely controlled, elements other than those described above may be applied as the angle variable section.

  In addition to the above, in the third embodiment, the optical lenses (not shown) constituting the pair of left and right projection lenses 30 constitute the focus lens group, and this is formed in the optical axis direction by the pair of left and right actuators ACa and ACb. That is, the projection lens 30 has a focus mechanism. That is, the focus mechanism functions as a focal length adjustment unit that adjusts the focal length. Since it becomes possible to adjust the focal length in the projection lens 30, when the value of the angle θ, that is, the parallax is changed by the angle variable unit of the parallax setting unit described above, each display device 100A corresponds to this. , 100B, the assumed display position of the image visually recognized by the video light GL (the assumed position of the virtual image IM) can be changed.

In the configuration of FIG. 21, the operation of changing the angle of the second optical member 101b with respect to the first optical member 101a is mechanical using the right optical system driving unit 85 and the left optical system driving unit 87 under the control of the change control unit 184. Together with the adjustment, it can be performed as part of the mechanical adjustment. That is, the change control unit 184 mechanically adjusts the adjustable amount by the operation of changing the angle of the second optical member 101b with respect to the first optical member 101a using the right optical system driving unit 85 and the left optical system driving unit 87. In addition to the adjustable amount, a mechanically adjustable range can be set. Based on this adjustable range, the change control unit 184 can obtain the adjustment amount of the mechanical adjustment and the adjustment amount of the image processing when the mechanical adjustment and the image processing are combined.
Further, the change control unit 184 adjusts by an operation of changing the adjustment amount of the mechanical adjustment using the optical system driving unit 85 and the left optical system driving unit 87 and the angle of the second optical member 101b with respect to the first optical member 101a. The amount and the adjustment amount by image processing can be obtained individually.

  In step ST53 of FIG. 15 and FIG. 20, the change control unit 184 is a process for obtaining the adjustment amount of the mechanical adjustment, and the adjustment amount of the mechanical adjustment using the optical system driving unit 85 and the left optical system driving unit 87, An adjustment amount by an operation of changing the angle of the second optical member 101b with respect to the first optical member 101a is obtained. Then, in step ST54, according to the obtained adjustment amount, mechanical adjustment using the optical system driving unit 85 and the left optical system driving unit 87, and an operation of changing the angle of the second optical member 101b with respect to the first optical member 101a. It can be performed.

  Thus, the process which adjusts a viewing distance by adjusting the display mode of the image in the virtual image display apparatus 100 is not limited to the structure mentioned above, It can be set as various aspects.

In addition, this invention is not restricted to the structure of said each embodiment, In the range which does not deviate from the summary, it can be implemented in a various aspect.
For example, in the above embodiment, as an example of controlling the display mode according to the gaze distance of the user, the case where the user controls the display position for viewing (viewing) the display image has been described as an example. The invention is not limited to this. For example, when performing AR display, the display size, display color, and the like of an AR image or a display object (image, graphic, character, etc.) included in the AR image may be changed according to the gaze distance. In addition, a display mode in which a display object that is displayed so as to overlap the object OB is displayed so as to overlap in front of the target object OB, and a display mode in which the display object is displayed so as to be overlapped behind the target object OB. You may switch according to a gaze distance. Alternatively, the image processing may be performed on the AR image to switch on / off the display mode in which the display object is visually recognized as being shaded according to the gaze distance.

  Further, as another example of controlling the display mode according to the user's gaze distance, the display mode for displaying the image superimposed on the object OB in the real space that the user sees through the light guide member 10 is controlled. May be. For example, a mask display that reduces the visibility of the target object OB by displaying the image so as to overlap the target object OB, or a display that allows the target object OB to be viewed in a different color depending on the gaze distance. May be switched or adjusted. In this case, the control unit 140 performs image processing such as brightness adjustment, color arrangement, dot removal, sharpness or edge enhancement, gradation adjustment, and focus adjustment (blurring process) on the image that is visually recognized while being superimposed on the object OB. May be. Further, the adjustment of the display position of the image may be performed in combination with the image processing.

  Further, for example, in the above embodiment, the control unit 140 performs the same adjustment in both the first display device 100A and the second display device 100B based on the convergence angle calculated in the convergence angle detection process of FIG. explained. That is, in steps ST53 and ST56 in FIG. 15, the adjustment amount of mechanical adjustment and / or the adjustment amount by image processing is set to the same amount on the left and right. The present invention is not limited to this, and different adjustment amounts may be determined for the dominant eye side and the non-dominant eye side. In this case, the difference in the adjustment amount may follow a preset parameter or the like, or may be obtained from the detection data calculated in step ST13 in FIG.

  Further, for example, the virtual image display device 100 is not limited to a configuration that is held on the user's head by the temple unit 104 and the nose receiving unit 40 similarly to the glasses. For example, a method of wearing like a hat may be adopted, and a display unit that displays an image corresponding to the left eye of the user and a display unit that displays an image corresponding to the right eye of the user They may be separated and attached to the user's head. Moreover, you may mount | wear with a user's body indirectly via glasses, a hat, a helmet, etc. Further, for example, it may be configured as a head mounted display mounted on a vehicle such as an automobile or an airplane. Moreover, for example, it may be configured as a head-mounted display built in a body protective device such as a helmet, or may be a head-up display (HUD) used for a windshield of an automobile.

  Further, in the above embodiment, the virtual image display device 100 and the control device 300 are separated and connected through the connection cable 41 as an example. However, the control device 300 and the virtual image display device 100 are integrated. It is also possible to be configured to be mounted on the user's head. Further, when the control device 300 and the virtual image display device 100 are connected by a longer cable, a notebook computer, a tablet computer, or a desktop computer may be used as the control device 300. The control device 300 may be a game machine, a portable phone, a portable electronic device including a smartphone or a portable media player, other dedicated devices, or the like. Here, when the control device 300 has a display screen, the display screen included in the control device 300 may be stopped in the battery replacement mode.

For example, as a configuration for generating image light in the virtual image display device 100, a configuration including an organic EL (Organic Electro-Luminescence) display and an organic EL control unit may be used. Further, as a configuration for generating image light, a configuration using LCOS (Liquid crystal on silicon, LCoS is a registered trademark) can be used. Moreover, it can also be set as the structure which produces | generates image light using an inorganic EL (Electro-Luminescence) display, an LED array, a digital micromirror device (DMD), etc. FIG.
Further, for example, the present invention can be applied to a laser retinal projection type head mounted display. That is, the image generation unit includes a laser light source and an optical system that guides the laser light source to the user's eye, makes the laser light enter the user's eye, scans the retina, and forms an image on the retina. A configuration that allows the user to visually recognize an image may be employed. When a laser retina projection type head-mounted display is employed, the “image light emitting area in the image light generation unit” can be defined as an image area recognized by the user's eyes.

  As an optical system that guides image light to the user's eyes, an optical member that transmits external light that is incident from the outside toward the apparatus and that enters the user's eyes together with the image light can be employed. Moreover, you may use the optical member which is located ahead of a user's eyes and overlaps a part or all of a user's visual field. Further, a scanning optical system that scans a laser beam or the like to obtain image light may be employed. Further, the optical member is not limited to guiding the image light inside the optical member, and may have only a function of guiding the image light by refracting and / or reflecting it toward the user's eyes.

  Further, the present invention can be applied to a display device that employs a scanning optical system using a MEMS mirror and uses MEMS display technology. That is, as an image display element, a signal light forming unit, a scanning optical system having a MEMS mirror that scans light emitted from the signal light forming unit, and an optical member on which a virtual image is formed by light scanned by the scanning optical system May be provided. In this configuration, the light emitted from the signal light forming unit is reflected by the MEMS mirror, enters the optical member, is guided through the optical member, and reaches the virtual image forming surface. When the MEMS mirror scans the light, a virtual image is formed on the virtual image forming surface, and the user recognizes the virtual image with the eyes, thereby recognizing the image. The optical component in this case may be one that guides light through a plurality of reflections, such as the right light guide plate 261 and the left light guide plate 262 of the above embodiment, and may use a half mirror surface. . In addition, the above-mentioned laser retinal projection type head mounted display is configured in which light is incident on a user's eye using MEMS, scanned on the retina, and imaged on the retina to allow the user to visually recognize the image. May be.

  Further, at least a part of the functional blocks shown in FIG. 9 may be realized by hardware, or may be realized by cooperation of hardware and software, as shown in FIG. It is not limited to a configuration in which independent hardware resources are arranged. The program executed by the control unit 140 may be stored in the storage unit 120 or the storage device in the control device 300, or the program stored in the external device is acquired via the communication unit 117 or the interface 114. It is good also as a structure to execute. Of the components formed in the control device 300, only the operation unit 135 may be formed as a single user interface (UI). Moreover, the structure formed in the control apparatus 300 may be formed in the virtual image display apparatus 100 overlapping. For example, the control unit 140 illustrated in FIG. 9 may be formed in both the control device 300 and the virtual image display device 100, or the control unit 140 formed in the control device 300 and the CPU formed in the virtual image display device 100. The functions performed by and may be configured separately.

  DESCRIPTION OF SYMBOLS 1 ... HMD (display apparatus), 10 ... Light guide member, 10a, 10b ... Position, 35, 36 ... Outer camera (2nd imaging part, external imaging part), 37a, 37b, 38a, 38b ... Inner camera (1st (Imaging unit), 41 ... connection cable, 50 ... light transmission member, 70 ... projection see-through device (optical system), 80 ... image display device (emission unit), 84 ... drive control unit, 85 ... right optical system drive unit (change) Part), 86 ... image element case, 87 ... left optical system drive part (change part), 100 ... virtual image display device (display part), 100A ... first display device, 100B ... second display device, 101a ... first optical Member 101b second optical member 102 frame portion 104 temple portion 105a first image forming body portion 105b second image forming body portion 105d exterior member 107 frame 107a front portion , 107b... Side portion, 107 ... Side part 108 ... Resin part 110 ... Input information acquisition part 114 ... Interface 117 ... Communication part 120 ... Storage part 121 ... Setting data 130 ... Optical device part 135 ... Operation part 140 ... Control part , 160 ... Image processing unit (image generation unit), 170 ... Display control unit, 181 ... Imaging control unit, 182 ... Eye image analysis unit, 183 ... Outside image analysis unit, 184 ... Change control unit, 185 ... Drive control unit, 190 ... Audio processing unit 201 ... Right backlight control unit 202 ... Left backlight control unit 211 ... Right drive control unit 212 ... Left drive control unit 221 ... Right backlight 222 ... Left backlight 261 ... Right light guide plate, 262... Left light guide plate, 300... Control device, 302.

Claims (25)

A display unit that displays an image corresponding to each of the user's right eye and left eye so that the outside scene can be visually recognized;
A control unit that controls the display mode of the image by the display unit according to the gaze distance of the user;
A display device comprising: The control unit is configured to change a display position of the image and / or a display size of the image according to the gaze distance of the user.
The display device according to claim 1. The control unit controls a display mode of an image by the display unit based on a convergence angle of both eyes corresponding to the gaze distance of the user;
The display device according to claim 1 or 2. The control unit obtains the convergence angle of both eyes of the user, and changes the position at which the user visually recognizes the image displayed by the display unit, corresponding to the obtained convergence angle,
The display device according to claim 3. The control unit shifts a display position where the display unit displays the image from a preset standard position in response to a convergence angle of both eyes of the user.
The display device according to claim 4. A state detection unit for detecting the state of both eyes of the user;
The control unit calculates a gaze distance of the user based on a detection result of the state detection unit, and controls a display mode of an image by the display unit according to the calculated gaze distance;
The display device according to claim 1, wherein: The state detection unit includes a first imaging unit provided corresponding to each of the user's right eye and left eye,
The control unit obtains at least one of a distance between pupils of both eyes of the user, a pupil diameter, or a gaze time in a gaze direction based on a captured image of the first imaging unit. Calculating the gaze distance,
The display device according to claim 6. The state detection unit includes a second imaging unit that images the gaze direction of the user,
The control unit calculates a gaze distance of the user by obtaining a distance to a gaze object that the user gazes based on a captured image of the second imaging unit.
The display device according to claim 6 or 7. The display unit includes an emission unit that emits image light;
An optical system that guides the image light emitted by the emission unit to each of the right eye and the left eye of the user;
A displacement unit that displaces at least a part of the light emitting unit or the optical system and changes the angle of image light incident on the right and left eyes of the user from the optical system, and
The control unit operates the changing unit according to the gaze distance of the user;
The display device according to claim 1, wherein: The control unit controls the position of the image in a display range in which the user visually recognizes the image displayed by the display unit according to the gaze distance of the user;
The display device according to claim 1, wherein: An image generation unit that generates an image to be displayed by the display unit based on image data;
The control unit causes the image generation unit to cut out part of the image data to generate an adjustment image according to the user's gaze distance, and to display the adjustment image on the display unit. ,
The display device according to claim 10. The control unit causes the display unit to display an image corresponding to either the right eye or the left eye of the user when the gaze distance of the user is closer than a preset reference distance;
The display device according to claim 1, wherein: A display device that displays an image corresponding to each of a user's right eye and left eye so that the outside scene can be visually recognized,
An exit for emitting image light;
An optical system that guides the image light emitted by the emission unit to each of the right eye and the left eye of the user;
A changing unit that changes an incident angle of image light that is incident on the right eye and the left eye of the user from the emitting direction of the emitting unit or the optical system;
An image generation unit that generates an image to be displayed by the display unit based on image data;
A process of calculating the gaze distance of the user, operating the changing unit according to the calculated gaze distance, and a part of the image data by the image generation unit according to the gaze distance of the user. A control unit that cuts out and generates an adjustment image, and executes at least one of the processes of displaying the adjustment image on the display unit;
A display device comprising: With an external imaging unit,
The control unit is capable of executing a process of displaying an image based on a captured image of the external imaging unit on the display unit;
The display device according to claim 1, wherein: The display unit is configured so that the user can visually recognize an outside scene by external light transmitted through the display unit, and an image based on a captured image of the external imaging unit is displayed so as to be visible simultaneously with the outside scene.
The display device according to claim 14. The control unit, when displaying an image based on the captured image on the display unit, performs a process of increasing the visibility of the image based on the captured image compared to external light transmitted through the display unit;
16. A display device according to claim 14 or 15, wherein: When the control unit displays an image based on the captured image on the display unit, the control unit changes the luminance or color of the image based on the captured image, or performs edge enhancement processing on the image based on the captured image To perform a process of improving the visibility of the image based on the captured image,
The display device according to claim 16. The control unit displays an image based on a captured image of the external imaging unit in correspondence with either the right eye or the left eye of the user by the display unit,
The display device according to claim 14, wherein: The control unit displays an image based on a captured image of the external imaging unit on the display unit in correspondence with each of the right eye and the left eye of the user in a manner that does not have left and right parallax;
The display device according to claim 14, wherein: The control unit enlarges a captured image of the external imaging unit and displays the enlarged image on the display unit;
The display device according to claim 14, wherein: The control unit displays an image based on a captured image of the external imaging unit when an operation of the user is received;
The display device according to claim 14, wherein the display device is a display device. The control unit detects a gaze distance of the user, and displays an image based on a captured image of the external imaging unit when the detected gaze distance is shorter than a preset distance;
The display device according to claim 14, wherein the display device is a display device. The external imaging unit captures an image of a range including the line-of-sight direction of the user when the display unit is mounted;
The display device according to claim 14, wherein: The display device includes a display unit that displays an image corresponding to each of the right eye and the left eye of the user so that the outside scene can be visually recognized, and the display mode of the image by the display unit depends on the gaze distance of the user. Control,
A control method of a display device characterized by the above. A program that can be executed by a computer that controls a display device that includes a display unit that displays an image corresponding to each of the user's right eye and left eye so that the outside scene can be visually recognized,
The program which makes the said computer function as a control part which controls the display mode of the image by the said display part according to the said user's gaze distance.

Images