JP2014071663A - Head-mounted display, method of actuating the same and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve comfort when an observer views a reference content in a monocular type head-mounted display enabling the observer to observe at least one object and at least one reference content in an object space in association with each other.SOLUTION: In a list mode, a plurality of reference contents S and R related to a plurality of objects A and B is displayed so as to exhibit a first depth, respectively, and while the reference contents S and R are displayed, when an observer transitions to a state of gazing at any of the reference content R, in a gazing mode, the depth of the reference content R is changed to a second depth substantially identical to a depth of the object B corresponding to the reference content R from the first depth.

Description

  The present invention relates to a head mounted display that displays an image to an observer while being worn on the observer's head, and in particular, relates at least one object in a target space and at least one reference content to each other. It relates to a technique that enables an observer to observe.

  A head-mounted display (hereinafter abbreviated as “HMD”) that displays an image to an observer by projecting image light onto the observer's eye while being mounted on the observer's head is already known. ing. According to the HMD, an observer can directly observe an image without using a screen on which the image is projected.

  This HMD is classified into, for example, a spatial modulation type and a scanning type from the viewpoint of a method of forming image light. The spatial modulation type is a system in which the image light is formed from light emitted from a light source using a spatial modulation element (for example, liquid crystal or organic EL (electroluminescence)) that operates according to an image signal. On the other hand, the scanning type is a system in which the image light is formed by scanning light emitted from a light source and having intensity corresponding to the image signal.

  The HMD is classified into, for example, a see-through type and a sealed type from the viewpoint of a method in which an observer observes an image. In the see-through method, image light representing an image according to an image signal and external light from an actual external environment (an example of the “target space”) positioned in front of the observer are superimposed and incident on the eyes of the observer. In this way, the observer can observe an image corresponding to the image signal superimposed on an image of the actual outside world. On the other hand, the sealed type is a system that blocks the incident light of the external light representing the actual external environment and allows the observer to mainly observe only the image corresponding to the image signal.

  When the latter sealed type is adopted, it is possible to design the HMD so that the image corresponding to the image signal does not include the image of the target space. In this case, the observer cannot observe the image of the target space, and observes the image corresponding to the image signal alone. However, it is also possible to design the HMD so as to capture an image of the target space, combine the captured image with another image corresponding to the image signal, and display the combined image to the observer. . In this case, the observer can observe another image superimposed on the image of the target space, as in the case of adopting the see-through method. Hereinafter, for convenience of explanation, this method is referred to as a “video see-through type”, while the above-described see-through type is referred to as an “optical see-through type” to distinguish between them.

  There is already a technology for augmenting and expanding the real world perceived by humans using a computer, which is called augmented reality (AR). One conventional example of a technique for realizing this augmented reality using an optical see-through type or video see-through type HMD is disclosed in Patent Document 1.

  According to the technique disclosed in Patent Document 1, while an observer observes a virtual object existing in a virtual space (another example of the “target space”), the observation position of the virtual object is set. In association with each other, reference content (for example, annotation) related to the virtual object is displayed. According to this technology, it becomes possible for the observer to visually and automatically acquire information related to the virtual object while observing the virtual object, and the convenience of the observer is improved.

JP 2005-038008 A

  According to the technique disclosed in Patent Document 1, when the observer observes the only virtual object, when the observer gazes at the virtual object for a certain period of time, in the vicinity of the observation position of the virtual object, Annotation as reference content related to the virtual object is automatically displayed. At this time, in order to make the focus on the annotation coincide with the focus on the virtual object, the depth from the observer's viewpoint position to the annotation is almost the same as the depth from the viewpoint position to the virtual object.

  However, in the technology disclosed in Patent Document 1, unless the observer gazes at the same virtual object for a certain period of time, the annotation related to the virtual object is not displayed. Therefore, the observer is in front of the gaze. Cannot access any information related to the virtual object. This is inconvenient for an observer who wishes to obtain some information about the virtual object as long as the virtual object is observed.

  Patent Document 1 also discloses that regarding the setting of the depth of the annotation, the depth of the same annotation related to the virtual object is changed with time depending on whether or not the observer gazes at one virtual object. Not.

  On the other hand, as a result of conducting research on this type of HMD, the present inventors, as long as the observer is observing the object, when the reference content related to the object is displayed, the observer Depending on whether or not the user has focused on one object, it is desirable to change the depth of the same reference content related to the object over time in order to improve comfort when the viewer visually recognizes the reference content. I realized that.

  Throughout this specification, the act of observing a specific object by the observer means that the observer views the specific object so that the observer's line of sight intersects the specific object. Means an observer's external behavior, whereas an act of an observer “focusing on” a particular object means that the observer is more interested in that particular object than other objects Means the inner act of. If an observer is looking at a particular object such that the observer's line of sight intersects with that particular object, it is clear that the observer is “attention” to that particular object.

  However, even if the observer is not looking at the specific object so that the observer's line of sight intersects with the specific object, the observer may be “attention” to the specific object obtain. Even if the target that the observer is watching and the target that the observer is paying attention to are different from each other, there is a strong connection between the objects (for example, an object described later and at least the contents of the object) If the target that the observer is gazing at is specified, the other target that the observer should be paying attention to can be estimated with high accuracy. Therefore, even if the object that the observer is watching and the object that the observer is paying attention do not necessarily match each other, if the object that the observer is watching is identified, the observer will pay attention. Other objects that should be present can also be estimated with high accuracy.

  With the above-described knowledge as a background, the present invention displays an image to the observer in a state of being worn on the observer's head, thereby providing at least one object and at least one reference content in the target space. An object of the present invention is to improve comfort when an observer visually recognizes a reference content in a head-mounted display that allows an observer to observe in association with each other.

  The following aspects are obtained by the present invention. Each aspect is divided into sections, each section is numbered, and is described in the form of quoting the section numbers of other sections as necessary. In this way, each section is referred to the section number of the other section. By describing in the format, the technical features described in each section can be appropriately made independent according to the properties.

(1) An image is displayed to the observer in a state of being worn on the observer's head, whereby the observer observes the at least one object in the target space and at least one reference content associated with each other. A head-mounted display that enables
A first display control unit for displaying the at least one reference content so as to have a first depth;
While the display of the at least one reference content, when the observer shifts to a state of gazing at any reference content or any object related to the reference content, the depth of the reference content is And a second display control unit configured to change the first depth to a second depth substantially the same as the depth of any one of the objects.

(2) The at least one object includes a plurality of objects having at least two objects having different distances from the head mounted display,
The at least one reference content includes a plurality of reference content;
The first display control unit includes a list display unit that performs a list display for displaying the plurality of reference contents so that each of the plurality of reference contents has the first depth,
The second display controller is
During execution of the list display, the observer extracts the reference content that is substantially most intensely watched from among the plurality of reference contents displayed in the list as the gaze reference content, and among the plurality of objects, A target object estimation unit that estimates an object corresponding to the extracted gaze reference content as a target object;
When the attention object is estimated, at least attention display for performing attention display that changes the depth of the gaze reference content from the first depth to the second depth substantially the same as the depth of the attention object. The head mounted display according to item (1), including:

(3) Furthermore,
An object acquisition unit that acquires a plurality of objects existing in a predetermined region with respect to the head-mounted display in the target space, together with depth-related information that can specify a depth of each object;
A reference content selection unit that selects a plurality of reference contents respectively corresponding to the plurality of acquired objects from a plurality of reference contents that can be displayed by the head-mounted display,
The head mounted display according to item (2), wherein the list display unit displays a list of the plurality of selected reference contents.

(4) The attention display unit displays the reference content corresponding to the attention object so as to have a display size larger than a display size when the reference content is displayed in the list (2) or (3 ) Head-mounted display.

(5) The list display unit includes a display size determination unit that determines the display size of the reference content corresponding to the object according to the depth of the object from the head-mounted display. The head mounted display according to any one of 4).

(6) The list display unit includes a display item determination unit that determines the content of the information item included in the reference content corresponding to the object according to the depth of the object from the head-mounted display ( The head mounted display according to any one of items 2) to (5).

(7) The head mounted display according to any one of (2) to (6), wherein the attention display unit starts the attention display when the movement of the observer's head substantially stops.

(8) The head mounted display according to any one of (2) to (7), wherein the attention display unit terminates the attention display when a substantial movement occurs in the observer's head.

(9) The attention display unit terminates the attention display when the attention object is out of a field of view determined relatively to an observer's observation eye. Head mounted display.

(10) When the plurality of reference contents correspond to at least one object existing in the actual outside world, the list display unit, among the plurality of reference contents, based on the importance assigned in advance to each reference content The head-mounted display according to any one of (2) to (9), wherein the number of reference contents displayed in a list is reduced by displaying one excluding at least one reference content.

(11) Further, a line-of-sight detector that detects the line of sight of the observer is included,
The attention object estimation unit determines whether or not the detected line of sight is in a continuous intersection state in which the detected line of sight continuously intersects any one of the plurality of reference contents displayed in a list for a predetermined time or more. The head-mounted display according to any one of claims 2 to 10, wherein when any of the reference contents is determined to be in the continuous intersection state, the reference content is determined to be the gaze reference content.

(12) An image is displayed to the observer in a state of being worn on the observer's head, whereby the observer observes the at least one object and at least one reference content in the target space in association with each other. A method of operating a head mounted display that enables
A first display step of displaying the at least one reference content to have a first depth;
While the display of the at least one reference content, when the observer shifts to a state of gazing at any reference content or any object related to the reference content, the depth of the reference content is A second display step of changing from the first depth to a second depth substantially the same as the depth of any one of the objects.

(13) A program executed by a computer to execute the method according to (12).

(14) Display configured to be able to be emitted by superimposing image light indicating reference content related to an object existing in the actual external environment and external light indicating an image of the actual external environment, and capable of being mounted on an observer's head Unit,
A position acquisition unit capable of acquiring the position of the display unit;
A focus adjustment unit that changes a focus position of the image light;
When the display unit is mounted on the observer's head, a gaze target determining unit that determines a gaze target that is a target to be watched by the observer;
A control unit for controlling the operation of the display unit,
Its control part is
Content acquisition means capable of acquiring reference content related to an object located within a predetermined region with respect to the display unit based on the position of the display unit detected by the position detection unit;
Display control means capable of displaying the reference content acquired by the content acquisition means on the display unit;
A determination unit capable of determining whether or not the gaze target determined by the gaze target determination unit corresponds to the reference content acquired by the content acquisition unit;
When the determination unit determines that the gaze target corresponds to the reference content, the focus adjustment unit is controlled so that the focus position of the display image is associated with the reference content. A focus changing means for changing to a focus position displayed at the same distance as the object.

  According to the present invention, while the observer is observing the object, the reference content related to the object is displayed even before the observer pays attention to the object, and further, the observer pays attention to the object. Depending on whether or not, the depth of the reference content related to the object changes with time. Therefore, according to the present invention, comfort when the viewer visually recognizes the reference content is improved.

FIG. 1 is a perspective view showing a head-mounted display (HMD) according to an exemplary embodiment of the present invention in a left-eye observation mode and with a frame used for mounting on an observer's head. FIG. 2 is a longitudinal sectional view showing the display unit shown in FIG. FIG. 3A is a side view showing the focus adjustment mechanism shown in FIG. 2 with the LCD adjuster removed, and FIG. 3B shows the focus adjustment mechanism with the LCD adjuster assembled. FIG. FIG. 4 is a functional block diagram conceptually showing the configuration of the HMD shown in FIG. FIG. 5 illustrates an example of development of an image visually recognized by an observer when the HMD illustrated in FIG. 1 is operated, in order to explain a difference between an image visually recognized in the list mode and an image visually recognized in the gaze mode. FIG. FIG. 6 (a) is a diagram illustrating the entire exhibition room in order to explain how an observer is viewing a plurality of exhibits in a certain exhibition room while using the HMD shown in FIG. FIG. 6B is an example of content-related information stored in the content information storage unit illustrated in FIG. 4, and is an example illustrated in FIG. It is a conceptual diagram showing a corresponding thing in a tabular form. FIG. 7A is a conceptual diagram for explaining an example of a technique for narrowing down the reference content to be displayed in the examples shown in FIGS. 6A and 6B. FIG. In order to explain an example of an image visually recognized by an observer when the HMD shown in FIG. 1 is operated, and an example of a layout in the image display area of the HMD of a plurality of reference contents narrowed down by the method FIG. FIG. 8 is a diagram illustrating an example of an image visually recognized by an observer when the HMD illustrated in FIG. 1 is operated in order to describe a modification of the list mode illustrated in FIG. FIG. 9 is a flowchart conceptually showing an image display method executed by the image display program stored in the memory shown in FIG. 4 being executed by the processor shown in FIG. FIG. 10 is a flowchart conceptually showing a part of the list mode module constituting step S8 shown in FIG. FIG. 11 is a flowchart conceptually showing another part of the list mode module. FIG. 12 is a flowchart conceptually showing the gaze mode module constituting step S12 shown in FIG.

  Hereinafter, some more specific and exemplary embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a perspective view of a head mounted display (hereinafter abbreviated as “HMD”) 10 according to an exemplary embodiment of the present invention.

  First, to explain schematically, the HMD 10 is used by being mounted on the head of an observer so as to be positioned in front of the eyes of the observer. In the state of use, the HMD 10 guides external light representing an image of the real external environment (an example of the “target space”) positioned in front of the observer to the observer's eyes. Observe objects that exist in the outside world (for example, three-dimensional objects such as humans, machines, buildings, animals, plants, sculptures, and manufactured products, and two-dimensional objects such as photographs, paintings, movies, and printed materials) Designed to be possible.

  Further, the HMD 10 generates reference content related to the object that the observer is observing (for example, an image that represents information to be provided to the observer, text, graphics, an annotation, etc.) according to the image signal. The generated reference content is designed to be displayed to the observer in association with the observation position of the object currently being observed. In one example, the object is an exhibit that is displayed in a display room in a facility such as a museum, and the reference content is text or graphics to supplement the exhibit. Annotation.

  Further, the HMD 10 displays the reference content related to the object that the observer is observing so as to have the first depth, and during the display, the viewer shifts to the state in which the viewer gazes at the reference content or the object. Then, it is also designed to change the depth of the reference content from the first depth to a second depth substantially the same as the depth of the object. That is, in the HMD 10, the depth of the reference content related to the object changes with time depending on whether or not the observer has focused on the object. Therefore, according to the HMD 10, the reference content is observed. Display can be performed under conditions that match the user's sense, and comfort when the viewer visually recognizes the reference content is improved.

  Next, the configuration of the optical system employed by the HMD 10 will be described with reference to FIGS.

  The HMD 10 includes a display unit 12, an attachment 14, and a control unit 16. The control unit 16 is connected to the display unit 12 via a cable 18 (including at least a signal line, usually including a power line) in a wired manner (or a wireless manner is possible), so that data between the two can be obtained. And signals can be exchanged.

  The display unit 12 is a monocular type, and is configured to project image light representing an image onto an observation eye that is one of the eyes of the observer (the other eye is a non-observation eye). The display unit 12 is an optical see-through type, that is, by causing image light representing an image corresponding to an image signal and external light from the real external environment located in front of the observer to be superimposed on the observation eye, A method is employed in which an observer can observe an image corresponding to an image signal superimposed on an image of the actual outside world.

  However, in the display unit 12, the present invention can be implemented in a mode in which the monocular type is changed to the binocular type or in a mode in which the optical see-through type is changed to the video see-through type. However, when the video see-through method is adopted, the HMD 10 performs, for example, an image display according to an image signal and an image of the actual outside world in a time-sharing manner at different times. Accordingly, it is necessary that the focus position of the image corresponding to the image signal and the focus position of the image of the actual outside world can be adjusted independently of each other.

  On the other hand, when a plurality of objects having different depths exist in the actual outside world, the HMD 10 guides the outside light emitted from these objects to the observation eye, so that the observer can connect the objects to each other. While allowing observation with different depths, it is possible for an observer to visually recognize a plurality of reference contents so as to have the same depth with each other by image light common to them.

  Therefore, in the HMD 10, light representing a plurality of objects is reflected light from each object and does not pass through a lens common to the objects, whereas light representing a plurality of reference contents. Are passing through a lens common to the reference contents, the depth of the reference contents can be freely adjusted relative to the depth of the object by changing the position of the lens.

  As shown in FIG. 1, a frame 20 as a member to be attached is attached to the observer's head, an attachment 14 is attached to the frame 20, and a display unit 12 is attached to the attachment 14. As a result, the display unit 12 is attached to the frame 20 via the attachment 14.

  In the example shown in FIG. 1, the display unit 12 is attached to the left side portion 30 </ b> L of the frame 20, whereby the HMD 10 is used in the left eye observation mode, but the display unit 12 is attached to the right side of the frame 20 by the attachment 14. It can be used by changing between the portion 30R and the left portion 30L.

  Specifically, the attachment 14 includes a left-eye frame side member 40L fixed to the left side 30L of the frame 20, and a right-eye frame side member 40R (not shown) fixed to the right side 30R of the frame 20. It has. The attachment 14 has a display unit attached to the left-eye frame side member 40L attached to the left side 30L of the frame 20 in order to realize a left-eye observation mode in which the observer's left eye is an observation eye according to the observer's selection. 12 and the right-eye frame side member attached to the right side 30R of the frame 20 in order to realize the right eye observation mode in which the right eye of the observer is the observation eye (see FIG. 1). The state is switched to a state in which the display unit 12 is attached to 40R (not shown).

  The display unit 12 is used with its direction reversed between the right eye observation mode and the left eye observation mode. Specifically, the direction in which the display unit 12 is mounted on the frame 20 via the attachment 14 is set between the right-eye observation mode and the left-eye observation mode with respect to the left-right direction of the observer in the head-mounted state. They are reversed with respect to each other in a vertical plane that is substantially parallel.

  As shown in FIG. 1, the display unit 12 has a housing 200 that is generally hollow extending in the longitudinal direction thereof. The housing 200 is made of synthetic resin. Housed in the housing 200 is an image light forming unit 202 (see FIG. 2) that forms image light for displaying reference content as a display image.

  As shown in FIG. 2, the display unit 12 includes a main body portion 210 that accommodates the image light forming unit 202 and an emission port 212 that emits image light formed by the image light forming unit 202 toward the observation eye. The emission port portions 214 are arranged so as to be aligned with each other along the longitudinal direction of the display unit 12. The display unit 12 further includes a connecting mechanism 270 that removably connects the main body portion 210 and the emission port portion 214 to each other.

  As shown in FIG. 2, the image light forming unit 202 generates the image light by two-dimensional spatially modulating incident light, and an LCD 220 (an example of a display unit) as a generally plate-shaped spatial light modulation element. Have The LCD 220 is a liquid crystal display in which a plurality of pixels are arranged two-dimensionally.

  In addition, in the present embodiment, the display unit 12 is a spatial light modulation type. However, the retinal scanning type, that is, a light beam from a light source such as a laser is scanned by a scanner, and the scanned light beam is observed by an observer. You may change to what is projected on the retina.

  The image light forming unit 202 further includes a plate-like driver 222 (an example of a display control unit). The driver 222 is electrically connected to the LCD 220 via a cable (not shown). The driver 222 drives the LCD 220 based on an image signal input from the outside, thereby causing the LCD 220 to emit image light representing an image to be displayed to the observer. The LCD 220 includes a backlight light source, but a light source independent of the LCD 220 may be used instead of the light source.

  As shown in FIG. 2, the display unit 12 further includes an eyepiece optical system 246. The eyepiece optical system 246 is configured as a one-dimensional array in which a plurality of lenses as a plurality of optical elements are arranged in series. These lenses share the same optical axis. The optical axis extends in a straight line parallel to the longitudinal direction of the housing 200 and without being bent by a component such as a mirror. The eyepiece optical system 246 is accommodated in a portion of the housing 200 that belongs to the main body portion 210.

  The eyepiece optical system 246 includes a terminal lens 248 located on the most downstream side among a plurality of lenses belonging to the eyepiece optical system 246, and a protective transparent body (for example, a transparent disc made of synthetic resin) 250 disposed downstream of the terminal lens 248. And.

  The exit portion 214 is a half mirror as an example of a partial reflection / partial transmission optical element that bends the image light emitted from the eyepiece optical system 246 and guides it to the observation eye (an example of a deflection member that deflects the image light toward the observation eye). 260.

  The image light emitted from the eyepiece optical system 246 is reflected by the half mirror 260, passes through the pupil of the observation eye, and enters the retina (not shown). Thereby, the observer can observe the two-dimensional image as a virtual image. Not only the image light reflected by the half mirror 260 but also external light, which is light from the actual external environment, passes through the half mirror 260 and enters the observation eye. As a result, the observer can observe the actual outside world in parallel with the observation of the image displayed by the image light.

  Next, a configuration for the HMD 10 to adjust the focus position of the display image will be described mainly with reference to FIGS. 2 and 3.

  As shown in FIG. 2, the image light forming unit 202 further includes a focus adjustment mechanism 230 that adjusts the focus position of the image light by displacing the LCD 220 back and forth in the optical axis direction. The focus adjustment mechanism 230 includes an LCD holder 232, a support mechanism 234, an LCD adjuster 236 (operation unit), and a motion conversion mechanism 238.

  The LCD 220 has an optical axis direction that coincides with the longitudinal direction of the image light forming unit 202, and is supported in the housing 200 so as to be displaceable in the optical axis direction. Specifically, the LCD 220 is held by an LCD holder 232 so as to be integrally rotatable and movable, and the LCD holder 232 is a part of the housing 200 or another member fixed to the housing 200 (in this embodiment, The support member 234 provided on the stationary member 240) is supported so as to be movable in the optical axis direction and not rotatable.

  As shown in FIG. 2, the LCD adjuster 236 is installed in the housing 200 so as to be coaxial with the optical axis and surround the outer periphery of the LCD holder 232. The LCD adjuster 236 is supported by the housing 200 so that it cannot move in the optical axis direction and can rotate coaxially. Thereby, the LCD adjuster 236 is rotated by a necessary angle in accordance with a user operation at a fixed position in the optical axis direction.

  In order to enable access to the LCD adjuster 236 from the user who is an observer, as shown in FIG. 1, the LCD adjuster 236 has a front portion and a rear portion (see FIG. In FIG. 1, they are exposed from the front surface and the rear surface of the plurality of surfaces constituting the housing 200. A plurality of teeth are formed on the outer peripheral surface of the LCD adjuster 236 in order to improve the operability for the user.

  The motion conversion mechanism 238 is associated with the LCD adjuster 236 and the LCD holder 232. The motion conversion mechanism 238 converts the rotational motion of the LCD adjuster 236 into the linear motion of the LCD holder 232, thereby moving the LCD 220 to a desired position along the optical axis direction.

  3A is a side view showing the focus adjustment mechanism 230 with the LCD adjuster 236 removed, and FIG. 3B shows the focus adjustment mechanism 230 with the LCD adjuster 236 assembled. It is a side view.

  In the present embodiment, the motion conversion mechanism 238 is formed on the spiral cam groove 242 formed on the outer peripheral surface of the LCD holder 232 and the inner peripheral surface of the LCD adjuster 236 as shown in FIG. And a drive pin (not shown) extending in the radial direction. The drive pin is engaged with the cam groove 242 without any change, and is relatively moved along the cam groove 242 in the engaged state. When the LCD adjuster 236 is rotated, the drive pin performs a circular motion around the optical axis, and the circular motion is converted into a linear motion of the LCD holder 232 by the joint action of the cam groove 242 and the support mechanism 234. .

  However, the motion conversion mechanism 238 can perform necessary motion conversion by a mechanism other than the cam mechanism. For example, the motion conversion mechanism 238 can be a screw mechanism (for example, a screw formed on the inner peripheral surface of the LCD adjuster 236 and the LCD holder 232. May be employed that is formed on the outer peripheral surface of the inner thread of the female screw and the female screw is screwed together.

  As described above, the HMD 10 has a variable depth display function for displaying the reference content so as to have a variable depth in the usage state. In order to realize this variable depth display function, in this embodiment, the focus adjustment mechanism 230 can be electrically operated in addition to being manually operable as described above. It has become.

  Specifically, as shown in FIGS. 1 and 4, an electric actuator 280 (for example, a step motor) is engaged with the focus adjustment mechanism 230 in its LCD adjuster 236. The actuator 280 is supported by the housing 200. The actuator 280 is driven by an electrical signal supplied from the driver 282, and is controlled so that the actual operation amount of the actuator 280 matches the target amount. The operation of the actuator 280 is transmitted to the LCD adjuster 236, and the LCD adjuster 236 is rotated, whereby the focus adjustment mechanism 230 is electrically operated.

  Next, with reference to FIG. 4, a hardware configuration of a part of the HMD 10 that displays the reference content in association with the object will be described.

  The display unit 12 includes an input device 12a and an output device 12b.

  The input device 12a is the position of the observer wearing the display unit 12 (for example, the position of the head of the observer, the position of the display unit 12, the position on the global coordinate system, and the facility A position detection unit 300 that detects a position on a fixed local coordinate system). The position detection unit 300 is, for example, a device mounted on the display unit 12 for detecting the position of the observer by the global positioning system GPS, or for detecting the position of the observer by the three-point positioning method. In addition, a device mounted on the display unit 12 can be used.

  The input device 12a further includes an orientation detection unit 302 that detects the orientation of the observer (for example, the orientation of the observer's head and the orientation of the display unit 12). The orientation detection unit 302 is, for example, a device mounted on the display unit 12 for detecting the orientation of the observer using an electronic compass, or an angular acceleration sensor or an angular velocity sensor mounted on the display unit 12 as a main component. A device mounted on the display unit 12 can be used to calculate the angle of the observer by time-integrating the detected value.

  As another example of the direction detection unit 302, a configuration that detects a relative direction with respect to an object may be employed. In this case, for example, a camera is connected to the display unit 12, and the relative direction of the display unit 12 with respect to the object may be detected based on an image obtained by imaging the object with the camera. . In this case, if necessary, the detection result of the position detection unit 300 may be used to specify the position information of the object to be imaged.

  The input device 12a further includes a line-of-sight detection unit 304 that detects the line of sight of the observer. The line-of-sight detection unit 304 can be, for example, a device that calculates the angle of the observer by using a device mounted on the display unit 12 to detect the line of sight of the observer with an eye mark recorder.

  According to the line-of-sight detection unit 304, it is possible to accurately detect the observer's line-of-sight direction regardless of whether or not it matches the front direction of the observer. It is possible to adopt a method. In one example, on the assumption that the observer's line-of-sight direction coincides with the front direction of the observer, the front direction of the observer is detected based on the detection result of the orientation detection unit 302, and the detected front direction. The line-of-sight direction of the observer is determined so as to match.

  In contrast, the output device 12b includes the driver 222 and the LCD 220 described above for image display, and further includes the driver 282, the actuator 280, and the focus adjustment mechanism 230 described above for focus adjustment, that is, depth adjustment. Have

  The control unit 16 includes a processor 310 and a memory 312. The memory 312 is an example of a recording medium. A program storage unit in which an image display program (see FIGS. 9 to 12) described later is recorded in advance, and reference content is displayed by the HMD 10 in association with the position of the object. And a content information storage unit for storing information necessary for the storage. The content information storage unit includes, for example, a content information storage unit that stores content related information related to a plurality of reference contents that can be displayed by the HMD 10 in association with a plurality of corresponding objects.

  Note that one reference content is not necessarily displayed in association with one object, and a plurality of reference contents may be displayed in association with one object.

  The content related information (see FIG. 6B) includes, for example, image data representing the reference content, coordinate data representing the position of the object displayed in association with the reference content (position of at least one representative point), reference content Data representing the display size of the data, data representing the configuration of information items included in the reference content, and the like. This content-related information is the position of a plurality of objects that the observer observes as a through image through the display unit 12 (for example, the position of at least one representative point that geometrically represents each object, and depends on the size of the object). The object-related information for specifying the number of representative points can be increased. By referring to the object-related information, it is possible to obtain the distance from each HMD 10, that is, the depth of each object.

  The processor 310 implements a plurality of functions by executing the image display program while using data stored in the memory 312. These functions are shown in a functional block diagram in FIG. 4, which will be described in detail later.

  As shown in FIG. 4, the control unit 16 further includes a communication module 320, and the communication module 320 is connected to an external server 322 by wireless or wired communication. The server 322 transmits the content related information to the control unit 16. When the processor 310 receives the content-related information, the processor 310 temporarily stores it in the memory 312.

  In the present embodiment, the content-related information is temporarily received from the server 322 and stored in the HMD 10 itself. For example, the HMD 10 does not receive from the server 322 and stores the memory in the server 322. Necessary data processing can be performed using (not shown).

  Next, a content display algorithm for displaying the reference content in association with the object by the HMD 10 will be described mainly with reference to FIGS.

  First, to explain schematically, according to this content display algorithm, a plurality of objects that exist in the actual external world and are within the viewer's field of view (having at least two objects with different distances from the HMD 10) Is selected, and a plurality of reference contents corresponding to the selected objects are selected. Here, as shown in FIG. 6A, the “viewer's field of view” is a fixed three-dimensional that is determined relative to the position (x0, y0, z0) and direction (z-axis direction) of the observation eye. The target angle range is generally a fan-shaped range in plan view. Here, the “z-axis direction” means the front direction of the observer, and this direction does not necessarily coincide with the line-of-sight direction of the observer. The fan-shaped range is a range that is referred to in order to detect an object to be displayed in association with the reference content from among a plurality of objects existing in the actual outside world. Focusing on this, this range is hereinafter also referred to as “object detection range”.

  In this embodiment, there are a list mode and a gaze mode as modes for displaying these reference contents. Among these list modes and gaze modes, the list mode is the default mode, and is executed before the gaze mode after the HMD 10 is turned on. The list mode is a mode for displaying a plurality of reference contents so as to have a first depth in association with a plurality of objects within the object detection range.

  Here, the “first depth” is a fixed value (for example, a maximum change range of the depth length (distance from the HMD 10), for example, a plurality of ranges (for example, “near range (near)”, “middle range”). (Medium) ”and“ far range (far) ”), or variable values that are automatically adjusted (for example, object detection) It is possible to set as a variable value that can be freely adjusted by an observer.

  There are various examples of this list mode. In one example, even if a plurality of reference contents are displayed at the same time, there is a limit to the number of reference contents that can be reliably viewed by the observer (the total value of the display sizes of the plurality of reference contents to be displayed is The display size of the reference content is based on the fact that if the reference content is displayed all at once even though it exceeds the total visible image display area, duplication between the reference content will occur) If the total value exceeds the total area of the image display area, the reference content is based on the depth of each object (the longer the depth, the less likely the viewer is paying attention to the specific object). It is possible to narrow down objects displayed in association with each other.

  According to this example, an object is located within the object detection range, but if the distance from the observer is quite far, it is unlikely that the observer is paying attention to such an object. The display of the corresponding reference content is omitted.

  In the list mode, in order to determine the timing for shifting to the gaze mode, it is appropriate that the behavior of the observer is sequentially detected, and based on the detection result, the current list mode is terminated and the mode is shifted to the gaze mode. It is determined whether or not. This is the first gaze determination.

  Specifically, in the present embodiment, when it is estimated that any one of the plurality of reference contents displayed at the same time by executing the list mode is the gaze reference content being watched by the observer In addition, it is determined that it is appropriate to end the current list mode and shift to the gaze mode.

  More specifically, in the present embodiment, when an observer is gazing at any reference content (the observer's line of sight is directed exclusively to specific reference content), this is handled. It is presumed that the object is not the object of interest at that time, but is the object of interest that is the strongest (stronger than other objects) attention (the strongest desire for detailed information about the object). The Thus, in the present embodiment, the object of interest is estimated by referring to the observer's gaze degree for a plurality of reference contents displayed at the same time.

  There are various examples of the method for performing the first gaze determination.

  In one example, the observer's behavior is detected sequentially, and based on the detection results, the observer is in a substantially moving state (eg, substantially walking and / or head substantially moving). It is determined whether the state has shifted from a substantially stationary state to a substantially stationary state (eg, a substantially stationary state and / or a state where the head is substantially stationary). When it is determined that the transition has occurred, it is estimated that there is a high possibility that the observer is gazing at any reference content (in a gazing state).

  In this example, when it is estimated that the user is in the gaze state, the observer is actually gazing among a plurality of reference contents displayed at the same time based on the detection result of the behavior of the observer. Gaze reference content is extracted. In an example of the extraction method, an observer's line of sight is detected, and a plurality of reference contents that are on the extension of the detected line of sight (that is, an observer's viewpoint position (observer's eye position)) The reference content intersecting with the line of sight extending in the direction of the detected line of sight) is extracted as the gaze reference content. In another example of the extraction method, when an observer wants to gaze at an object located in front of himself / herself, the gaze object moves unconsciously to the center position of his / her field of view. In this way, paying attention to the habit of moving the head or moving the body, the reference content within a predetermined position (for example, the center position) within the object detection range is extracted as the gaze reference content.

  When the gaze mode is started, the gaze reference content extracted as described above is displayed so as to have a second depth that is substantially the same as the depth of the object corresponding to the gaze reference content. Thereby, the depth of the gaze reference content is changed from the first depth to the second depth.

  Here, in relation to the setting of “second depth”, the relationship between the second depth of the gaze reference content and the depth of the target object will be described. In one example, the depth of the target object is substantially equal. When it is detected as a continuous value or a plurality of discrete values, the second depth is set so as to completely or substantially match the detected value of the depth. In another example, when it is determined which of a plurality of different ranges (values having a width) the depth of the object of interest belongs to, it matches with an arbitrary value within the range to which the target object belongs. Is set to the second depth.

  In yet another example, the difference between the depth of the object of interest and the second depth for the gaze reference content is the depth of the other object and the depth of the other non-gaze reference content (in this case, the first The second depth for the gaze reference content is set so as to be smaller than the difference between the gaze reference content and the gaze reference content. In this example, whether or not the depth of the object of interest and the second depth of the gaze reference content are substantially equal to each other is absolutely determined considering only the difference between the two depths. Rather, it is performed relatively by comparing the difference between the depth of the other object and the depth of the other non-gaze reference content.

  As a result, according to the present embodiment, in the gaze mode, the observer can visually recognize the object of interest and the gaze reference content at the same depth position, that is, the same focus position at the same time. The fatigue of the observer's eyes is reduced while improving. Further, according to the present embodiment, the observer can visually recognize the gaze reference content at a position close to the object of interest, so that when moving both objects alternately, the moving distance of the focus position can be shortened. This also improves the visibility of both objects and reduces the fatigue of the eyes of the observer.

  There are various examples of methods for displaying gaze reference content in the gaze mode. In one example, the gaze reference content is displayed so as to be larger than the display size when the gaze reference content is displayed as a list (for example, enlarged at a predetermined magnification larger than 1). According to this example, the viewer can display the gaze reference content separately from the other reference content displayed together, not only with respect to the focus position but also with respect to the display size. The visibility of content is improved.

  In the present embodiment, the HMD 10 does not have a function of displaying each reference content at a different focus position for each reference content. For this reason, when a plurality of reference contents are displayed at the same time, these references are referred to. The second focus depth is set variably according to the focus position common to the content, that is, the depth of the target object (if the target object changes to another object having a depth different from that of the current object, As the second depth also changes).

  However, as described above, in the gaze mode, the gaze reference content is enlarged as compared with the case where the gaze reference content is displayed in the list mode. Since it is displayed in association with the position of the target object corresponding to the target object, the observer can easily recognize the correspondence between the gaze reference content and the target object.

  However, the present invention is applied to an HMD having a function of displaying each reference content at a different focus position for each reference content, or in a mode in which the reference content other than the gaze reference content is not displayed in the gaze mode. Can be implemented.

  In the gaze mode, in order to determine the timing for shifting to the list mode, it is appropriate that the behavior of the observer is sequentially detected, and based on the detection result, the current gaze mode is terminated and the list mode is shifted to. It is determined whether or not. This is the second gaze determination.

  There are various examples of the method for performing the second gaze determination.

  In one example, the behavior of the observer is sequentially detected, and based on the detection result, it is determined whether or not the observer is in a non-gaze state in which no reference content is gazing. Specifically, the observer is moving substantially from a substantially stationary state (eg, substantially stationary and / or a head that is substantially stationary). It is determined whether or not a first condition that a state (for example, a state where the user is substantially walking and / or a head is substantially moving) is satisfied. When it is determined that the first condition is satisfied, it is determined that the transition to the list mode is appropriate.

  In another example, the gaze of the observer is sequentially detected while waiting for the first condition to be satisfied or regardless of whether or not the first condition is met, and the gaze reference content of this time on the extension of the detected gaze It is determined whether or not the second condition that no exists is established. When it is determined that the second condition is established alone or together with the first condition, it is determined that the transition to the list mode is appropriate.

  In yet another example, the current object of interest waits for the first condition and / or the second condition to be satisfied, or regardless of whether the first condition and / or the second condition is satisfied. It is determined whether or not the third condition of deviating from a predetermined position within the object detection range is satisfied. When it is determined that the third condition is established alone or together with the first condition and / or the second condition, it is determined that the transition to the list mode is appropriate.

  As is clear from the above description, in the present embodiment, the observer automatically operates according to the presence / absence of the observer's specific action in which the observer gazes at the specific reference content that does not require an operation on the HMD 10. Since the mode is switched between the list mode and the gaze mode, the observer can use the HMD 10 in a hands-free state, and as a result, the usability of the HMD 10 is improved.

  FIG. 5 shows an image (both an image representing a plurality of objects in the actual outside world and a display object of a plurality of reference contents) visually recognized by an observer when implemented in an environment with the content display algorithm described above. An example of the development of is shown.

  In this example, an environment is a facility (eg, a museum, a museum, etc.) having an exhibition room that houses a plurality of exhibits (eg, art, dinosaur fossils, replicas, etc.). The objects are the plurality of exhibits, and the plurality of reference contents are annotations (for example, names and features of the exhibits) about the plurality of exhibits.

  In the example shown in FIG. 5, the exhibit A is present in front of the observer and the exhibit B is present in the back. For these exhibits A and B, their reference contents (portions surrounded by a square frame) S and R are displayed in a positional relationship. In the list mode, the depth of the exhibit A is shorter than the depth of the exhibit B. Therefore, the display size of the reference content S corresponding to the exhibit A is the display size of the reference content R corresponding to the exhibit B. Greater than size.

  In this situation, it is assumed that the viewer gazes at the reference content corresponding to the exhibit B (that is, the reference content corresponding to the exhibit B (the content located at the upper right in FIG. 5) is the gaze reference content. In addition, when the exhibit B is an object of interest), the list mode is automatically ended and the mode is shifted to the gaze mode.

  In the gaze mode, the display size of the reference content R corresponding to the exhibit B is enlarged from the display size in the list mode. At this time, for example, the reference content S corresponding to the exhibit A is larger than the normal size in the preceding list mode because the exhibit A is located more easily than the exhibit B. When shifting to the mode, this time, since it does not correspond to the gaze reference content, it is returned to the normal size.

  FIG. 6A shows that the observer is viewing a plurality of exhibits (for example, replicas of dinosaurs) in a certain exhibition room while using the HMD 10. It is shown in a plan view shown by looking down from above. In this exhibition room, a plurality of antennas are installed in advance, and the HMD 10 uses the position detection unit 300 of at least the position detection unit 300 and the orientation detection unit 302 to connect the three closest antennas. By using the principle of triangulation used, the position (x0, y0, z0) of the observer is detected on a local coordinate system or a global coordinate system fixed in the exhibition room. Note that the z-axis direction represents the front direction of the observer and may or may not coincide with the observer's line-of-sight direction.

  In an example that relies on the principle of triangulation, a signal is transmitted from each of a plurality of antennas simultaneously so that the signals are directed from the antenna to a plurality of other antennas, and the timing at which the plurality of antennas respectively receive the signals. The position of the observer is detected based on the difference or ratio. In another example that relies on the principle of triangulation, a signal having the same strength is transmitted from each of a plurality of antennas at the same time so as to be directed from the antenna to a plurality of other antennas. The position of the observer is detected based on the difference or ratio of the received intensity when received by the plurality of antennas.

  In this example, exhibits A, B, C, and D are displayed in the exhibition room, reference contents A1 and A2 are associated with the exhibit A, and reference to the exhibit B The contents B1 and B2 are associated with each other, the reference contents C1 and C2 are associated with the exhibit C, and the reference contents D1 are associated with the exhibit D.

  FIG. 6B shows an example of content-related information stored in the content information storage unit shown in FIG. 4 and corresponding to the example shown in FIG. . This example of content-related information is information necessary for displaying each exhibit in the list mode and the gaze mode for all exhibits existing in the exhibition room.

  In this example of the content related information, the display size of each reference content decreases as the depth of the object corresponding to each reference content increases (the distance from the viewer increases, that is, the distance from the viewer increases). The number of information items included in each reference content or the amount of information decreases as the depth of the object corresponding to each reference content is longer. These first and second rules rely on the presumption that the greater the depth of each object, the less the viewer is paying attention to a particular object.

  According to this example, the shorter the depth of each object, the larger the display size of the corresponding reference content. Therefore, in an environment in which a plurality of reference content is displayed at the same time, the observer can determine which object is the reference content. It becomes easy to intuitively determine whether they are associated. That is, it is possible to display reference content appealing to the viewer's intuition.

  Further, in this example, the position of the exhibit displayed in association with each reference content is defined by a three-dimensional coordinate value of (X, Y, Z). This three-dimensional coordinate system is initially defined, for example, on a first local coordinate system fixed to the exhibition room, but the position and orientation of the observer is relative to the first local coordinate system. Based on the positional relationship, the first three-dimensional coordinate value is converted into a coordinate value on the second local coordinate system fixed to the HMD 10 in the HMD 10. By using the three-dimensional coordinate values (X, Y, Z) representing the position of each exhibit, it is possible to calculate the distance, that is, the depth, of each exhibit from the HMD 10.

  Specifically, in this example, the maximum change range of the depth length of the object (the distance of the object from the HMD 10) is “near range (near)”, “middle range (middle)”, and “far range”. (Far) "is divided into three ranges.

  Further, in this example, the amount of information items (for example, icons and descriptions (text and graphics)) included in each reference content and the display size (for example, “large size”, “medium size”, or “Small size”) increases as the corresponding object is closer to the HMD 10.

  In this example, image data for displaying the same information item may be prepared in advance for each of a plurality of display sizes. However, in order to save the capacity of the memory 312, for example, “large” When it is necessary to store the image data for displaying each information item with “size” in the memory 312 and display the same information item with “medium size” and “small size”, The image data to be displayed in the “large size” may be generated by reading from the memory 312 and reducing the image data or deleting a part of the image data for editing.

  In this example, as shown in FIG. 6B, the exhibit A corresponds to two reference contents A1 and A2, and the exhibit B corresponds to two reference contents B1 and B2. The exhibit C corresponds to two reference contents C1 and C2, and the exhibit D corresponds to one reference content D1.

  For example, for the reference content A1, when the corresponding object is located in the “near range”, the information item (content) included in the reference content A1 is a combination of “icon A1” and “description A1”. In addition, the display size of the reference content A1 is “large size” among the three sizes “large size”, “medium size”, and “small size”.

  For the reference content A1, when the corresponding exhibit A is located in the “medium range”, the information item (content) included in the reference content A1 is only “explanation A1”, and as a result, the exhibition The number of information items and the amount of information are reduced as compared with the case where the object A is located in the “near range”. In addition, the display size of the reference content A1 is “medium size”, and as a result, the display size is reduced as compared with the case where the reference content A1 is located in the “near range”.

  When the exhibit A is located in the “medium range”, the image data necessary for displaying the reference content A1 is data representing “description A1”. This data is obtained from data representing “icon A1” and data representing “explanation A1”, which are image data necessary for displaying the same reference content A1 when the exhibit A is located in the “near range”. It is created by deleting data representing “icon A1”. Instead of this creation method, image data necessary for displaying the reference content A1 can be individually created in advance and stored in the memory 312 for each range where the exhibit A is located. In this case, there is a tendency that the effort for creating image data in advance and the capacity of the memory 312 increase.

  As for the reference content A1, when the corresponding object is located in the “far range”, the information item (contents) included in the reference content A1 is only the “icon A1”. As a result, the exhibit A The information amount of the information item is also reduced as compared with the case where is located in the “medium range” (the information amount of the icon A is smaller than the information amount of the description A1). In addition, the display size of the reference content A1 is “small size”, and as a result, the display size is also reduced as compared with the case of being located in the “medium range”.

  When the exhibit A is located in the “far range”, the image data necessary for displaying the reference content A1 is data representing the “icon A1”. This data is obtained from data representing “icon A1” and data representing “explanation A1”, which are image data necessary for displaying the same reference content A1 when the exhibit A is located in the “near range”. It is created by deleting data representing “Description A1”.

  In the example shown in FIG. 6A, among the exhibits A, B, C, and D, only the exhibits B and C exist within the object detection range of the observer. Two reference contents B1 and B2 correspond to the exhibit B, and two reference contents C1 and C2 correspond to the exhibit C. As described above, it may be necessary to display a plurality of reference contents all at once in a finite image display area. In this case, some measures must be taken to optimize the display. Is desirable.

  Specifically, when a plurality of reference contents correspond to the actual outside world observed by the observer, and when the reference contents are displayed all together, the total value of the display sizes of the displayed reference contents is the image. When the total area of the display area is exceeded, if the reference contents are displayed all at once without taking any measures, the reference contents are displayed in duplicate. In this case, there is a possibility that trouble may occur when the viewer visually recognizes the reference contents individually.

  In addition, if the reference contents are displayed in the image display area at a position determined independently of the positions of the objects corresponding to the reference contents, the observer can select which reference contents correspond to which object. Cannot be recognized easily and accurately. If a plurality of reference contents are displayed in an overlapping manner, the observer cannot normally observe the desired reference contents.

  As a technique for preventing such inconvenience, for example, when importance is assigned to each reference content in advance, and all reference content corresponding to all exhibits in the object detection range are set as candidate content, Among these candidate contents, it is possible to adopt a method of finally determining the contents to be displayed by excluding those having the lowest importance and those having the importance less than or equal to a predetermined value.

  FIG. 7A shows an example of a table necessary for narrowing display target contents according to this technique, and this example corresponds to the examples shown in FIGS. 6A and 6B. .

  In FIG. 7A, in association with each reference content, the importance of each reference content (for example, classified into two levels of “large” and “small” or more levels) and within the object detection range Information on whether or not it is located, the distance from the HMD 10, that is, the depth of the corresponding exhibit, the range in which the distance is classified (either “near”, “medium”, or “far”), Information indicating the content of the information item included in the reference content, the display size of each reference content (either “large”, “medium”, or “small”), and information indicating whether or not to allow display of each reference content ( (○: Displayed, ×: Not displayed) is displayed in a table format.

  In this example, since the objects B and C are within the object detection range, four reference contents B1, B2, C1, and C2 corresponding to them are extracted as candidate contents. These candidate contents are all displayed when the importance is not considered, but in this example, the candidate contents are narrowed down in consideration of the importance. In this example, since the importance of the reference content B2 is “small”, the reference content B2 is excluded, and finally, the three reference content B1, C1, and C2 are displayed as shown in FIG. 7B, for example. The

  In the example shown in FIG. 7B, the display positions of the plurality of reference contents are the relative positions in the horizontal direction of the objects corresponding to the reference contents (more precisely, the horizontal relative positions between the representative points). Relationship).

  As shown in FIG. 6A, the exhibit B is located on the left side with respect to the observer, while the exhibit C is located on the right side with respect to the observer. Therefore, in the example shown in FIG. 7B, such a horizontal relative positional relationship between the exhibits B and C (a relative positional relationship in a two-dimensional direction perpendicular to the line of sight of the observer) is reflected. As described above, the relative positional relationship between the display positions of the reference contents B1 and C1 and C2 is determined.

  Specifically, the reference content B1 is displayed at a position deviated to the left from the center position in the image display area, while the reference contents C1 and C2 are both displayed in the image display area. It is displayed at a position off the right side from the center position.

  In the example shown in FIG. 7B, the display position of each of the plurality of reference contents is the relative front-rear relative position relationship between the objects corresponding to the reference contents (more precisely, the front-rear relative position between the representative points). (Relationship) is reflected.

  Specifically, as shown in FIG. 6A, the position of the representative point B1 of the exhibit B is closest to the observer, and the position of the representative point C2 of the exhibit C is farthest from the observer. The representative point C1 of the exhibit C is located between the representative points B1 and C2. Therefore, in the example shown in FIG. 7B, such a front-rear direction relative positional relationship between the exhibits B and C (relative positional relationship in a direction parallel to the observer's line-of-sight direction) is reflected. In this way, the relative positional relationship between the display positions of the reference contents B1 and C1 and C2 is determined.

  Specifically, since the representative point B1 of the exhibit B is located in the “near range”, the reference content B1 corresponding to the representative point B1 is the upper partial region in the vertical direction with respect to the image display region visually recognized by the observer. The center partial area and the lower partial area are displayed in the lower partial area when divided into three, and the representative point C1 of the exhibit C is located in the “medium range”, and therefore corresponds to it. Since the reference content C1 is displayed in the central partial area, and the representative point C2 of the exhibit C is located in the “far range”, the corresponding reference content C2 is displayed in the upper partial area. Is done.

  In the example shown in FIG. 6A, two representative points B1 and B2 are associated with the exhibit B, and two reference contents B1 and B2 are associated with the representative points B1 and B2. However, since the position coordinates of the representative points B1 and B2 coincide with each other, the display positions of the reference contents B1 and B2 are determined according to the rules described above, and the reference contents B1 and B2 are simultaneously displayed. When displayed, these reference contents B1 and B2 may be displayed in duplicate. If such overlapping display is left unattended, it becomes difficult for the observer to visually distinguish the reference contents B1 and B2 from each other.

  In order to prevent such overlapping display, for example, the reference display contents B1 and B2 may be displayed at the same time after changing the initial display position so that they are arranged without overlapping. Thus, they can be displayed alternately in time.

  As described above, when a plurality of reference contents are displayed simultaneously in a finite image display area, if the number of displayed reference contents is excessive in relation to the viewing ability of the observer, the observer There is a possibility of feeling a sense of pressure and discomfort.

  As an example of a technique for solving such inconvenience, a technique is adopted in which the display state is switched from the non-selected state (total number display state) to the selected state (main part display state) during execution of the list mode. desirable.

  FIG. 8 shows an example similar to the example shown in FIG. In this example, the list mode is first executed in the non-selected state, and as a result, a plurality of reference contents B1, B2, C1-C3 corresponding to a plurality of objects B, C and D within the object detection range. And D1-D4 are displayed simultaneously.

  Thereafter, the list mode shifts to the selected state, and as a result, at least one specific object C, that is, at least one object that is likely to be observed by the observer in the future, is estimated, and at least corresponding to the object. Only one reference content is displayed in the image display area. The at least one object that is likely to be noticed by the observer in the future is selected as, for example, at least one object located in front of the observer. As described above, the list mode is switched between the non-selected state and the selected state, thereby improving the visibility when the observer visually recognizes a plurality of reference contents.

  When the observer gazes at the reference content C2 corresponding to the object C in the selected state, the viewer shifts to the gaze mode, and the reference content C2 is enlarged and displayed.

  As described above, several examples of the image display method for the HMD 10 to display an image (that is, a method for operating the HMD 10) have been described. These image display methods are stored in advance in the program storage unit of the memory 312. It is executed by the processor 310 executing the image display program. Therefore, the processor 310 conceptually has a plurality of portions for executing a plurality of functions as shown in FIG. 4 for convenience of explanation.

  Specifically, the processor 310 includes an object acquisition unit 400, a reference content display unit 402, an image signal generation unit 404, and a focus adjustment signal generation unit 406. The image signal generation unit 404 generates an image signal based on the data from the reference content display unit 402 and supplies the image signal to the LCD 230 via the driver 222. Thereby, the reference content is displayed to the observer. Further, the focus adjustment signal generation unit 406 generates a focus adjustment signal based on the data from the reference content display unit 402 and supplies the focus adjustment signal to the actuator 280 via the driver 282. As a result, the focus adjustment mechanism 230 is driven, and as a result, the focus position of the image displayed on the LCD 220 is adjusted to the target position.

  The object acquisition unit 400 receives the content related information from the server 322 via the communication module 320. The content related information includes the object related information. The object acquisition unit 400 further acquires information related to an object that exists in the field of view of the observer.

  The reference content display unit 402 includes a reference content selection unit 410 that selects, from among a plurality of displayable reference contents, one corresponding to a plurality of objects existing in the viewer's field of view. The reference content display unit 402 further displays a list display unit 412 that displays the selected reference contents in the list mode, and an attention object estimation unit 414 that estimates the attention object by extracting the gaze reference content. And an attention display unit 416 for displaying the corresponding reference content, that is, the gaze reference content in the gaze mode, in association with the attention object. The reference content display unit 402 further includes a gaze position calculation unit 418 that calculates the gaze position of the observer, and a gaze determination unit 420 that determines whether or not the observer is gazing at the specific reference content. Have

  Next, the image display method and the software configuration for executing the image display method, that is, the image display program will be described with reference to FIGS.

  FIG. 9 conceptually shows the image display program in a flowchart. This image display program is executed by the processor 310 when the power of the HMD 10 is turned on.

  When this image display program is executed, first, in step S1, the position and orientation of the observer are detected based on signals from the position detection unit 300 and the orientation detection unit 302. Next, in step S2, based on the detected position and orientation, and information for geometrically specifying the observer's field of view (stored in the memory 312), the object detection range (FIG. 9 is simply expressed as “detection range”).

  Subsequently, in step S 3, the content related information and the object related information are received from the server 322 in response to a request transmitted from the HMD 10 to the server 322, and the received information is stored in the memory 312. The Here, the “content related information” includes information regarding all reference contents that can be displayed by the HMD 10, and the “object related information” includes information regarding all objects existing in the actual outside world. Thereafter, in step S4, based on the relationship between the calculated object detection range and the object related information (including the three-dimensional position coordinates of each object), whether or not there are a plurality of objects in the object detection range. (That is, whether there are a plurality of objects in the observer's field of view).

  If it is assumed that there is only one object this time, in step S5, data for displaying the reference content corresponding to the object is read from the memory 312 and the reference content is displayed based on the data. At this time, the reference content is preferably displayed under the variable second depth so as to have substantially the same depth as the corresponding object, but is displayed under a fixed depth. It is also possible to do.

  Thereafter, in step S6, it is determined whether or not the observer has issued an end command for ending the operation of the HMD 10. When the termination command is issued, the execution of the image display program is terminated. When the termination command is not issued, the process returns to step S1.

  As described above, the case where a plurality of objects do not exist within the object detection range has been described. If there are, a plurality of objects detected within the object detection range at step S4 are selected as the current processing target at step S7. Further, by referring to the content related information, a plurality of reference contents corresponding to these objects are selected. Subsequently, in step S8, the above-described list mode is executed, whereby a plurality of reference contents associated with the positions of the plurality of objects are displayed under the first depth. This step S8 will be described in detail later.

  Thereafter, while maintaining the display state of the reference content, in step S9, based on the signals from the position detection unit 300 and / or the orientation detection unit 302, the observer's movement (for example, body movement, head movement, etc.) Whether or not is detected is determined. If it is assumed that the observer's movement is detected, the process proceeds to step S6 without shifting to the gaze mode in step S12.

  On the other hand, if it is assumed that the observer is in the gaze state, the observer is likely to be in a stationary state. Therefore, in the movement determination process based on the signal from the direction detection unit 302, the movement of the observer is detected. Not. Therefore, the determination in step S9 is “NO”, and in step S10, the line of sight of the observer (the direction of the line of sight) is detected based on the signal from the line-of-sight detection unit 304. Subsequently, in step S11, by determining whether or not the line of sight has been maintained for a predetermined time (for example, about several seconds), the observer selects one of the plurality of reference contents displayed in a list for a predetermined time. It is determined whether or not the gaze has been continued. If it is assumed that the observer has not continued to watch the specific reference content for a predetermined time or longer, the process proceeds to step S6 without shifting to the gaze mode in step S12.

  On the other hand, if it is assumed that the observer keeps gazing at any reference content for a predetermined time or more, it is determined that the reference content is the gaze reference content, and the gaze reference among the plurality of objects is determined. It is determined that the object corresponding to the content is the object of interest. Thereafter, in step S12, the above-described gaze mode is executed, whereby the reference content being watched by the observer is associated with the position of the target object corresponding to the reference content, and the depth of the target object. It has the second depth that is substantially the same, and is displayed so as to have a display size that is larger than the display size when the reference content is displayed as a list. This step S12 will be described in detail later. Thereafter, the process proceeds to step S6.

  Next, referring to FIG. 10 and FIG. 11, step S8 shown in FIG. 9, that is, a list mode module for executing the list mode will be described.

  First, in step S101, the depth d from the HMD 10 is the detection result of the position of the observer described above for one object selected in order among a plurality of objects in the field of view (the object detection range) of the observer. And the content-related information (including the three-dimensional position coordinates of each object). Next, in step S102, it is determined whether or not the calculated depth d is shorter than the lower threshold value d1. If the current depth d is shorter than the lower threshold d1, the second object when the reference content corresponding to the current object is displayed in the gaze mode in step S103, assuming that the current object is in the “near range”. Among the three discrete depth levels "Near level (predetermined minimum value)", "Medium level (predetermined intermediate value)" and "Far level (predetermined maximum value)" Set to “Near Range”. This means that the display size of the current reference content is determined to be “large” later.

  On the other hand, if the current depth d is greater than or equal to the lower threshold d1, whether or not the current depth d is greater than or equal to the lower threshold d1 and shorter than the upper threshold d2 (> d1) in step S104. Is determined. If the current depth d is equal to or greater than the lower threshold value d1 and shorter than the upper threshold value d1, it is determined in step S105 that the current object is located in the “middle range” and the corresponding reference content is the gaze mode. The second depth when displayed in is set to “medium level”. This means that the display size of the current reference content is determined to “medium” later.

  On the other hand, if the current depth d is equal to or greater than the lower threshold value d2, in step S106, the current object is positioned in the “far range” and the corresponding reference content is displayed in the gaze mode. The second depth is set to “far level”. This means that the display size of the current reference content is determined to be “small” later.

  In any case, after that, in step S107, it is determined whether or not the setting of the second depth of the reference content has been completed for all the objects in the field of view. If not completed, the process returns to step S102, and the same processing is executed for the next object. If completed, the process proceeds to step S108.

  In step S108, by referring to the content related information, the display contents and the display sizes of the plurality of reference contents corresponding to the plurality of objects in the field of view are determined according to the above-described rules. Subsequently, in step S109, whether or not the total display area T1 of the plurality of reference contents is smaller than the total area T2 of the image display area, that is, the number of reference contents to be displayed is decreased. Whether or not is unnecessary is determined. If the total value T1 is not smaller than the total area T2, that is, it is necessary to reduce the number of reference contents to be displayed, the process proceeds to step S110.

  On the other hand, when the total value T1 is smaller than the total area T2, in step S111, among the plurality of reference contents, the reference contents whose second depth is set to “near level” (the display size is “large”). )) Is 3 or more, the number of reference contents whose depth is set to “medium level” (display size is “medium”) is 4 or more, and the depth is “far level”. Whether or not at least one of the conditions that the number of reference contents (display size is “small”) set to 5 or more is satisfied, that is, the number of reference contents to be displayed may be reduced. It is determined whether it is necessary.

  If it is assumed that at least one of the three conditions is satisfied, the process proceeds to step S110. If none of the three conditions is satisfied, the process proceeds to step S112. This step S112 will be described later with reference to FIG.

  In step S110, in order to narrow down the reference content to be displayed, as described above, the plurality of objects whose depth d calculated for the object is longer than the predetermined value dmax are displayed in association with the reference content. The total area T2 is reduced by excluding the reference content to be displayed from the objects or excluding the reference content having a lower importance than a predetermined value from the reference content. Thereafter, the process returns to step S109.

  After step S109 is executed again or after step S111 is executed again, step S110 is executed again, so that if narrowing is performed again, the reference content is narrowed down under conditions more severe than the previous narrowing down. Is possible. In one example, the current value of the predetermined value dmax is decreased by a predetermined amount Δd from the previous value.

  If it is determined in step S111 that none of the above three conditions is satisfied, the process proceeds to step S112 shown in FIG. In step S112, as described above with reference to FIG. 7, the relative positions between the display positions of the plurality of reference contents are determined based on the relative positions between the plurality of objects. Then, based on the determined relative position, image data representing the reference content to be displayed is generated.

  Subsequently, in step S113, a target focus position common to the plurality of reference contents (that is, corresponding to the first depth) is determined as described above, and an actual focus position of the reference contents is further determined. Is adjusted to match the determined target focus position. Specifically, a focus adjustment signal for realizing the target focus position is generated, and the generated focus adjustment signal is output to the driver 282. As a result, the actuator 280 is driven by the target amount.

  Thereafter, in step S114, the plurality of reference contents are imaged at the target focus position, and the determined contents are respectively determined at the determined display positions (two-dimensional positions on the display surface of the LCD 220). And the display contents are displayed so as to have the determined contents (information items). This completes the execution of the list mode module.

  Next, referring to FIG. 12, step S12 shown in FIG. 9, that is, the gaze mode module for executing the gaze mode will be described.

  First, in step S201, the actual focus position of the gaze reference content is adjusted so as to coincide with the target focus position corresponding to the second depth determined according to the depth d of the object of interest. Here, the “gaze reference content” is acquired as the reference content watched by the observer for a predetermined time or more in step S11 shown in FIG. 9, and the “target object” is the gaze reference content according to the content related information. Is obtained as an object corresponding to. In step S201, specifically, a focus adjustment signal for realizing the second depth is generated, and the generated focus adjustment signal is output to the driver 282. As a result, the actuator 280 is set to the target. Driven in quantity.

  Next, in step S202, the gaze reference content is enlarged and displayed as compared with the case where it is displayed in the list mode. Specifically, image data for displaying the gaze reference content in a size larger than the other reference content is generated, an image signal reflecting the image data is generated, and the generated image signal is sent to the driver 222. As a result, a plurality of reference contents including gaze reference contents are displayed in a list. Thereby, the gaze reference content is visually emphasized over the other reference content, and as a result, the visibility of the gaze reference content is improved.

  There are various examples in step S202. In one example, the display of a plurality of other reference contents displayed together with the gaze reference content is continued without any change in the conditions / modes. In another example, as described above, the display sizes of the other reference contents are all the minimum size (for example, when the corresponding object is located in the “far range” in the list mode). Display size). According to this example, in the gaze mode, the gaze reference content is further highlighted because the display size of the other reference content is small in addition to the large display size of the gaze reference content. In yet another example, the display of a plurality of other reference contents is stopped, while the display of only the gaze reference contents is continued.

  Subsequently, while maintaining the enlarged display state of the gaze reference content, in step S203, based on the signals from the position detection unit 300 and / or the orientation detection unit 302, the observer's movement (for example, body movement, head movement) It is determined whether or not motion is detected. Assuming that the movement of the observer is detected, in step S204, the latest object detection range is calculated in the same manner as in steps S1 and S2 shown in FIG. 9, and the latest object of interest is calculated from the calculated object detection range. It is determined whether or not is off.

  If the object of interest is out of the object detection range, the enlarged display of the gaze reference content is canceled in step S205, whereby the same display size as when the gaze reference content is displayed in the preceding list mode. Is displayed. As a result, the transition from the gaze mode to the list mode is completed. This completes the execution of the gaze mode module.

  On the other hand, if it is assumed that the observer's movement is not detected, the execution of the gaze mode module ends while the enlarged display of the gaze reference content is maintained.

  As is clear from the above description, according to some of the embodiments described above, when there are a plurality of objects in the target space and there is reference content for each of these objects, The reference content is displayed in the list mode for all the objects even if the observer does not pay attention to any of them. Therefore, the observer can access related information at once for all of the objects being observed, even before such gaze (ie, prior to the selection of the object of interest). is there.

  Further, according to some of the embodiments described above, the display mode is switched between the list mode and the gaze mode depending on whether or not the observer has focused on a specific object, so that the object The related reference content depth varies over time between a state where the reference content depth is likely not to match the object depth and a state where the reference content depth substantially matches the object depth.

  On the other hand, changing the depth of the reference content related to the object with time in accordance with the temporal change in the attention level of the observer with respect to the same object means that the focus position of the observer is changed to the focus position of the object. As long as it is considered to be maintained, the degree of blurring of the reference content visible to the viewer will change over time. This change coincides with the change in the attention level.

  Thus, according to some of the embodiments described above, it is possible to display the reference content under conditions that match the viewer's senses, and comfort when the viewer visually recognizes the reference content is improved.

  As is clear from the above description, in the present embodiment, for convenience of description, steps S1 to S4 in FIG. 9 constitute an example of the object acquisition step, and step S5 in FIG. An example of the selection process is configured, and step S8 in the figure constitutes an example of the list display process, and steps S9 to S11 in the figure constitute an example of the attention object estimation process, and It can be considered that step S12 in the figure constitutes an example of the attention display step.

  As mentioned above, although several embodiment of this invention was described in detail based on drawing, these are illustrations and are based on the knowledge of those skilled in the art including the aspect as described in the above-mentioned [summary of invention] column. The present invention can be implemented in other forms with various modifications and improvements.

Claims (14)

An image is displayed to the observer while worn on the observer's head, thereby enabling the observer to observe at least one object in the target space and at least one reference content associated with each other A head mounted display that
A first display control unit for displaying the at least one reference content so as to have a first depth;
While the display of the at least one reference content, when the observer shifts to a state of gazing at any reference content or any object related to the reference content, the depth of the reference content is And a second display control unit configured to change the first depth to a second depth substantially the same as the depth of any one of the objects. The at least one object includes a plurality of objects having at least two objects with different distances from the head mounted display;
The at least one reference content includes a plurality of reference content;
The first display control unit includes a list display unit that performs a list display for displaying the plurality of reference contents so that each of the plurality of reference contents has the first depth,
The second display controller is
During execution of the list display, the observer extracts the reference content that is substantially most intensely watched from among the plurality of reference contents displayed in the list as the gaze reference content, and among the plurality of objects, A target object estimation unit that estimates an object corresponding to the extracted gaze reference content as a target object;
When the attention object is estimated, at least attention display for performing attention display that changes the depth of the gaze reference content from the first depth to the second depth substantially the same as the depth of the attention object. The head mounted display according to claim 1, comprising: further,
An object acquisition unit that acquires a plurality of objects existing in a predetermined region with respect to the head-mounted display in the target space, together with depth-related information that can specify a depth of each object;
A reference content selection unit that selects a plurality of reference contents respectively corresponding to the plurality of acquired objects from a plurality of reference contents that can be displayed by the head-mounted display,
The head mounted display according to claim 2, wherein the list display unit displays a list of the plurality of selected reference contents.   4. The head according to claim 2, wherein the attention display unit displays the reference content corresponding to the attention object so as to have a display size larger than a display size when the reference content is displayed in the list. Mount display.   The list display unit includes a display size determination unit that determines a display size of the reference content corresponding to the object according to a depth of the object from the head-mounted display. The head mounted display as described in.   The list display unit includes a display item determination unit that determines the content of the information item included in the reference content corresponding to the object according to the depth of each object from the head-mounted display. The head mounted display according to any one of 5.   The head mounted display according to claim 2, wherein the attention display unit starts the attention display when the movement of the head of the observer substantially stops.   The head mounted display according to any one of claims 2 to 7, wherein the attention display unit terminates the attention display when a substantial movement occurs in an observer's head.   The head mounted display according to any one of claims 2 to 8, wherein the attention display unit terminates the attention display when the attention object is out of a field of view determined relatively to an observer's observation eye.   The list display unit, when a plurality of reference contents correspond to at least one object existing in the real outside world, based on the importance assigned in advance to each reference content, at least one of the plurality of reference contents The head mounted display according to any one of claims 2 to 9, wherein the number of reference contents displayed in a list is reduced by displaying one excluding one reference content. Furthermore, it includes a line-of-sight detector that detects the line of sight of the observer,
The attention object estimation unit determines whether or not the detected line of sight is in a continuous intersection state in which the detected line of sight continuously intersects any one of the plurality of reference contents displayed in a list for a predetermined time or more. The head-mounted display according to any one of claims 2 to 10, wherein when any of the reference contents is determined to be in the continuous intersection state, the reference content is determined to be the gaze reference content. An image is displayed to the observer while worn on the observer's head, thereby enabling the observer to observe at least one object in the target space and at least one reference content associated with each other A method of operating a head mounted display,
A first display step of displaying the at least one reference content to have a first depth;
While the display of the at least one reference content, when the observer shifts to a state of gazing at any reference content or any object related to the reference content, the depth of the reference content is A second display step of changing from the first depth to a second depth substantially the same as the depth of any one of the objects.   A program executed by a computer to perform the method of claim 12. A display unit configured to be able to emit by superimposing image light indicating reference content related to an object existing in the real external environment and external light indicating an image of the real external environment, and a display unit that can be mounted on the head of an observer;
A position acquisition unit capable of acquiring the position of the display unit;
A focus adjustment unit that changes a focus position of the image light;
When the display unit is mounted on the observer's head, a gaze target determining unit that determines a gaze target that is a target to be watched by the observer;
A control unit for controlling the operation of the display unit,
Its control part is
Content acquisition means capable of acquiring reference content related to an object located within a predetermined region with respect to the display unit based on the position of the display unit detected by the position detection unit;
Display control means capable of displaying the reference content acquired by the content acquisition means on the display unit;
A determination unit capable of determining whether or not the gaze target determined by the gaze target determination unit corresponds to the reference content acquired by the content acquisition unit;
When the determination unit determines that the gaze target corresponds to the reference content, the focus adjustment unit is controlled so that the focus position of the display image is associated with the reference content. A focus changing means for changing to a focus position displayed at the same distance as the object.

Images